Interaction of illumination compensation and inter-frame prediction

Abstract

Systems and techniques for performing illumination compensation in processing video data include deriving one or more illumination compensation parameters for a block of a picture based on one or more tools available for inter prediction of the block. Illumination compensation may be selectively applied to the block based on whether bi-directional prediction is to be applied to inter prediction of the block. In some cases, illumination compensation for the block may be avoided if it is determined that bi-directional prediction is to be applied to inter prediction of the block.

Description

Technical Field

[0001] The present application is related to video coding and compression. More specifically, the present application relates to systems and methods for performing improved local illumination compensation. Background Art

[0002] Many devices and systems allow video data to be processed and output for consumption. Digital video data includes large amounts of data to meet the needs of consumers and video providers. For example, consumers of video data expect the highest quality video with high fidelity, resolution, frame rate, etc. Therefore, the large amounts of video data required to meet these needs place a burden on communication networks and devices that process and store video data.

[0003] Various video codec techniques can be used to compress video data. Video coding is performed according to one or more video codec standards. For example, video codec standards include high-efficiency video coding (HEVC), advanced video coding (AVC), moving picture experts group (MPEG) coding, etc. Video coding usually uses prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that exploit the redundancy present in video images or sequences. An important goal of video coding technology is to compress video data into a form that uses a lower bit rate while avoiding or minimizing the reduction of video quality. With the emergence of evolving video services, coding techniques with higher coding efficiency are needed. Summary of the invention

[0004] Illumination compensation can be used to effectively compensate for illumination changes between one or more pictures. In some embodiments, this document describes techniques and systems for estimating or deriving one or more illumination compensation parameters for a block of a picture based on one or more tools that can be used for inter-frame prediction of the block. For example, illumination compensation can be selectively applied to the block based on whether bidirectional prediction is applied to the inter-frame prediction of the block. In some examples, after bidirectional prediction has been performed on the block, one or more illumination compensation parameters for the block can be derived, and the one or more illumination compensation parameters are based on one or more of the bidirectional prediction values ​​of the block or one or more neighboring blocks of two or more reference blocks used in the bidirectional prediction. In some examples, illumination compensation for the block can be based on whether one or more motion vectors of the block are obtained by modifying one or more motion vectors of one or more other blocks. In some examples, estimating or deriving one or more illumination compensation parameters for the block can also be based on one or more characteristics of the block, such as the size of the block, whether the block includes multiple partitions, etc.

[0005] According to at least one example, a method for decoding video data is provided. The method includes obtaining an encoded video bitstream including video data. The method also includes obtaining a current block of a picture of the video data from the encoded video bitstream. The method also includes determining whether to apply bidirectional prediction to inter-frame prediction of the current block. The method also includes, after determining whether to apply bidirectional prediction to inter-frame prediction of the current block, determining whether to apply illumination compensation to the current block.

[0006] In another example, an apparatus for decoding video data is provided. The apparatus includes a memory and a processor implemented in a circuit. The processor is configured to obtain a coded video bitstream including video data. The processor is further configured to obtain a current block of a picture of the video data from the coded video bitstream. The processor is further configured to determine whether to apply bidirectional prediction to inter-frame prediction of the current block. The processor is further configured to determine whether to apply illumination compensation to the current block after determining whether to apply bidirectional prediction to inter-frame prediction of the current block.

[0007] In another example, a non-transitory computer-readable medium is provided on which instructions are stored, which, when executed by one or more processors, cause the one or more processors to: obtain an encoded video bitstream including video data; obtain a current block of a picture of the video data from the encoded video bitstream; determine whether to apply bidirectional prediction to inter-frame prediction of the current block; and after determining whether to apply bidirectional prediction to inter-frame prediction of the current block, determine whether to apply illumination compensation to the current block.

[0008] In another example, an apparatus for decoding video data is provided. The apparatus includes a component for obtaining an encoded video bitstream including the video data. The apparatus also includes a component for obtaining a current block of a picture of the video data from the encoded video bitstream. The apparatus also includes a component for determining whether to apply bidirectional prediction to inter-frame prediction of the current block. The apparatus also includes a component for determining whether to apply illumination compensation to the current block after determining whether to apply bidirectional prediction to inter-frame prediction of the current block.

[0009] According to at least one example, a method for encoding video data is provided. The method includes obtaining a current block of a picture of video data. The method also includes determining whether to apply bidirectional prediction to inter-frame prediction of the current block. The method also includes, after determining whether to apply bidirectional prediction to inter-frame prediction of the current block, determining whether to apply illumination compensation to the current block. The method also includes generating an encoded video bitstream including at least a portion of the current block.

[0010] In another example, a device for encoding video data is provided. The device includes a memory and a processor implemented in a circuit. The processor is configured to obtain a current block of a picture of the video data. The processor is further configured to determine whether to apply bidirectional prediction to the inter-frame prediction of the current block. The processor is further configured to determine whether to apply illumination compensation to the current block after determining whether to apply bidirectional prediction to the inter-frame prediction of the current block. The processor is further configured to generate a coded video bitstream including at least a portion of the current block.

[0011] In another example, a non-transitory computer-readable medium is provided on which instructions are stored, which, when executed by one or more processors, cause the one or more processors to: obtain a current block of a picture of video data; determine whether to apply bidirectional prediction to inter-frame prediction of the current block; after determining whether to apply bidirectional prediction to inter-frame prediction of the current block, determine whether to apply illumination compensation to the current block; and generate an encoded video bitstream including at least a portion of the current block.

[0012] In another example, an apparatus for encoding video data is provided. The apparatus includes a component for obtaining a current block of a picture of the video data. The apparatus also includes a component for determining whether to apply bidirectional prediction to inter-frame prediction of the current block. The apparatus also includes a component for determining whether to apply illumination compensation to the current block after determining whether to apply bidirectional prediction to inter-frame prediction of the current block. The apparatus also includes a component for generating an encoded video bitstream including at least a portion of the current block.

[0013] In some aspects of the above methods, apparatus, and computer-readable media, determining whether to apply illumination compensation to the current block may be based on whether bi-directional prediction is to be applied to inter-frame prediction of the current block.

[0014] Some aspects of the above methods, apparatus, and computer-readable media also include determining that bi-prediction is to be applied to inter prediction of the current block; and determining not to apply illumination compensation to the current block based on determining that bi-prediction is to be applied to inter prediction of the current block.

[0015] Some aspects of the above methods, apparatus, and computer-readable media also include determining that bi-prediction will not be applied to inter prediction of the current block; and applying illumination compensation to the current block based on determining that bi-prediction will not be applied to inter prediction of the current block.

[0016] In some aspects of the above methods, apparatus, and computer-readable media, applying illumination compensation to the current block includes deriving one or more illumination compensation parameters of the current block using one or more neighboring blocks of the current block and one or more neighboring blocks of a reference block used for inter-frame prediction of the current block.

[0017] In some aspects of the above methods, apparatuses, and computer-readable media, whether to apply bi-directional prediction to inter-prediction of the current block is determined based on an illumination compensation flag associated with motion information of the current block.

[0018] In some aspects of the above methods, apparatus, and computer-readable media, a value of the illumination compensation flag set to false indicates that bi-directional prediction is to be applied to inter prediction of the current block and illumination compensation is not to be applied to the current block.

[0019] In some aspects of the above methods, apparatus, and computer-readable media, a value of the illumination compensation flag set to true indicates that bi-directional prediction is not to be applied to inter prediction of the current block and illumination compensation is to be applied to the current block.

[0020] Some aspects of the above-mentioned methods, devices and computer-readable media also include obtaining signaling information from an encoded video bitstream; determining whether the signaling information includes an illumination compensation flag associated with the motion information of the current block; and determining whether to apply bidirectional prediction to inter-frame prediction of the current block based on determining whether the signaling information includes an illumination compensation flag associated with the motion information of the current block.

[0021] Some aspects of the above-mentioned methods, apparatuses and computer-readable media also include determining that the signaling information does not include an illumination compensation flag; and based on the signaling information not including the illumination compensation flag, determining that bidirectional prediction will be applied to inter-frame prediction of the current block and illumination compensation will not be applied to the current block.

[0022] Some aspects of the above-mentioned methods, apparatuses and computer-readable media also include applying one of bidirectional prediction of inter-frame prediction to the current block or illumination compensation to the current block; and reconstructing samples of the current block based on the inter-frame prediction applied to the current block or the illumination compensation applied to the current block by the bidirectional prediction.

[0023] In some aspects of the above methods, apparatus, and computer-readable media, reconstructing samples of the current block further comprises applying residual values.

[0024] Some aspects of the above-mentioned methods, apparatuses and computer-readable media also include determining that bidirectional prediction will be applied to inter-frame prediction of the current block; determining that illumination compensation will be applied to the current block; and deriving illumination compensation parameters for the current block using one or more neighboring blocks of the current block and one or more of the following: a first set of one or more neighboring blocks of a first reference block for inter-frame prediction of the current block, a second set of one or more neighboring blocks of a second reference block for inter-frame prediction of the current block, or a combination of the first set of one or more neighboring blocks and the second set of one or more neighboring blocks.

[0025] In some aspects of the above methods, apparatuses, and computer-readable media, the one or more neighboring blocks of the current block include one or more of the top neighboring blocks or the left neighboring blocks of the current block, the first set of one or more neighboring blocks includes one or more of the top neighboring blocks or the left neighboring blocks of the first reference block, and the second set of one or more neighboring blocks includes one or more of the top neighboring blocks or the left neighboring blocks of the second reference block.

[0026] Some aspects of the above-mentioned methods, devices and computer-readable media also include determining that an illumination compensation flag is associated with the motion information of the current block; and determining whether to use the illumination compensation flag for comparing the motion information of the current block with the motion information stored in the motion information candidate list, wherein the motion information of the current block is added to the motion information candidate list based on the comparison.

[0027] Some aspects of the above-mentioned methods, apparatuses and computer-readable media also include determining whether to obtain one or more motion vectors of the current block by modifying one or more motion vectors of one or more other blocks; and determining whether to apply illumination compensation to the current block based on whether the one or more motion vectors of the current block are obtained by modifying one or more motion vectors of one or more other blocks.

[0028] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings and each claim.

[0029] The above and other features and embodiments will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Exemplary embodiments of the present application are described in detail below with reference to the following drawings:

[0031] Figure 1 is a block diagram illustrating examples of encoding devices and decoding devices according to some examples;

[0032] Figure 2A is a conceptual diagram illustrating example spatial neighboring motion vector candidates for merge mode according to some examples;

[0033] Figure 2B is a conceptual diagram illustrating example spatial neighboring motion vector candidates for an Advanced Motion Vector Prediction (AMVP) mode according to some examples;

[0034] Figure 3A is a conceptual diagram illustrating example temporal motion vector predictor (TMVP) candidates according to some examples;

[0035] Figure 3B is a conceptual diagram illustrating examples of motion vector scaling according to some examples;

[0036] Figure 4A is a conceptual diagram illustrating an example of neighboring samples of a current codec unit for estimating illumination compensation (IC) parameters of the current codec unit according to some examples;

[0037] Figure 4B is a conceptual diagram showing an example of neighboring samples of a reference block for estimating IC parameters of a current codec unit according to some examples;

[0038] Figure 5A is a conceptual diagram illustrating an example of neighboring samples of a current codec unit for deriving illumination compensation (IC) parameters of the current codec unit according to some examples;

[0039] Figure 5B is a conceptual diagram showing examples of neighboring samples of a reference block for deriving IC parameters of a current codec unit according to some examples;

[0040] Figure 6 is a conceptual diagram showing examples of neighborhood reconstruction samples of a current block and neighborhood samples of a reference block for unidirectional inter prediction according to some examples;

[0041] Figure 7 is a conceptual diagram showing examples of neighborhood reconstruction samples of a current block and neighborhood samples of two reference blocks for bidirectional inter prediction according to some examples;

[0042] Figure 8 is a flow chart illustrating an example of a process for applying illumination compensation to bi-directional prediction without performing illumination compensation during component uni-directional prediction according to some examples;

[0043] Fig. 9 is a flow chart illustrating an example of a process for decoding video data according to some embodiments, including selectively applying illumination compensation based on whether bi-directional prediction is applied;

[0044] Fig.10 is a flow chart illustrating an example of a process for encoding video data according to some embodiments, including selectively applying illumination compensation based on whether bi-directional prediction is applied;

[0045] Fig.11 is a flow chart illustrating another example of a process for selectively applying illumination compensation based on whether motion information obtained for inter prediction has been modified according to some embodiments;

[0046] Figures 12A-12Dis a conceptual diagram illustrating an example of a triangular prediction unit (PU) and neighboring samples of the prediction unit according to some examples;

[0047] Fig.13 is a block diagram illustrating an example video encoding device according to some examples;

[0048] Fig.14 is a block diagram illustrating an example video decoding device according to some examples. DETAILED DESCRIPTION

[0049] Certain aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some of them can be applied in combination, as will be apparent to those skilled in the art. In the following description, for the purpose of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it is apparent that various embodiments can be implemented without these specific details. The drawings and description are not intended to be limiting.

[0050] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. Instead, the following description of the exemplary embodiments will provide an enabling description for implementing the exemplary embodiments for those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the application as described in the appended claims.

[0051] Video codecs implement video compression techniques to efficiently encode and decode video data. Video compression techniques may include applying different prediction modes, including spatial prediction (e.g., intra-frame prediction), temporal prediction (e.g., inter-frame prediction), inter-layer prediction (across different layers of video data, and / or other prediction techniques to reduce or remove redundancy inherent in video sequences. A video encoder may partition each picture of an original video sequence into rectangular regions called video blocks or codec units (described in more detail below). These video blocks may be encoded using specific prediction modes.

[0052] A video block may be divided into one or more groups of smaller blocks in one or more ways. A block may include a codec tree block, a prediction block, a transform block, or other suitable blocks. Unless otherwise specified, references to "blocks" in general may refer to such video blocks (e.g., codec tree blocks, codec blocks, prediction blocks, transform blocks, or other suitable blocks or sub-blocks, as would be understood by a person of ordinary skill). In addition, each of these blocks may also be referred to interchangeably herein as a "unit" (e.g., codec tree unit (CTU), codec unit, prediction unit (PU), transform unit (TU), etc.). In some cases, a unit may refer to a codec logic unit encoded in a bitstream, and a block may refer to a portion of a target video frame buffer for which a process is directed.

[0053] For inter prediction mode, the video encoder can search for a block similar to the block being encoded in a frame (or picture) located in another temporal location, which is called a reference frame or reference picture. The video encoder can limit the search to a specific spatial displacement from the block to be encoded. A two-dimensional (2D) motion vector including a horizontal displacement component and a vertical displacement component can be used to locate the best match. For intra prediction mode, the video encoder can form a prediction block using spatial prediction techniques based on data from previously encoded adjacent blocks in the same picture.

[0054] The video encoder may determine a prediction error. For example, the prediction may be determined as the difference between the pixel values ​​in the coded block and the prediction block. The prediction error may also be referred to as a residual. The video encoder may also apply a transform (e.g., a discrete cosine transform (DCT) or other suitable transform) to the prediction error to generate transform coefficients. After the transform, the video encoder may quantize the transform coefficients. The quantized transform coefficients and motion vectors may be represented using syntax elements and, together with the control information, form a codec representation of the video sequence. In some instances, the video encoder may entropy encode and decode the syntax elements to further reduce the number of bits required for their representation.

[0055] The video decoder can use the syntax elements and control information discussed above to construct prediction data (e.g., a prediction block) for decoding the current frame. For example, the video decoder can add the prediction block and the compressed prediction error. The video decoder can determine the compressed prediction error by weighting the transform basis function using the quantization coefficients. The difference between the reconstructed frame and the original frame is called the reconstruction error.

[0056] In some examples, one or more systems and methods for processing video data are introduced for deriving or estimating illumination compensation (IC) parameters in block-based video codecs. In some instances, a video encoder and / or a video decoder may perform local illumination compensation (LIC) (or illumination compensation) to effectively encode and decode illumination (e.g., brightness) changes between one or more pictures. A video encoder and / or a video decoder may determine one or more IC parameters (e.g., offsets, one or more scaling factors, a number of shifts, or other suitable IC parameters) for a codec block or codec unit being encoded or decoded. The IC parameters may be determined based on samples of multiple reference blocks, samples of one or more neighboring blocks of a current block, and / or other information. A video decoder may utilize the IC parameters and / or other data to construct prediction data for decoding a current block.

[0057] The techniques described herein can be applied to any existing video codec (e.g., High Efficiency Video Codec (HEVC), Advanced Video Codec (AVC), or other suitable existing video codecs), or can be an effective codec tool for any future video codec standard, such as the Joint Exploration Model (JEM) and / or other video codec standards that are being developed or to be developed.

[0058] Figure 1 1 is a block diagram showing an example of a system 100 including an encoding device 104 and a decoding device 112. The encoding device 104 may be part of a source device, and the decoding device 112 may be part of a receiving device. The source device and / or the receiving device may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the source device and the receiving device may include one or more wireless transceivers for wireless communication. The coding and decoding techniques described herein are applicable to video coding and decoding in various multimedia applications, including streaming video transmission (e.g., via the Internet), television broadcasting or transmission, encoding of digital videos stored on data storage media, decoding of digital videos stored on data storage media, or other applications. In some examples, the system 100 may support one-way or two-way video transmission to support applications such as video conferencing, video streaming, video playback, video broadcasting, games, and / or video telephony.

[0059] The encoding device 104 (or encoder) can be used to encode video data using a video codec standard or protocol to generate an encoded video bitstream. Examples of video codec standards include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual, ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC), including its scalable video codec (SVC) and multi-view video codec (MVC) extensions, and high efficiency video codec (HEVC) or ITU-T H.265. There are various extensions of HEVC for multi-layer video codecs, including range and screen content codec extensions, three-dimensional video codec (3D-HEVC) and multi-view extensions (MV-HEVC) and scalable extensions (SHVC). HEVC and its extensions were developed by the Joint Collaborative Team on Video Codecs (JCT-VC) and the Joint Collaborative Team on 3D Video Codec Extension Development (JCT-3V) of the ITU-T Video Codec Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). MPEG and ITU-T VCEG also formed the Joint Exploration Video Team (JVET) to explore new codec tools for the next generation of video codec standards. The reference software is called JEM (Joint Exploration Model). A new video codec standard being developed by JVET is called Versatile Video Codec (VVC).

[0060] Many of the embodiments described herein provide examples using the JEM model, the HEVC standard, and / or its extensions. However, the techniques and systems described herein may also be applicable to other codec standards, such as AVC, MPEG, JPEG (or other codec standards for still images), their extensions, or other suitable codec standards that are already available or not yet available or have been developed. Therefore, although the techniques and systems described herein may be described with reference to a specific video codec standard, a person of ordinary skill in the art will understand that the description should not be interpreted as being applicable only to that specific standard.

[0061] refer to Figure 1 , video source 102 may provide video data to encoding device 104. Video source 102 may be part of a source device, or may be part of a device other than a source device. Video source 102 may include a video capture device (e.g., a video camera, a camera phone, a video phone, etc.), a video archive containing stored video, a video server or content provider that provides video data, a video feed interface that receives video from a video server or content provider, a computer graphics system for generating computer graphics video data, a combination of these sources, or any other suitable video source.

[0062] The video data from the video source 102 may include one or more input pictures or frames. A picture or frame is a still image, which in some cases is part of a video. In some examples, the data from the video source 102 may be a still image that is not part of a video. The encoder engine 106 (or encoder) of the encoding device 104 encodes the video data to generate a coded video bitstream. In some examples, the coded video bitstream (or "video bitstream" or "bitstream") is a series of one or more codec video sequences. The codec video sequence (CVS) includes a series of access units (AUs), starting from an AU with a random access point picture in the base layer and having a specific attribute, until and not including the next AU with a random access point picture in the base layer and having a specific attribute. For example, the specific attribute of the random access point picture that starts the CVS may include a RASL flag equal to 1 (e.g., NoRaslOutputFlag). Otherwise, the random access point picture (RASL flag equal to 0) will not start the CVS. The access unit (AU) includes one or more codec pictures and control information corresponding to the codec pictures sharing the same output time. The codec slices of a picture are encapsulated into data units called network abstraction layer (NAL) units at the bitstream level. For example, a HEVC video bitstream may include one or more CVSs including NAL units. Each NAL unit has a NAL unit header. In one example, the header is one byte for H.264 / AVC (except for multi-layer extensions) and two bytes for HEVC. The syntax elements in the NAL unit header use designated bits and are therefore visible to all types of systems and transport layers, such as transport streams, real-time transport (RTP) protocols, file formats, etc.

[0063] There are two types of NAL units in the HEVC standard, including video codec layer (VCL) NAL units and non-VCL NAL units. The VCL NAL unit includes a slice or slice fragment of codec picture data (described below), and the non-VCL NAL unit includes control information related to one or more codec pictures. In some cases, the NAL unit can be called a packet. The HEVC AU includes a VCL NAL unit containing codec picture data and a non-VCL NAL unit corresponding to the codec picture data (if any).

[0064] The NAL unit may contain a sequence of bits that form a codec representation of video data (e.g., a coded video bitstream, a CVS of a bitstream, etc.), such as a codec representation of a picture in a video. The encoder engine 106 generates a codec representation of a picture by partitioning each picture into a plurality of slices. The slices are independent of other slices, so the information in the slices does not depend on data from other slices in the same picture during encoding and decoding. The slices include one or more slice segments, including independent slice segments and one or more dependent slice segments that depend on the previous slice segments (if any). The slices are then partitioned into codec tree blocks (CTBs) of luminance samples and chrominance samples. The CTB of luminance samples and one or more CTBs of chrominance samples, together with the syntax of the samples, are referred to as codec tree units (CTUs). CTUs are the basic processing units for HEVC encoding. CTUs can be divided into multiple codec units (CUs) of different sizes. CUs contain arrays of luminance and chrominance samples called codec blocks (CBs).

[0065] Luma and chroma CBs can be further divided into prediction blocks (PBs). A PB is a block of samples of a luminance component or a chroma component that uses the same motion parameters for inter-frame prediction or intra-frame block copy prediction (when available or enabled). A luminance PB and one or more chroma PBs together with associated syntax form a prediction unit (PU). For inter-frame prediction, a set of motion parameters (e.g., one or more motion vectors, reference indices, etc.) is signaled for each PU in the bitstream and used for inter-frame prediction of the luminance PB and one or more chroma PBs. Motion parameters may also be referred to as motion information. A CB may also be partitioned into one or more transform blocks (TBs). A TB represents a square block of samples of a color component to which the same two-dimensional transform is applied to encode and decode a prediction residual signal. A transform unit (TU) represents a TB of luminance and chroma samples and corresponding syntax elements.

[0066] The size of the CU corresponds to the size of the codec mode and can be a square. For example, the size of the CU can be 8×8 samples, 16×16 samples, 32×32 samples, 64×64 samples, or any other appropriate size up to the size of the corresponding CTU. The phrase "N×N" is used herein to refer to the pixel size of the video block in the vertical and horizontal dimensions (e.g., 8 pixels×8 pixels). The pixels in the block can be arranged in rows and columns. In some embodiments, the block may not have the same number of pixels in the horizontal direction as in the vertical direction. For example, the syntax data associated with the CU may describe the partitioning of the CU into one or more PUs. The partitioning mode may be different between whether the CU is encoded in an intra-prediction mode or an inter-prediction mode. The PU may be partitioned into non-squares. For example, the syntax data associated with the CU may also describe the partitioning of the CU into one or more TUs according to the CTU. The shape of the TU may be square or non-square.

[0067] According to the HEVC standard, a transform unit (TU) may be used to perform a transform. A TU may be different for different CUs. The size of a TU may be determined based on the size of a PU within a given CU. A TU may be the same size or smaller than a PU. In some examples, a quadtree structure called a residual quadtree (RQT) may be used to subdivide the residual samples corresponding to a CU into smaller units. The leaf nodes of the RQT may correspond to TUs. The pixel difference values ​​associated with the TU may be transformed to generate transform coefficients. The transform coefficients may then be quantized by the encoder engine 106.

[0068] Once the picture of the video data is partitioned into CUs, the encoder engine 106 predicts each PU using a prediction mode. The prediction unit or prediction block is then subtracted from the original video data to obtain a residual (as described below). For each CU, the prediction mode can be signaled in the bitstream using syntax data. The prediction mode may include intra-frame prediction (or intra-picture prediction) or inter-frame prediction (or inter-picture prediction). Intra-frame prediction exploits the correlation between spatially adjacent samples in a picture. For example, using intra-frame prediction, each PU is predicted from adjacent image data in the same picture, for example, using DC prediction to find the average value of the PU, using plane prediction to fit a plane surface to the PU, using directional prediction to extrapolate from adjacent data, or using any other suitable type of prediction. Inter-frame prediction uses temporal correlation between pictures to derive motion compensated predictions for blocks of image samples. For example, using inter-frame prediction, each PU is predicted using motion compensated predictions from image data in one or more reference pictures (before or after the current picture in output order). For example, a decision can be made at the CU level whether to use inter-picture prediction or intra-picture prediction to encode or decode a picture area.

[0069] In some examples, one or more slices of a picture are assigned a slice type. Slice types include I slices, P slices, and B slices. I slices (intra, independently decodable) are slices of a picture that are encoded and decoded only by intra prediction, and are therefore independently decodable because an I slice only requires data within the frame to predict any prediction unit or prediction block of the slice. P slices (unidirectional predicted frames) are slices of a picture that can be encoded and decoded using intra prediction and unidirectional inter prediction. Each prediction unit or prediction block within a P slice is encoded and decoded using intra prediction or inter prediction. When inter prediction is applied, the prediction unit or prediction block is predicted by only one reference picture, and therefore the reference samples come from only one reference area of ​​a frame. B slices (bidirectional predicted frames) are slices of a picture that can be encoded and decoded using intra prediction and inter prediction (e.g., bidirectional prediction or unidirectional prediction). The prediction unit or prediction block of a B slice can be bidirectionally predicted from two reference pictures, where each picture contributes one reference area and the sample sets of the two reference areas are weighted (e.g., with equal weights or with different weights) to generate a prediction signal for the bidirectional prediction block. As described above, the slices of a picture are independently coded. In some cases, a picture can be coded as only one slice.

[0070] The PU may include data related to the prediction process (e.g., motion parameters or other suitable data). For example, when the PU is encoded using intra prediction, the PU may include data describing the intra prediction mode for the PU. As another example, when the PU is encoded using inter prediction, the PU may include data defining a motion vector for the PU. The data defining the motion vector for the PU may describe, for example, the horizontal component of the motion vector (Δx), the vertical component of the motion vector (Δy), the resolution of the motion vector (e.g., integer precision, quarter-pixel precision, or eighth-pixel precision), the reference picture to which the motion vector points, the reference index of the motion vector, a reference picture list (e.g., list 0, list 1, or list C), or any combination thereof.

[0071] The encoding device 104 may then perform transformation and quantization. For example, after prediction, the encoder engine 106 may calculate a residual value corresponding to the PU. The residual value may include a pixel difference value between a current block (PU) of pixels being encoded and decoded and a prediction block (e.g., a predicted version of the current block) used to predict the current block. For example, after generating a prediction block (e.g., issuing an inter-frame prediction or an intra-frame prediction), the encoder engine 106 may generate a residual block by subtracting the prediction block generated by the prediction unit from the current block. The residual block includes a set of pixel difference values ​​that quantize the difference between the pixel values ​​of the current block and the pixel values ​​of the prediction block. In some examples, the residual block may be represented in a two-dimensional block format (e.g., a two-dimensional matrix or a pixel value array). In such an example, the residual block is a two-dimensional representation of a pixel value.

[0072] Any residual data that may remain after performing the prediction is transformed using a block transform, which may be based on a discrete cosine transform, a discrete sine transform, an integer transform, a wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., of size 32×32, 16×16, 8×8, 4×4, or other suitable sizes) may be applied to the residual data in each CU. In some embodiments, a TU may be used for a transform and quantization process implemented by the encoder engine 106. A given CU with one or more PUs may also include one or more TUs. As described in further detail below, a block transform may be used to transform residual values ​​into transform coefficients, which may then be quantized and scanned using the TUs to produce serialized transform coefficients for entropy coding and decoding.

[0073] In some embodiments after encoding and decoding using the PU of the CU for intra prediction or inter prediction, the encoder engine 106 may calculate the residual data of the TU of the CU. The PU may include pixel data in the spatial domain (or pixel domain). After applying the block transform, the TU may include coefficients in the transform domain. As described above, the residual data may correspond to the pixel difference between the pixel of the uncoded picture and the predicted value corresponding to the PU. The encoder engine 106 may form a TU including the residual data of the CU, and then the TU may be transformed to generate the transform coefficients of the CU.

[0074] The encoder engine 106 may perform quantization of the transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all coefficients. In one example, a coefficient having an n-bit value may be rounded down to an m-bit value during quantization, where n is greater than m.

[0075] Once quantization is performed, the coded video bitstream includes quantized transform coefficients, prediction information (e.g., prediction mode, motion vectors, block vectors, etc.), segmentation information, and any other suitable data, such as other syntax data. The different elements of the coded video bitstream can then be entropy encoded by the encoder engine 106. In some examples, the encoder engine 106 can scan the quantized transform coefficients using a predefined scan order to produce a serialized vector that can be entropy encoded. In some examples, the encoder engine 106 can perform adaptive scanning. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), the encoder engine 106 can entropy encode the vector. For example, the encoder engine 106 can use context-adaptive variable length coding, context-adaptive binary arithmetic coding, grammar-based context-adaptive binary arithmetic coding, probability interval partitioning entropy coding, or another suitable entropy coding technology.

[0076] As described above, the HEVC bitstream includes a NAL unit group, including a VCL NAL unit and a non-VCL NAL unit. The VCL NAL unit includes coded picture data that forms a coded video bitstream. For example, there is a bit sequence that forms a coded video bitstream in the VCL NAL unit. In addition to other information, the non-VCL NAL unit may also contain a parameter set with high-level information related to the coded video bitstream. For example, a parameter set may include a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS). Examples of the goals of the parameter set include bit rate efficiency, error resilience, and providing a system layer interface. Each slice references a single active PPS, SPS, and VPS to access information that the decoding device 112 can use to decode the slice. A codec identifier (ID) may be provided for each parameter set, including a VPS ID, an SPS ID, and a PPS ID. The SPS includes an SPS ID and a VPS ID. The PPS includes a PPS ID and an SPS ID. Each slice header includes a PPS ID. Using the ID, an active parameter set may be identified for a given slice.

[0077] The PPS includes information applied to all slices in a given picture. Therefore, all slices in a picture reference the same PP. Slices in different pictures may also reference the same PPS. The SPS includes information applied to all pictures in the same coded video sequence (CVS) or bitstream. As previously described, a coded video sequence is a series of access units (AUs) that start with a random access point picture in the base layer (e.g., an instantaneous decoding reference (IDR) picture or a broken link access (BLA) picture, or other appropriate random access point pictures) and have certain properties (as described above) until and not including the next AU with a random access point picture in the base layer and having certain properties (or until the end of the bitstream). In a coded video sequence, the information in the SPS cannot be changed between pictures. Pictures in a coded video sequence may use the same SPS. The VPS includes information applied to all layers within a coded video sequence or bitstream. The VPS includes a syntax structure having syntax elements applied to the entire coded video sequence. In some embodiments, the VPS, SPS, or PPS may be sent in-band with the coded bitstream. In some embodiments, the VPS, SPS, or PPS may be sent out-of-band in a separate transmission from the NAL units containing the encoded video data.

[0078] The video bitstream may also include a supplemental enhancement information (SEI) message. For example, a SEI NAL unit may be part of a video bitstream. In some cases, an SEI message may contain information that is not required for the decoding process. For example, the information in the SEI message may not be necessary for a decoder to decode a video picture of the bitstream, but the decoder may use the information to improve the display or processing of the picture (e.g., decoded output). The information in the SEI message may be embedded in metadata. In an illustrative example, the information in the SEI message may be used by a decoder-side entity to improve the visibility of the content. In some instances, certain application standards may mandate the presence of such SEI messages in the bitstream so that quality improvements can be brought to all devices that comply with the application standard (e.g., carrying a frame packing SEI message for a frame-compatible planar stereoscopic 3DTV video format, carrying an SEI message for each frame of the video, processing a recovery point SEI message, using a pan scan rectangle SEI message in DVB, and many other examples).

[0079] The output 110 of the encoding device 104 may send the NAL units constituting the encoded video bitstream data to the decoding device 112 of the receiving device via a communication link 120. The input 114 of the decoding device 112 may receive the NAL units. The communication link 120 may include a channel provided by a wireless network, a wired network, or a combination of wired and wireless networks. The wireless network may include any wireless interface or combination of wireless interfaces and may include any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, a WiFi, or a wireless network). TM , Radio Frequency (RF), UWB, WiFi-Direct, Cellular, Long Term Evolution (LTE), WiMax TM The wired network may include any wired interface (e.g., optical fiber, Ethernet, power line Ethernet, coaxial cable Ethernet, digital signal line (DSL), etc.). The wired and / or wireless network may be implemented using various equipment, such as base stations, routers, access points, bridges, gateways, switches, etc. The encoded video bitstream data may be modulated according to a communication standard such as a wireless communication protocol and sent to a receiving device.

[0080] In some examples, encoding device 104 may store the encoded video bitstream data in storage device 108. Output 110 may retrieve the encoded video bitstream data from encoder engine 106 or from storage device 108. Storage device 108 may include any of a variety of distributed or locally accessed data storage media. For example, storage device 108 may include a hard drive, a storage disk, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0081] The input 114 of the decoding device 112 receives the encoded video bitstream data and can provide the video bitstream data to the decoder engine 116, or to the storage device 118 for later use by the decoder engine 116. The decoder engine 116 can decode the encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting elements of one or more coded video sequences that constitute the encoded video data. The decoder engine 116 can then rescale the encoded video bitstream data and perform an inverse transform. The residual data is then passed to the prediction stage of the decoder engine 116. The decoder engine 116 then predicts a pixel block (e.g., PU). In some examples, the prediction is added to the output (residual data) of the inverse transform.

[0082] The decoding device 112 may output the decoded video to a video destination device 122, which may include a display or other output device for displaying the decoded video data to a consumer of the content. In some aspects, the video destination device 122 may be part of a receiving device that includes the decoding device 112. In some aspects, the video destination device 122 may be part of a separate device other than the receiving device.

[0083] In some embodiments, the video encoding device 104 and / or the video decoding device 112 can be integrated with the audio encoding device and the audio decoding device, respectively. The video encoding device 104 and / or the video decoding device 112 may also include other hardware or software necessary to implement the above-mentioned coding and decoding techniques, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. The video encoding device 104 and the video decoding device 112 can be integrated as part of a combined encoder / decoder (codec) in each device. Refer to the following Fig.13 An example of specific details of the encoding device 104 is described below. Fig.14 An example describing specific details of the decoding device 112.

[0084] Extensions to the HEVC standard include a multi-view video codec extension (called MV-HEVC) and a scalable video codec extension (called SHVC). The MV-HEVC and SHVC extensions share the concept of layered coding, including different layers in the coded video bitstream. Each layer in the coded video sequence is addressed by a unique layer identifier (ID). The layer ID may be present in the header of the NAL unit to identify the layer associated with the NAL unit. In MV-HEVC, different layers may represent different views of the same scene in the video bitstream. In SHVC, different scalable layers are provided to represent the video bitstream with different spatial resolutions (or picture resolutions) or different reconstruction fidelity. The scalable layer may include a base layer (layer ID = 0) and one or more enhancement layers (layer ID = 1, 2, ... n). The base layer may conform to the profile of the first version of HEVC and represent the lowest available layer in the bitstream. Compared to the base layer, the enhancement layer has increased spatial resolution, temporal resolution or frame rate and / or reconstruction fidelity (or quality). Enhancement layers are organized in layers and may (or may not) depend on lower layers. In some examples, different layers may be encoded using a single standard codec (e.g., all layers are encoded using HEVC, SHVC, or other codec standards). In some examples, different layers may be encoded using multiple standard codecs. For example, the base layer may be encoded using AVC, while one or more enhancement layers may be encoded using SHVC and / or MV-HEVC extensions of the HEVC standard.

[0085] Typically, a layer consists of a set of VCL NAL units and a corresponding set of non-VCL NAL units. NAL units are assigned specific layer ID values. Layers can be hierarchical in the sense that layers can depend on lower layers. A layer set refers to a set of self-contained layers represented in a bitstream, which means that during the decoding process, layers in a layer set can depend on other layers in the layer set, but do not depend on any other layers for decoding. Therefore, the layers in a layer set can form an independent bitstream capable of representing video content. The set of layers in a layer set can be obtained from another bitstream through the operation of a sub-bitstream extraction process. A layer set may correspond to a set of layers to be decoded when a decoder wants to operate according to certain parameters.

[0086] As described above, for each block, a set of motion information (also referred to herein as motion parameters) may be available. The motion information set contains motion information for forward and backward prediction directions. The forward and backward prediction directions are the two prediction directions of the bidirectional prediction mode, in which case the terms "forward" and "backward" do not necessarily have a geometric meaning. Instead, "forward" and "backward" correspond to reference picture list 0 (RefPicList0 or L0) and reference picture list 1 (RefPicList1 or L1) of the current picture. In some examples, when only one reference picture list is available for a picture or slice, only RefPicList0 is available, and the motion information for each block of the slice is always forward.

[0087] In some cases, a motion vector and its reference index are used together in a coding process (e.g., motion compensation). Such a motion vector with an associated reference index is represented as a single prediction set of motion information. For each prediction direction, the motion information may contain a reference index and a motion vector. In some cases, for simplicity, the motion vector itself may be referenced in a manner that assumes that the motion vector has an associated reference index. The reference index is used to identify a reference picture in the current reference picture list (RefPicList0 or RefPicList1). The motion vector has horizontal and vertical components that provide an offset from a coordinate position in the current picture to a coordinate in the reference picture identified by the reference index. For example, a reference index may indicate a specific reference picture that should be used for a block in the current picture, and a motion vector may indicate the position of the best matching block (the block that best matches the current block) in the reference picture.

[0088] The picture order count (POC) can be used in video codec standards to identify the display order of pictures. Although there are cases where two pictures in a codec video sequence may have the same POC value, it does not usually happen in a codec video sequence. When there are multiple codec video sequences in the bitstream, pictures with the same POC value can be closer to each other in decoding order. The POC value of a picture can be used for reference picture list construction, derivation of reference picture sets in HEVC, and motion vector scaling.

[0089] In H.264 / AVC, each inter-frame macroblock (MB) can be partitioned in four different ways, including: one 16×16MB partition; two 16×8MB partitions; two 8×16MB partitions; and four 8×8MB partitions. Different MB partitions in one MB can have different reference index values ​​for each direction (RefPicList0 or RefPicList1). In some cases, when the MB is not partitioned into four 8×8MB partitions, there can only be one motion vector in each direction for each MB partition. In some cases, when the MB is partitioned into four 8×8MB partitions, each 8x8 MB partition can be further partitioned into sub-blocks, in which case each sub-block can have a different motion vector in each direction. In some examples, there are four different ways to get a sub-block from an 8×8MB partition, including: one 8×8 sub-block; two 8×4 sub-blocks; two 4×8 sub-blocks; and four 4×4 sub-blocks. Each sub-block can have a different motion vector in each direction. Therefore, the motion vector exists at a level equal to or higher than the sub-block.

[0090] In AVC, for skip and / or direct modes in B slices, temporal direct mode can be enabled at the MB level or MB partition level. For each MB partition, the motion vectors of the blocks co-located with the current MB partition in RefPicList1[0] of the current block are used to derive the motion vectors. Each motion vector in the co-located blocks is scaled based on the POC distance.

[0091] Spatial direct mode can also be implemented in AVC. For example, in AVC, direct mode can also predict motion information from spatial neighbors.

[0092] In HEVC, the largest codec unit in a slice is called a codec tree block (CTB). The CTB contains a quadtree whose nodes are codec units. In the HEVC Main Profile, the size of the CTB can range from 16×16 to 64×64. In some cases, an 8x8 CTB size can be supported. The codec unit (CU) can be the same size as the CTB, as small as 8×8. In some cases, each codec unit is coded and decoded in one mode. When the CU is inter-coded, the CU can be further split into 2 or 4 prediction units (PUs), or when no further splitting is applied, the CU can be just a PU. When there are two PUs in a CU, they can be half-sized rectangles, or 1 / 4 or 3 / 4 the size of the two CUs.

[0093] When a CU is inter-coded, there is a motion information set for each PU. In addition, each PU is coded and decoded in a unique inter prediction mode to derive a motion information set.

[0094] For motion prediction in HEVC, there are two inter prediction modes including merge mode and advanced motion vector prediction (AMVP) mode for prediction units (PUs). Skip is considered a special case of merge. In AMVP or merge mode, a motion vector (MV) candidate list is maintained for multiple motion vector predictors. The motion vector(s) of the current PU and the reference index in merge mode are generated by selecting a candidate from the MV candidate list.

[0095] In some examples, for merge mode, the MV candidate list contains up to five candidates, and for AMVP mode, the MV candidate list contains up to two candidates. In other examples, for merge mode and / or AMVP mode, different numbers of candidates may be included in the MV candidate list. The merge candidate may include a motion information set. For example, the motion information set may include motion vectors corresponding to both a reference picture list (list 0 and list 1) and a reference index. If the merge candidate is identified by the merge index, the reference picture is used for prediction of the current block, and the associated motion vector is determined. However, in AMVP mode, for each potential prediction direction from list 0 or list 1, the reference index as well as the MVP index of the MV candidate list needs to be explicitly signaled, because the AMVP candidate only contains motion vectors. In AMVP mode, the predicted motion vector can be further refined.

[0096] As mentioned above, the merge candidate corresponds to a complete set of motion information, while the AMVP candidate contains only one motion vector for a specific prediction direction and reference index. Candidates for both modes are similarly derived from the same spatial and temporal neighboring blocks.

[0097] In some examples, the merge mode allows an inter-predicted PU to inherit the same one or more motion vectors, prediction direction, and one or more reference picture indexes from an inter-predicted PU, which includes a motion data position selected from a group of spatially adjacent motion data positions and one of two temporally juxtaposed motion data positions. For the AMVP mode, one or more motion vectors of the PU can be predictively encoded and decoded relative to one or more motion vector prediction values ​​(MVPs) from an AMVP candidate list constructed by the encoder. In some instances, for unidirectional inter prediction of a PU, the encoder can generate a single AMVP candidate list. In some instances, for bidirectional prediction of a PU, the encoder can generate two AMVP candidate lists, one using motion data from spatially and temporally adjacent PUs in the forward prediction direction, and one using motion data from spatially and temporally adjacent PUs in the backward prediction direction.

[0098] Candidates for both modes can be derived from neighboring blocks in the spatial and / or temporal domains. Figure 2A and Figure 2B Included is a conceptual diagram illustrating spatial neighbor candidates in HEVC. Figure 2A Spatially neighboring motion vector (MV) candidates for merge mode are shown. Figure 2B Spatial neighboring motion vector (MV) candidates for AMVP mode are shown. Spatial MV candidates are derived from neighboring blocks of a particular PU (PU0), although the method of generating candidates from blocks is different for merge and AMVP modes.

[0099] In merge mode, the encoder can form a merge candidate list by considering merge candidates from different motion data locations. Figure 2A As shown, about Figure 2A A maximum of four spatial MV candidates can be derived from the spatial adjacent motion data positions indicated by numbers 0-4 in the merge candidate list. The MV candidates can be sorted in the merge candidate list in the order indicated by numbers 0-4. For example, the positions and order may include: left position (0), top position (1), top right position (2), bottom left position (3), and top left position (4).

[0100] exist Figure 2B In the AVMP mode shown, the neighboring blocks are divided into two groups: the left group includes blocks 0 and 1, and the upper group includes blocks 2, 3 and 4. For each group, potential candidates in the neighboring blocks that refer to the same reference picture as the reference picture indicated by the signaled reference index have the highest priority to be selected to form the final candidate for the group. It is possible that all neighboring blocks do not contain motion vectors pointing to the same reference picture. Therefore, if such a candidate cannot be found, the first available candidate will be scaled to form the final candidate, so that the temporal distance difference can be compensated.

[0101] Figure 3A and Figure 3B Included is a conceptual diagram illustrating temporal motion vector prediction in HEVC. Temporal motion vector predictor (TMVP) candidates, if enabled and available, are added to the MV candidate list after spatial motion vector candidates. The motion vector derivation process for TMVP candidates is the same for both merge and AMVP modes. However, in some instances, the target reference index of the TMVP candidate in merge mode can be set to zero or can be derived from the target reference index of a neighboring block.

[0102] The main block position for TMVP candidate derivation is the lower right block outside the collocated PU, such as Figure 3A The motion vector of the TMVP candidate is shown as block "T" in the figure to compensate for the deviation of the upper and left blocks used to generate the spatial neighboring candidates. However, if the block is outside the current CTB (or LCU) row or the motion information is not available, the block is replaced with the center block of the PU. The motion vector of the TMVP candidate is derived from the collocated PU of the collocated picture indicated in the slice level. Similar to the temporal direct mode in AVC, the motion vector of the TMVP candidate can be subjected to motion vector scaling, which is performed to compensate for the distance difference.

[0103] Other aspects of motion prediction are covered in the HEVC standard. For example, several other aspects of merge and AMVP modes are covered. One aspect includes motion vector scaling. With respect to motion vector scaling, it can be assumed that the value of a motion vector is proportional to the distance of the picture in presentation time. A motion vector relates two pictures—a reference picture and a picture containing the motion vector (i.e., a containing picture). When a motion vector is used to predict another motion vector, the distance between the containing picture and the reference picture is calculated based on the picture order count (POC) value.

[0104] For a motion vector to be predicted, its associated containing picture and reference picture can be different. Therefore, a new distance is calculated (based on POC). And the motion vector is scaled based on these two POC distances. For spatial neighbor candidates, the containing pictures of the two motion vectors are the same, but the reference pictures are different. In HEVC, motion vector scaling is applied to both spatial and temporal neighbor candidates for both TMVP and AMVP.

[0105] Another aspect of motion prediction includes artificial motion vector candidate generation. For example, if the motion vector candidate list is incomplete, artificial motion vector candidates are generated and inserted at the end of the list until all candidates are obtained. In merge mode, there are two types of artificial MV candidates: combined candidates derived only for B slices; and zero candidates used only for AMVP if the first type does not provide enough artificial candidates. For each pair of candidates that are already in the candidate list and have the necessary motion information, a bidirectional combined motion vector candidate is derived by combining the motion vector of the first candidate of the picture in reference list 0 and the motion vector of the second candidate of the picture in reference list 1.

[0106] Another aspect of the merge and AMVP modes includes a pruning process for candidate insertion. For example, candidates from different blocks may happen to be identical, which reduces the efficiency of the merge and / or AMVP candidate list. A pruning process can be applied to solve this problem. The pruning process compares one candidate in the current candidate list with other candidates to avoid inserting identical candidates to a certain extent. To reduce complexity, only a limited number of pruning processes are applied instead of comparing each potential candidate with all other existing candidates.

[0107] There are various related motion prediction techniques. One prediction technique is local illumination compensation (LIC) or luminance compensation. Illumination compensation for HEVC has been proposed. For example, in JCTVC-C041, partition-based illumination compensation (PBIC) is proposed. Unlike weighted prediction (WP), where WP is enabled and / or disabled at the slice level and WP parameters are signaled (described below), PBIC enables and / or disables illumination compensation (IC) at the prediction unit (PU) level and IC parameters are signaled to handle local illumination changes. In JVET-B0023, block-based LIC is extended to CU, similar to PU in HEVC, and CU becomes the basic unit carrying motion information in the QTBT structure.

[0108] Similar to weighted prediction (WP) described in more detail below, a scaling factor (also denoted by a) and an offset (also denoted by b) are used in IC, and the number of shifts is fixed to 6. An IC flag is encoded and decoded for each PU to indicate whether IC is applied to the current PU. If IC is applied to the PU, a set of IC parameters (e.g., a and b) are signaled to the decoder and used for motion compensation. In some examples, in order to save bits spent on IC parameters, the chroma components share the scaling factors with the luma components, and a fixed offset of 128 is used.

[0109] In 3D-HEVC, IC is enabled for inter-view prediction. Unlike WP and PBIC, which explicitly signal IC parameters, it derives IC parameters based on neighboring samples of the current CU and neighboring samples of the reference block. IC is only applicable to 2N×2N partition mode. For AMVP mode, an IC flag is signaled for each CU predicted from inter-view reference pictures. For merge mode, in order to save bits, the IC flag is signaled only when the merge index of the PU is not equal to 0. In some cases, IC is not applicable to CUs predicted only from time domain reference pictures.

[0110] Regarding the derivation of IC parameters, the linear IC model used in inter-view prediction is shown in formula (1):

[0111] p(i,j)=a*r(i+dv x ,j+dv y )+b, where (i,j)∈PU c Formula (1)

[0112] Here, PU c is the current PU, (i, j) is the PU c The coordinates of the pixel in (dv x , dv y ) is PU c The disparity vector of . p(i, j) is the PU c , r is the PU’s reference picture from the adjacent view, and a and b are the parameters of the linear IC model.

[0113] To estimate the parameters a and b of the PU, two sets of pixels are used, such as Figure 4A and Figure 4B The first pixel set is as shown. Figure 4A As shown, the reconstructed neighboring pixels available in the left column and top row of the current CU (the CU containing the current PU) are included. The second pixel set is as follows Figure 4B As shown, the corresponding neighboring pixels of the reference block of the current CU are included. The reference block of the current CU is found by using the disparity vector of the current PU.

[0114] Let Rec neig and Rec refneig Respectively represent the used adjacent pixel sets of the current CU and its reference block, and let 2N represent Rec neig and Rec refneig Then, a and b can be calculated as:

[0115]

[0116]

[0117] In some cases, only a is used in the linear model and b is always set to 0. In some cases, only b is used and a is always set to 1.

[0118] In HEVC, weighted prediction (WP) is supported, in which case a scaling factor (represented by a), a shift number (represented by s), and an offset (represented by b) are used in motion compensation. Assuming that the pixel value at the position (x, y) of the reference picture is p(x, y), p'(x, y) = ((a*p(x, y) + (1 << (s-1))) >> s) + b is used as a prediction value in motion compensation instead of p(x, y).

[0119] When WP is enabled, for each reference picture of the current slice, a flag is signaled to indicate whether WP is applied to the reference picture. If WP is applied to a reference picture, a set of WP parameters (i.e., a, s, and b) is sent to the decoder and used for motion compensation from the reference picture. In some examples, in order to flexibly turn on / off WP for luma and chroma components, the WP flag and WP parameters are signaled separately for luma and chroma components. In WP, all pixels in a reference picture use the same set of WP parameters.

[0120] In JEM, a local illumination compensation (LIC) method is proposed. The description of LIC in JEM can be found in JVET-G1001. Regarding LIC in JVET, LIC is based on a linear model for illumination variations, using a scaling factor a and an offset b. This LIC is adaptively enabled or disabled for each codec unit (CU) of inter-mode codec. When LIC is applied to a CU, the parameters a and b are derived using the least squares error method by using the neighboring samples of the current CU and its corresponding reference samples. For example, Figure 5A and Figure 5B As shown, the sub-sampling (2:1 sub-sampling) neighboring samples of the CU and the corresponding pixels in the reference picture (identified by the motion information of the current CU or sub-CU) are used. In some examples, IC parameters are derived and applied separately for each prediction direction. In some examples, an illumination compensation flag can be signaled for the CU to indicate whether LIC is applied. In some examples, for example, when the CU is encoded and decoded in merge mode, the illumination compensation flag can be copied from the neighboring block in a manner similar to the motion information copying in merge mode.

[0121] According to LIC, the luminance value (of a sample or pixel) is compensated in the inter-frame prediction in the linear form a*p+b, where p is the sample of the inter-frame prediction, a is the scaling factor, and b is the offset. The scaling factor a and the offset b are calculated using (e.g., Figure 5A and Figure 5BParameters derived from neighboring samples of the current block (shown) and neighboring samples of the reference block are used for inter prediction. For example, inter prediction may be first derived using motion information signaled by the block for inter coding, then parameters a and b may be derived, and then prediction refinement may be performed. Parameters may be derived by minimizing the difference between neighboring reconstructed samples of the current block and neighboring samples of the reference block used for inter prediction. In some cases, minimization may be performed using a linear least squares method and / or any other appropriate minimization method.

[0122] Figure 6 is a diagram showing an example of neighborhood reconstructed samples of a current block 602 and neighborhood samples of a reference block 604 for unidirectional inter prediction. A motion vector MV may be encoded and decoded for the current block 602, wherein the MV may include a reference index of a reference picture list and / or other motion information for identifying the reference block 604. For example, the MV may include horizontal and vertical components that provide an offset from a coordinate position in the current picture to a coordinate in the reference picture identified by the reference index.

[0123] Figure 7 is a diagram showing an example of neighborhood reconstruction samples of a current block 702 and neighborhood samples of a first reference block 704 and a second reference block 706 for bidirectional inter prediction. In this case, two motion vectors MV0 and MV1 may be encoded and decoded for the current block 702 to identify the first reference block 704 and the second reference block 706, respectively.

[0124] Illumination compensation (IC) parameters may include an offset, one or more weights, a shift number, or other suitable illumination compensation parameters. Weights may also be referred to as scaling factors. IC parameters may be derived for unidirectional inter prediction or bidirectional inter prediction. For bidirectional inter prediction, the one or more weights may include a first weight for a first reference picture and a second weight for a second reference picture.

[0125] In some embodiments, linear least squares regression may be used to estimate LIC parameters in bidirectional prediction motion compensation. In one example, the derivation of LIC parameters may be accomplished by solving a cost function. For example, the cost function may include using a least squares function. For example, a subset of samples from one or more neighboring blocks of the current block may be used to derive LIC parameters. Samples from neighboring blocks of the current block may be used to find possible illumination changes in the current block 702, because it may be assumed that there is a strong correlation between the neighboring samples (in the neighboring blocks) and the current samples (in the current block 702). For example, it may be assumed that the current block and the neighboring blocks that share the same motion information should contain very similar illumination values. Another reason for using neighboring samples is that the current block has not yet been predicted and the current block may not have pixels to use, in which case the (reconstructed) neighboring samples may be used to perform motion compensation for the current block.

[0126] In one illustrative example, the top neighbor, the left neighbor, or both the top and left neighbors may be used. For example, the top and left neighbors (N i ), a subset of pixels from the top and left neighbors (P0) of the first reference block 704, and a subset of pixels from the top and left neighbors (P1) of the second reference block 706 may be used to derive IC parameters for the current block 702. The samples of the neighboring blocks P0 and P1 may include the samples corresponding to the neighboring blocks N i In some cases, the corresponding samples used in the neighboring blocks P0 and P1 can be identified by the motion information of the current block. In an illustrative example, the motion vector can be signaled by the merge mode or the AMVP mode. The reference picture can be identified by the reference index of the reference picture, and the reference blocks 704 and 706 in the reference picture use motion vectors MV0 and MV1, respectively.

[0127] In some examples, more than one derivation method may be performed to derive LIC parameters. Examples of an inter-frame prediction engine or module for deriving LIC parameters at the encoder side may include Fig.13 The prediction processing unit 41, the motion estimation unit 42 and / or the motion compensation unit 44 are shown. Examples of inter-frame prediction engines or modules for deriving LIC parameters at the decoder side may include Fig.13 The prediction processing unit 81 and / or motion compensation unit 82 are shown. In such examples, the encoder or other transmitter-side device may signal to the decoder which derivation method to use at a sequence level (e.g., in a VPS and / or SPS), at a picture level (e.g., in a PPS), at a slice level (e.g., in a slice header), at a CTU level, at a CU level, at a PU level, or a combination thereof or other suitable signaling level.

[0128] In some examples, the least squares solution may be calculated based on multiple rows and / or columns of a neighborhood (e.g., a top neighborhood, a left neighborhood, both a top neighborhood and a left neighborhood, or other neighborhoods). Example numbers of rows and / or columns (and in some cases, typical numbers) include one, two, four, or any other suitable number of rows and / or columns. When multiple rows and / or columns of neighboring blocks are used, the above cost function may be modified. For example, if the block is a 16×16 block (16 rows of pixels×16 columns of pixels), and if two rows from the top neighboring block and two columns from the left neighboring block are used, then the neighboring block N i will include 64 samples (32 samples from the left neighboring block and 32 samples from the top neighboring block). In such an example, neighborhoods P0 and P1 will also include 64 samples.

[0129] In some cases, integer position samples (or pixels) are used for the derivation of LIC parameters. In some cases, fractional position samples are used for the derivation of LIC parameters. In some cases, both integer position samples and fractional position samples can be used. For example, the actual displacement of moving objects between pictures is continuous and tends not to follow the sampling grid of pictures in a video sequence. Therefore, fractional precision rather than integer precision can be used for motion vectors, thereby reducing residual errors and improving the encoding and decoding efficiency of video codecs. If the motion vector has a fractional value, the reference block needs to be interpolated accordingly. For example, the motion vector of the sample of the current block can point to a fractional pixel position in the reference block. The fractional pixel position refers to a sample (e.g., a luminance sample) at a fractional sample position (non-integer position) in the block. These positions need to be generated by interpolation. In one example, when using fractional position samples, an interpolated or filtered version of a reference block neighborhood (e.g., P0 or P1) can be used to reduce quantization errors from reconstructed pixels when deriving LIC parameters. One or more interpolation filters can be used to generate such an interpolated version of the template.

[0130] In some examples, only the luma component needs to be used to jointly optimize the LIC parameters for both Ref0 and Ref1. Alternatively, both luma and chroma components may be considered during the derivation of bidirectional prediction LIC parameters. The encoder may signal to the decoder (in a parameter set, in an SEI message, or in other suitable signaling mechanisms) whether to apply LIC to one or more chroma components, or to apply LIC to the luma component only.

[0131] In some examples, one or more flags may be signaled between the encoder and decoder at a block level to indicate whether a particular tool (e.g., LIC, merge mode, etc.) has been applied in the prediction of the block. For example, the merge mode flag may be stored with the motion information of the current block. Based on the merge mode flag, it may be inferred from the motion information whether the current block has been coded or decoded using the merge mode. In addition to the merge mode flag, an illumination compensation flag may be used to indicate that the CU has been coded or decoded. When the CU is coded or decoded using the merge mode, the illumination compensation flag may be copied from neighboring blocks in a manner similar to the motion information copying in the merge mode. Otherwise, if the CU has not been coded or decoded using the merge mode (e.g., the AMVP mode is used instead), an illumination compensation flag may be signaled for the CU to indicate whether LIC is applied.

[0132] There are challenges associated with deriving, encoding, and signaling LIC parameters. For example, consider reference Figure 7 In the bidirectional inter prediction discussed, the bidirectional prediction of the current block 702 can be considered to include two unidirectional predictions, a first unidirectional prediction using a first reference block (Ref0) 704 and a second unidirectional prediction using a second reference block (Ref1) 706. To derive LIC parameters for the current block 702, illumination compensation can be applied to each of the two unidirectional predictions, and then the unidirectional predictions can be weighted with illumination refinement to form a bidirectional prediction with illumination compensation. In some examples, LIC parameters can be derived independently from the two unidirectional predictions using Ref0 and Ref1 without considering their joint impact on the bidirectional prediction value. In some examples of bidirectional prediction, separate LIC compensated prediction patches are determined, and the LIC compensated prediction patches can be combined using equal weights (e.g., 0.5) to generate a final bidirectional prediction value.

[0133] In some examples of bidirectional prediction with illumination compensation, for both unidirectional predictions, LIC parameters are derived using nearby reconstruction samples (e.g., nearby reconstruction samples from neighboring blocks P0 and P1). Although stored nearby reconstruction samples may be available in some cases, such nearby reconstruction samples may not be available for one or both of the reference blocks Ref0 and Ref1.

[0134] In addition, as described above, the illumination compensation flag or LIC flag can be stored together with the motion information of the encoded block. For example, the illumination compensation flag can be included together with the motion information of the current block encoded and decoded using inter-frame prediction. In some examples, the motion information of the current block is obtained from the motion information of the reference block when a temporal motion vector predictor (TMVP) is used. In such examples regarding TMVP, the illumination compensation flag can be stored together with the motion information of the reference block to be used for the current block. In various examples, storing the illumination compensation flag (e.g., stored together with the motion information of the current block, the reference block, or stored in another form) incurs a storage cost. It is desirable to minimize the storage of the illumination compensation flag.

[0135] Systems, methods, and computer-readable media are described herein to address the above-referenced issues (and others) and to provide solutions for coordinating and integrating illumination (or illuminance) compensation with other inter-prediction modes.

[0136] In some examples, various techniques for processing bidirectional inter-frame prediction (or bidirectional prediction) are described. As described above, illumination compensation can be performed during bidirectional prediction motion compensation. For example, as previously described with reference to Figure 7 As described above, the conventional LIC algorithm for illumination compensation includes deriving two unidirectional LIC prediction values ​​and then applying weighting to the two unidirectional LIC prediction values ​​to obtain a bidirectional prediction value for illumination compensation.

[0137] In some examples, the above challenges can be overcome by performing bidirectional prediction without performing illumination compensation for each unidirectional prediction. For example, illumination compensation can be performed on the bidirectional prediction values ​​as a post-processing step. By avoiding deriving illumination compensation for each unidirectional prediction during the process of bidirectional inter-frame prediction, the complexity of solving the cost function using unidirectional prediction for illumination compensation can be avoided. In addition, avoiding deriving illumination compensation for each unidirectional prediction during the process of bidirectional inter-frame prediction also avoids relying on neighboring blocks of both the first reference block and the second reference block when obtaining LIC parameters. Accordingly, bidirectional inter-frame prediction with illumination compensation can be performed with the example techniques even when one or more neighboring blocks of one or both of the first reference block and the second reference block may not be available.

[0138] Figure 8800 is a flow chart illustrating an example of a process 800 for applying illumination compensation to bidirectional prediction without performing illumination compensation during component unidirectional prediction. At 802, process 800 includes obtaining video data. In some examples, the video data may include encoded video data (e.g., an encoded video bitstream), such as when process 800 is performed by a decoding device. In some examples, the video data may include unencoded video data, such as when process 800 is performed by an encoding device. As described above, the video data may include multiple pictures, and the pictures may be divided into multiple blocks. The video data may also include motion information of the pictures and / or blocks that may be used to perform motion compensation.

[0139] At 804, process 800 includes performing bidirectional motion prediction on a current block of a picture of video data using a first reference block and a second reference block to obtain a bidirectional prediction value for the current block. In one illustrative example, the current block may be Figure 7 , the first reference block may be a reference block 704, and the second reference block may be a reference block 706. Performing bidirectional motion prediction (or bidirectional prediction motion compensation) on the current block 702 using the first reference block 704 and the second reference block 706 to obtain a bidirectional prediction value of the current block 702 may include obtaining a first unidirectional prediction value and a second unidirectional prediction value. For example, obtaining the first unidirectional prediction value may include deriving a motion vector MV0 using the first reference block 704 among other motion information, and obtaining the second unidirectional prediction value may include deriving a motion vector MV1 among other motion information using the first reference block 706. In some examples, performing bidirectional prediction motion compensation excludes performing illumination compensation in obtaining the first unidirectional prediction value and the second unidirectional prediction value.

[0140] At 806, process 800 includes deriving one or more illumination compensation parameters of the current block using one or more bidirectional predictors, one or more neighboring blocks of the first reference block, or one or more neighboring blocks of the second reference block. For example, the bidirectional predictors obtained at 804 may be used, and the LIC parameters of the current block 702 may be derived using neighboring block P0 of the first reference block and / or neighboring block P1 of the second reference block.

[0141] In some examples of process 800, illumination compensation is performed on the current block 702 by deriving one or more illumination compensation parameters (or local illumination compensation, "LIC", parameters) after deriving the bidirectional prediction. For example, the illumination compensation parameters can be applied to the bidirectional predictor as a post-processing operation for the bidirectional predictor. For example, the decoding device can apply the LIC parameters to the current block after predicting the current block using the bidirectional prediction of the bidirectional predictor. In such examples, an inter-frame prediction module for performing bidirectional prediction can be implemented without implementing illumination compensation. In addition, since reconstructed neighboring samples (e.g., neighboring blocks P0, P1, etc.) do not need to be used in bidirectional prediction, the inter-frame prediction module does not need to consider the reconstructed neighboring samples.

[0142] In some examples, reconstructed neighborhood samples (e.g., neighboring blocks P0, P1, etc.) may be used to obtain illumination compensation parameters for the current block. For example, LIC parameters may be derived using one or more neighboring blocks of the first reference block or one or more neighboring blocks of the second reference block. In some examples, the first reference block and the second reference block may be used for unidirectional prediction of the current block, where the first reference block and the second reference block may be referred to as unidirectional prediction value blocks. A combination of two unidirectional prediction value blocks may be used for bidirectional prediction of the current block. In some examples, one or more neighborhood samples of the unidirectional prediction value block may be used to derive one or more illumination compensation parameters for the current block.

[0143] In some examples, one or more neighboring samples of the first reference block may be referred to as a first set of neighboring samples, and one or more neighboring samples of the second reference block may be referred to as a second set of neighboring samples. Thus, using neighboring blocks of two unidirectional prediction value blocks to derive one or more illumination compensation parameters may include applying a combination of the first set of neighboring blocks and the second set of neighboring blocks to derive a single LIC parameter set. For example, the combination of the first set of neighboring blocks and the second set of neighboring blocks may include a half-sum, an average, a weighted average, or other combination of samples obtained from the first set of neighboring blocks and the second set of neighboring blocks. In some examples, the weights used for the weighted average or weighted sum of the neighboring samples may include weighting coefficients used for weighted inter-frame prediction of the current block.

[0144] In some examples, the above challenges can be overcome by performing bidirectional prediction without performing illumination compensation for each unidirectional prediction. For example, illumination compensation can be performed on the bidirectional prediction values ​​as a post-processing step. By avoiding deriving illumination compensation for each unidirectional prediction during the process of bidirectional inter-frame prediction, the complexity of solving the cost function using unidirectional prediction for illumination compensation can be avoided. In addition, avoiding deriving illumination compensation for each unidirectional prediction during the process of bidirectional inter-frame prediction also avoids relying on neighboring blocks of both the first reference block and the second reference block when obtaining LIC parameters. Accordingly, bidirectional inter-frame prediction with illumination compensation can be performed with the example techniques even when one or more neighboring blocks of one or both of the first reference block and the second reference block may not be available.

[0145] In another example technique for encoding and decoding a current block, one or more of the above challenges associated with applying illumination compensation can be overcome by selectively avoiding illumination compensation in some cases. For example, if the current block is to be encoded and decoded using bidirectional prediction, illumination compensation for the current block can be suppressed or not performed. For example, illumination compensation can be disabled for a current block to which bidirectional prediction has been applied or is to be applied.

[0146] In some examples, disabling illumination compensation can be implemented by suppressing or avoiding signaling of an illumination compensation flag (or LIC flag). For example, when constructing a motion vector candidate list for inter-mode prediction of the current block using bidirectional prediction, the illumination compensation flag can be set to a "false" value. In some examples, the illumination compensation flag can be set to false for all bidirectional motion vector candidates.

[0147] In an alternative embodiment, the illumination compensation flag may be stored with the motion vector prediction value for the current block, where the illumination compensation flag and motion information may be propagated to other blocks that may be predicted with that information (e.g., in merge mode). However, if bi-directional prediction is used for inter prediction of any block, illumination compensation for that block may be suppressed or not performed even if the associated illumination compensation flag is set to a "true" value.

[0148] Fig. 9900 is a flowchart illustrating an example of a process 900 for decoding video data. In some examples, the process 900 may include selectively applying illumination compensation based on whether bidirectional prediction is applied. At block 902, the process 900 includes obtaining an encoded video bitstream including video data. At block 904, the process 900 includes obtaining a current block of a picture of the video data from the encoded video bitstream. In some examples, the video data may include encoded video data (e.g., in an encoded video bitstream), such as when the process 900 is performed by a decoding device. As described above, the video data may include multiple pictures, and the pictures may be divided into multiple blocks. The video data may also include motion information of the pictures and / or blocks that may be used to perform motion compensation.

[0149] At block 906, process 900 includes determining whether to apply bidirectional prediction to inter-predict a current block of a picture of video data. In some examples, information about whether to apply bidirectional prediction may be signaled, or the information about whether to apply bidirectional prediction may be based on stored information or may be associated with the current block. For example, signaling from an encoding device to a decoding device may indicate whether the current block has been encoded or decoded with bidirectional prediction motion compensation.

[0150] In some examples, the indication of whether to apply bi-prediction to inter-prediction of the current block can be based on an illumination compensation flag associated with the motion information used for inter-prediction of the current block. For example, a value of the illumination compensation flag set to false can indicate that bi-prediction will be applied to inter-prediction of the current block (and illumination compensation will not be applied to the current block). On the other hand, a value of the illumination compensation flag set to true can indicate that bi-prediction will not be applied to inter-prediction of the current block (and illumination compensation will be applied to the current block).

[0151] In some examples, whether bidirectional prediction is applied to inter-frame prediction of the current block can be determined based on whether the signaling information includes an illumination compensation flag associated with the motion information of the current block. For example, the signaling information not including the illumination compensation flag can indicate that bidirectional prediction is to be applied to inter-frame prediction of the current block (and illumination compensation is not to be applied to the current block).

[0152] In some examples, the motion information of the current block may be inherited. For example, in merge mode, the motion information of the current block may be inherited from the motion information of an adjacent block or another block in the candidate list. In some examples, the illumination compensation flag may be stored together with the motion information of the current block in a motion vector candidate list including the motion information of the current block.

[0153] In some examples, information about whether to apply bidirectional prediction may be determined. For example, applying bidirectional prediction to the current block may include using a first reference block and a second reference block to obtain a bidirectional prediction value for the current block. Considering Figure 7 In an illustrative example of , determining whether to apply bidirectional prediction motion compensation to the current block 702 can be based on whether the first reference block 704 and the second reference block 706 can be used to obtain the first unidirectional prediction value and the second unidirectional prediction value, respectively. In some examples, if the first reference block 704 and the second reference block 706 are available, the first reference block 704 and the second reference block 706 can be used to obtain the first unidirectional prediction value and the second unidirectional prediction value, respectively. In some cases, one or both of the first reference block 704 and the second reference block 706 may not be available, and accordingly, bidirectional prediction motion compensation may not be applied to the current block 702.

[0154] At block 908, process 900 includes determining whether to apply illumination compensation to the current block after determining whether to apply bi-prediction to the inter prediction of the current block. For example, after the determination at block 906, a determination may be made as to whether illumination compensation is to be applied to the current block. In some examples, the determination as to whether illumination compensation is to be applied to the current block may be based on whether bi-prediction is to be applied to the inter prediction of the current block. For example, if bi-prediction is to be applied to the current block, illumination compensation for the current block may be avoided or suppressed. In some examples, an illumination compensation flag may indicate whether illumination compensation is to be applied. For example, an illumination compensation flag set to false indicates that illumination compensation is not to be applied to the current block. Otherwise, an illumination compensation flag set to true may indicate that illumination compensation is to be applied to the current block. In some examples, if it is determined that bi-prediction is to be applied to the current block, the illumination compensation flag may be set to false. When bi-prediction is used for inter prediction of the current block, challenges associated with applying illumination compensation to bi-predicted blocks may be avoided by avoiding, suppressing, or disabling illumination compensation.

[0155] In some examples, applying illumination compensation to the current block may include deriving one or more illumination compensation parameters for the current block using one or more neighboring blocks of the current block and one or more neighboring blocks of a reference block used for inter-frame prediction of the current block. For example, as previously described, the illumination value (of a sample or pixel) may be compensated in the inter-frame prediction in a linear form a*p+b, where p is a sample in the inter-frame prediction, a is a scaling factor, and b is an offset. The scaling factor a and the offset b may be used (e.g., as Figure 5A and Figure 5B In some examples, the one or more neighboring blocks of the current block may include one or more of a top neighboring block or a left neighboring block of the current block, and the one or more neighboring blocks of the reference block may include one or more of a top neighboring block or a left neighboring block of the reference block.

[0156] In another example technique for encoding and decoding a current block, one or more of the above challenges associated with applying illumination compensation can be overcome by selectively avoiding illumination compensation in some cases. For example, if the current block is to be encoded and decoded using bidirectional prediction, illumination compensation for the current block can be suppressed or not performed. For example, illumination compensation can be disabled for a current block to which bidirectional prediction has been applied or is to be applied.

[0157] As previously described, an illumination compensation flag (or LIC flag) may be stored with the motion information of the current block. In some examples, the illumination compensation flag may be set to false for all bidirectional motion vector candidates. In some examples, illumination compensation may be avoided even if the illumination compensation flag indicates otherwise (e.g., set to true). For example, if an illumination compensation flag set to true has been inherited from another block, but bidirectional motion compensation is to be performed on the current block, illumination compensation for the current block may be avoided based on a determination that bidirectional motion compensation is to be applied to the current block.

[0158] As previously described, the illumination compensation flag may be stored with the motion information of the block. In some examples, the stored illumination compensation flag may be inherited with the associated motion information. For example, the illumination compensation flag and motion information of the neighboring block may be inherited and used for motion prediction of the current block (e.g., in merge mode). In some examples, the motion prediction of the current block may include using the inherited motion information without modification. In such examples, illumination compensation may be performed on the current block based on the value of the inherited illumination compensation flag. For example, the motion vector inherited from the neighboring block may be used as the motion vector of the current block, and illumination compensation may be performed if the illumination compensation flag inherited from the neighboring block is true. In such examples, if the illumination compensation flag is true, the LIC parameters may also be inherited from the neighboring block and used as the LIC parameters of the current block.

[0159] In other examples, motion prediction of the current block may include deriving motion information of the current block from motion information of another block, such as an adjacent block. In such examples, deriving motion information of the current block may include modifying motion information of an adjacent block or another block (e.g., in AMVP or TMVP mode). For example, in TMVP mode, the motion vector of the adjacent block may be scaled, and the scaled motion vector may be used as the motion vector of the current block. In paired motion vector derivation, the motion vector MV of the current block may be derived using two motion vectors MV1 and MV2 of two adjacent blocks. For example, the motion vector MV of the current block may be derived as a half sum of two motion vectors MV1 and MV2, where MV=(MV1+MV2) / 2. In such examples (among other examples), the motion vector of the current block may be different from one or more motion vectors of other blocks used to derive the motion vector of the current block. When the motion vector of the current block may be different from one or more motion vectors of other blocks used to derive the motion vector of the current block, inheriting the illumination compensation flag of the current block from one or more other blocks may be problematic. For example, the derived (modified) motion vector for the current block may point to a block in the reference picture that is different from the block pointed to by one or more motion vectors (unmodified) of other blocks. Accordingly, even if illumination compensation is performed on one or more other blocks (or the inherited illumination compensation flag is true), illumination compensation may not be required for the current block whose motion information is derived from the motion vectors of one or more other blocks.

[0160] Fig.10 1 is a flow chart illustrating an example of a process 1000 for encoding video data. In some examples, the process 1000 includes selectively applying illumination compensation based on whether bidirectional prediction is applied. At block 1002, the process 1000 includes obtaining a current block of a picture of video data. In some examples, the video data may include unencoded video data, such as when the process 1000 is performed by an encoding device. As described above, the video data may include multiple pictures, and the pictures may be divided into multiple blocks. The process 1000 may determine motion information of the pictures and / or blocks that may be used to perform motion compensation.

[0161] At block 1004, process 1000 includes determining whether to apply bi-directional prediction to inter-predict a current block of a picture of video data. In some examples, the indication of whether to apply bi-directional prediction to inter-prediction of the current block may be based on an illumination compensation flag associated with motion information for inter-prediction of the current block. For example, a value of the illumination compensation flag set to false may indicate that bi-directional prediction will be applied to inter-prediction of the current block (and illumination compensation will not be applied to the current block). On the other hand, a value of the illumination compensation flag set to true may indicate that bi-directional prediction will not be applied to inter-prediction of the current block (and illumination compensation will be applied to the current block). In some examples, the illumination compensation flag may be stored. For example, the illumination compensation flag may be stored with the motion information of the current block in a motion vector candidate list containing the motion information of the current block.

[0162] At block 1006, process 1000 includes determining whether to apply illumination compensation to the current block after determining whether to apply bi-prediction to inter prediction of the current block. For example, determining whether to apply illumination compensation to the current block may be based on whether bi-prediction is to be applied to inter prediction of the current block. For example, if bi-prediction is to be applied to the current block, illumination compensation for the current block may be avoided or suppressed. In some examples, an illumination compensation flag may indicate whether illumination compensation is to be applied. For example, an illumination compensation flag set to false indicates that illumination compensation is not to be applied to the current block. Otherwise, an illumination compensation flag set to true may indicate that illumination compensation is to be applied to the current block. In some examples, if it is determined that bi-prediction is to be applied to the current block, the illumination compensation flag may be set to false. When bi-prediction is used for inter prediction of the current block, challenges associated with applying illumination compensation to bi-predicted blocks may be avoided by avoiding, suppressing, or disabling illumination compensation.

[0163] As previously described, an illumination compensation flag (or LIC flag) may be stored with the motion information of the current block. In some examples, the illumination compensation flag may be set to false for all bidirectional motion vector candidates. In some examples, illumination compensation may be avoided even if the illumination compensation flag indicates otherwise (e.g., set to true). For example, if an illumination compensation flag set to true has been inherited from another block, but bidirectional motion compensation is to be performed on the current block, illumination compensation for the current block may be avoided based on a determination that bidirectional motion compensation is to be applied to the current block.

[0164] At block 1008, process 1000 includes generating an encoded bitstream including at least a portion of the current block. In some examples, the illumination compensation flag may be included in the encoded bitstream. In some examples, if it is determined that illumination compensation is to be applied to the current block, the illumination compensation flag may be signaled. In some examples, not signaling the illumination compensation flag for the current block may indicate a determination that illumination compensation is not to be applied to the current block.

[0165] Fig.11 1 is a flow chart illustrating an example of a process 1100 for processing video data by selectively applying illumination compensation based on whether the motion information of the current block is obtained by modifying the motion vector of one or more other blocks. At 1102, the process 1100 includes obtaining video data. In some examples, the video data may include encoded video data (e.g., an encoded video bitstream), such as when the process 1100 is performed by a decoding device. In some examples, the video data may include unencoded video data, such as when the process 1100 is performed by an encoding device. As described above, the video data may include multiple pictures, and the pictures may be divided into multiple blocks. The video data may also include motion information of the pictures and / or blocks that can be used to perform motion compensation.

[0166] At 1104, process 1100 includes determining whether one or more motion vectors of a current block of a picture of video data are obtained by modifying one or more motion vectors of one or more other blocks. For example, in TMVP mode, one or more motion vectors of the current block may be obtained by modifying (e.g., scaling or changing) one or more motion vectors of one or more previously encoded blocks. In some examples, when one or more motion vectors of the current block are obtained by performing an operation on one or more motion vectors of one or more other blocks, one or more motion vectors of the current block may be obtained by modifying one or more motion vectors of one or more other blocks. For example, the operation may include averaging the motion vectors of adjacent blocks used in paired MV derivation, performing a half sum, a weighted sum, etc.

[0167] At 1106, process 1100 includes determining whether to apply illumination compensation to the current block based on whether the one or more motion vectors of the current block are obtained by modifying one or more motion vectors of one or more other blocks. For example, if it is determined at 1104 that the one or more motion vectors of the current block are obtained by modifying one or more motion vectors of one or more other blocks, illumination compensation for the current block may be avoided or suppressed. For example, illumination compensation for the current block may be avoided or suppressed by setting an illumination compensation flag of the current block to false.

[0168] In some examples, when the illumination compensation flag is set to false, the illumination compensation flag may not be stored. For example, in TMVP mode, an illumination compensation flag having a value of false may be prevented from being stored with other motion information stored for the current block. In some examples, storage efficiency may be achieved by not storing the illumination compensation flag when the illumination compensation flag is set to false.

[0169] In some examples, illumination compensation may be performed on a block including a partition. An example of block partitioning is triangular partitioning described in the Versatile Video Codec (VVC). VVC is a video codec standard being developed by JVET that has significant improvements in compression performance over the existing HEVC standard. In VVC, a block can be partitioned into two triangular PUs using diagonal or anti-diagonal partitioning. Other types of block partitioning may include different partitioning for partitioning a block into two or more blocks of other shapes. For partitioned blocks, a local illumination compensation (LIC) flag may be derived, used, or signaled at the block level. However, the illumination compensation flag of the entire block may represent the illumination compensation requirements of the constituent blocks within the partition. For example, the illumination compensation flag of the current block partitioned into triangular PUs may not be accurate or may not represent the LIC to be applied to the two triangular PUs. For example, it may not be efficient or necessary to perform illumination compensation on the two PUs of a block. This problem may also occur in other partitioning modes (e.g., in VVC, HEVC, AVC, any MPEG standard, and / or other video codec standards).

[0170] In some examples, illumination compensation techniques are described to address such issues in partitioned blocks. In one example, more than one illumination compensation flag may be used for a partitioned block. For example, for one or more of two or more PUs present in a partitioned block, an illumination compensation flag may be signaled at the PU level. In another example, illumination compensation may be avoided or not used for a block that has been partitioned to include more than one PU. In another example, illumination compensation parameters may be derived for a PU from neighboring samples that share a boundary with the PU.

[0171] Fig. 12A , Fig. 12B , Fig. 12C and Fig.12D 1 is a conceptual diagram showing an example of a block 1202 partitioned into two triangular PUs (PU0 and PU1). Block 1202 is also shown to have two neighboring blocks, a left neighboring block 1204 and a top neighboring block 1206. Based on how block 1202 is partitioned, PU0 and PU1 may have neighbors (e.g., share a border with) neighboring blocks 1204 and 1206, none, one, or both of them. Illumination compensation for PU0 and PU1 may be based on samples of their respective neighboring blocks (if any). If a PU has no neighbors, illumination compensation is not performed on the PU.

[0172] For example, in Fig. 12A and Fig. 12B , PU1 shares a border with the top adjacent block 1206, and PU2 shares a border with the left adjacent block 1204. Fig. 12AAs shown, the illumination compensation of PU1 can be based on the samples of the top neighboring block 1206. Similarly, Fig. 12B As shown, the illumination compensation of PU2 may be based on the samples of the neighboring block 1204 on the left.

[0173] exist Fig. 12C and Fig.12D , PU1 shares a boundary with the top neighboring block 1206 and the left neighboring block 1204, while PU0 does not share a boundary with any of the two neighboring blocks 1204 and 1206. Fig. 12C As shown, the illumination compensation of PU1 can be based on the samples of both the top neighboring block 1206 and the left neighboring block 1204. Fig.12D As shown, no illumination compensation is performed on PU2.

[0174] As mentioned above, the block can be partitioned into any other shape or number of partitions. For example, the illumination compensation technique based on the samples of the neighboring blocks (if any) with which the partition shares a boundary can be applied to any partitioning method (not only for triangulation), such as partitioning the block using quad partitioning, binary partitioning, ternary partitioning and / or any other partitioning.

[0175] In some example techniques described herein, illumination compensation may be applied based on other inter-frame codec tools applicable to encode or decode one or more blocks of a picture. For example, for blocks encoded and decoded using some inter-frame codec techniques, illumination compensation at the block level may not result in performance or quality improvements. For example, some blocks may be partitioned into sub-blocks as described above. Inter-frame prediction of sub-blocks may be based on the use of different motion vectors or motion models. For example, tools such as optional temporal motion vector prediction (ATMVP), affine prediction, etc. may be used to perform sub-block-based prediction. In an illustrative example, each sub-block (4×4, 8×8, or any other size) may have its own motion vector, and having one illumination compensation flag for all sub-blocks may not be accurate. However, it may also be expensive to signal a separate individual illumination compensation flag for each sub-block. For example, multiple illumination compensation flags for signaling illumination compensation for multiple sub-blocks may incur overhead in terms of computing resources such as storage space, bandwidth, and processing requirements.

[0176] In some example techniques for addressing such issues that may occur for a partitioned block, illumination compensation may be disabled for a sub-block of the partitioned block. For example, as described above, if a block is partitioned into sub-blocks, each sub-block may have associated motion vector information specific to the sub-block. In some examples, an illumination compensation flag may be associated with the motion vector of the sub-block and stored with the motion vector of the sub-block. In such examples where an illumination compensation flag is associated with a sub-block, the illumination compensation flag of the sub-block may be set to false. Accordingly, illumination compensation may be avoided for sub-blocks whose illumination compensation flags associated with their respective motion information are set to false. By avoiding illumination compensation for a sub-block, the associated overhead of computational resources for the sub-block may be avoided.

[0177] In another example, the illumination compensation flag can be derived from the neighboring blocks, as shown in FIG. Fig.12D As described. For example, if a sub-block or PU shares a boundary with an adjacent block, the sub-block can derive an illumination compensation flag of a neighboring block. In some examples, the illumination compensation flag of a sub-block that shares a boundary with an adjacent block can be set to true. In such an example, the illumination compensation flag (derived from an adjacent block and / or set to true) can be associated with motion information used to encode and decode the sub-block. In some cases, the illumination compensation flag can be stored with the motion information of the sub-block and can be transmitted or assigned to another block that uses the motion information of the sub-block for encoding and decoding. Although in some examples, illumination compensation of a sub-block can be performed based on the value of the illumination compensation flag, according to the technology herein, illumination compensation of a sub-block can be avoided regardless of the value of its associated illumination compensation flag. For example, even if a sub-block whose motion information includes a true illumination compensation flag is encountered, illumination compensation for the sub-block can be avoided. In this way, the overhead of computing resources for performing illumination compensation for the sub-block can be avoided.

[0178] In some examples, for example, as described in VVC, the techniques described herein can be used for merging with a motion vector differential mode (MMVD mode). In MMVD, a merge vector predictor can be used, to which an offset can be added when performing motion vector prediction for a block. In some examples, illumination compensation can be disabled when a specific offset is added to the merge vector predictor. In one example, when MMVD is performed on a block, an illumination compensation flag for the block can be set to false. In another example, the value of the illumination compensation flag can be retained, but illumination compensation is not performed regardless of the value of the illumination compensation flag (true or false). In some examples where an illumination compensation flag can be signaled, the signaling can be blocked or suppressed when it has been determined that illumination compensation is disabled or not to be applied.

[0179] In some examples, illumination compensation can be disabled for other techniques discussed herein by suppressing the signaling of an illumination compensation flag. For example, in various above-described processes for selectively enabling or disabling (avoiding) illumination compensation for specific blocks, disabling illumination compensation can be implemented by not signaling an illumination compensation flag for blocks for which illumination compensation has been determined to be disabled. For example, as described in reference Fig.10 As discussed in 1006 of process 1000 in , when the motion vector of the current block is obtained by modifying the motion vectors of one or more other blocks, illumination compensation can be determined to be disabled. For example, the illumination compensation flag can be inherited for the merged MV candidate, but its motion vector can include the modified motion vector. In this case, when the motion vector of the block is derived or obtained based on the modification of one or more other motion vectors, illumination compensation for the block is disabled regardless of the value of the illumination compensation flag.

[0180] In another example of using the MMVD mode, illumination compensation for a block with a modified motion vector can be based on the nature of the modification. For example, in performing motion vector prediction of a block, illumination compensation can be performed on the block based on the value of an offset added to the motion vector of one or more other blocks. In some examples, the motion vector prediction of a block can be based on an offset of a small value added to the motion vector of another block (e.g., a merged MV candidate block). In such an example where the offset value is small (e.g., less than a predetermined threshold), the motion vector of the block will not include a large change relative to the motion vector of another block used for motion vector prediction. Therefore, even if there may be a change in the motion information, since the change is considered to be small, illumination compensation can be performed on the block. However, if the offset is large (e.g., greater than a predetermined threshold), illumination compensation can be disabled for the block. Therefore, in some examples, by performing illumination compensation based on the amount of change or modification in the motion vector obtained for motion vector prediction of the block, improvements in accuracy or quality can be obtained.

[0181] In some examples, a block may be both inter-predicted and intra-predicted. For example, VVC supports an intra+inter prediction mode in which both inter- and intra-prediction values ​​are blended to form a block prediction. In such examples where both inter- and intra-prediction are performed, performing illumination compensation may create a bottleneck because performing illumination compensation will require both inter- and intra-prediction to be completed and blended. Therefore, in order to avoid the latency and complexity involved in applying illumination compensation in such situations, an example technique includes disabling illumination compensation in intra+inter prediction mode. Therefore, if a block is both inter- and intra-predicted, illumination compensation for the block may be disabled.

[0182] According to some example techniques discussed herein, bandwidth reduction can be achieved when performing illumination compensation. For example, as described above, illumination compensation parameters for a block can be obtained using samples of neighboring blocks of one or more reference blocks. Accordingly, obtaining LIC parameters for a current block can involve obtaining samples of neighboring blocks and sending the samples to an illumination compensation engine. The acquisition and transmission of samples can incur overhead in terms of computing resources such as bandwidth, storage, and processing.

[0183] In some examples, this overhead can be avoided by using samples of the reference block instead of samples of neighboring blocks of the reference block. In some examples, samples of the reference block can be obtained from rows and / or columns of the reference block, where the rows and columns are adjacent to the neighboring blocks of the reference block. For example, Figure 4B The top row (or first row) of the reference block shown will be adjacent to the neighboring pixels of the top neighborhood, and Figure 4B The left column (or first column) of the reference block shown will be adjacent to the neighboring pixels of the left neighborhood. In some examples, the samples obtained from the top row of the reference block may not change significantly from the samples obtained from the top neighborhood of the reference block. Similarly, the samples obtained from the left column of the reference block may not change significantly from the samples obtained from the left neighborhood of the reference block. Therefore, samples from the top row and / or left column of the reference block instead of the corresponding top neighborhood and / or left neighborhood of the reference block can be obtained to derive the LIC parameters of the current block. Obtaining samples from the top row and / or left column of the reference block can avoid the additional storage and overhead associated with obtaining the top neighborhood and / or left neighborhood. For example, samples of the reference block including samples from the top row and left column will be stored and sent anyway to derive motion information of the current block. Therefore, using samples from the top row and / or left column of the reference block for illumination compensation may improve efficiency because the values ​​of the samples obtained from the neighboring blocks and the samples of the top row / left column of the reference block may be similar (by being adjacent in the spatial domain).

[0184] In some cases, the above techniques can be combined with interpolation. As previously described, if the motion vector has a fractional value, the reference block needs to be interpolated accordingly. Interpolation can refer to motion compensated interpolation (MCI) or inter-frame prediction interpolation. In some examples, illumination compensation can be performed on blocks interpolated using motion compensated interpolation. In some cases, before performing motion compensated interpolation, samples of the reference block used in illumination compensation of the current block (e.g., samples from the top row and / or left column) can be utilized. In some cases, after performing motion compensated interpolation, samples of the reference block can be used for illumination compensation. In some examples, inter-frame prediction interpolation may be required depending on the motion vector used, because the motion vector does not need to indicate integer motion, but can include fractional pixel motion, such as half-pixel, quarter-pixel, N-pixel (N-pel), or other fractional pixel motion compensation can be used.

[0185] In some examples, the process of using samples of neighboring blocks of a reference block for illumination compensation can be improved by selectively reducing the reliance on one or more neighboring blocks used to obtain the samples. For example, illumination compensation parameters can be derived based on only one of the two neighboring blocks available for the reference block. Figure 6 or Figure 7 , illumination compensation may be performed on the current block using only the left neighbor (rather than the upper / top neighbor) of each reference block used in motion prediction of the current block. In another example, illumination compensation may be performed on the current block using only the top neighbor (rather than the left neighbor) of each reference block used in motion prediction of the current block. In other examples, less than all available immediate neighbors of the reference blocks may be used for illumination compensation of the current block.

[0186] In some examples, motion information comparison can be performed when constructing a motion vector candidate list. For example, in merge mode, the candidate list can be constructed to include motion information of previously predicted blocks. To avoid redundancy and improve storage efficiency, the motion information of the block to be added to the list can be compared with the motion information already in the list. In the event that there may be a duplication, the motion information of the block can be prevented from being added. While this is an example, motion information comparison can be performed in various other examples or processes of encoding or decoding video data.

[0187] In some examples, an illumination compensation flag (or LIC flag) is included in the motion information, and the illumination compensation flag can be used for motion information comparison. For example, when determining whether any two motion information sets are the same, the values ​​of the illumination compensation flags included in the two motion information sets can also be compared. In one example, if the illumination compensation flags of the motion information in the two sets are not equal, the motion information in the two sets can be considered different.

[0188] In some cases, it may not be possible or efficient to perform illumination compensation for a given block or prediction unit. For example, illumination compensation may be disabled for certain types of predictions, which may specify restrictions based on characteristics of the block. For example, in other scenarios, there may be block size restrictions (e.g., illumination compensation may not be applicable to certain block sizes). In this case, when illumination compensation is not possible or effective, the illumination compensation flag may not be considered in the motion information comparison. For example, when determining whether two motion information sets are the same or different, the motion information comparison may be performed by excluding the illumination compensation flag. For example, when comparing motion information, if the illumination compensation flags are not equal, but the rest of the motion information in the two sets is equal, the two motion information sets may be considered equal. In some examples, even if the illumination compensation flag may not be considered in the motion information comparison, the illumination compensation flag included in the motion information may be used for illumination compensation. For example, even if the illumination compensation flag may be excluded when comparing two motion information sets, the illumination compensation flag in one or both of the motion information sets may be used for illumination compensation of the block predicted using the motion information set. For example, a motion information set including an illumination compensation flag may be obtained from a motion candidate list created using the above-mentioned motion information comparison technique, and the illumination compensation flag may be used to determine whether illumination compensation is to be performed on a block.

[0189] In some examples, the illumination compensation flag may be excluded from motion information comparisons, regardless of the type of prediction used for the motion information. For example, whether to include the illumination compensation flag in motion information comparisons may not be based on block size restrictions or other criteria as described above. Instead, in some embodiments, the illumination compensation flag may be excluded from all motion information comparisons. Excluding the illumination compensation flag from motion information comparisons may simplify and improve the efficiency of motion information comparisons because illumination compensation flag comparisons are not performed when comparing motion information.

[0190] In some examples, motion information comparison can be used in a pruning process, which can be applied during motion vector candidate list construction as described above. During the pruning process, some motion candidates can be compared to each other and only unique candidates can be added. In some examples, motion information comparison can be used to convert a bidirectional prediction to a unidirectional prediction. For example, such a technique can be performed when two unidirectional motion information sets are determined to be the same based on a motion information comparison of the two unidirectional motion information sets. In such examples where motion information is used (e.g., for pruning, converting a bidirectional prediction to a unidirectional prediction, and other examples), the illumination compensation flag can be considered or excluded according to the above-described techniques for including or excluding an illumination compensation flag in a motion information comparison.

[0191] In some examples, illumination compensation may be applied to a portion of a prediction block. In one example, illumination compensation may be applied to a prediction sample of a reference block based on a distance of the prediction sample of the reference block from a neighboring sample. For example, a weighting function may be used to weight the samples of the reference block. In an example implementation of a weighting function, a higher weight may be assigned to samples of the reference block that are closer to a neighboring block (e.g., a top neighbor or a left neighbor) of the reference block, and a lower weight may be assigned to samples of the reference block that are farther from the neighboring block. Using a weighting function and a predetermined threshold for weight, illumination compensation may be applied to samples with a higher weight (e.g., greater than a weight threshold) and closer to a neighboring block, while illumination compensation may be avoided for samples with a lower weight (e.g., less than a weight threshold).

[0192] In some examples, one or more of the above techniques may be combined. Figure 8 , Fig. 9 and Fig.10 One or more of the processes 800, 900, and 1000 described in . For example, at 906, Fig. 9 The process 900 may include determining that bidirectional prediction is to be applied to inter prediction of the current block, and at 908, Fig. 9 The process 900 may include determining that illumination compensation is to be applied to the current block. In some examples where illumination compensation is to be applied to the current block, the illumination compensation may be applied to the current block according to the reference Figure 8 806 of the described process 800 derives illumination compensation parameters for the current block. For example, the illumination compensation parameters for the current block may be derived using one or more neighboring blocks of the current block and one or more of the following: a first set of one or more neighboring blocks of a first reference block used for inter-frame prediction of the current block, a second set of one or more neighboring blocks of a second reference block used for inter-frame prediction of the current block, or a combination of the first set of one or more neighboring blocks and the second set of one or more neighboring blocks.

[0193] In some examples, the one or more neighboring blocks of the current block may include one or more of the top neighboring blocks or the left neighboring blocks of the current block, the first set of one or more neighboring blocks may include one or more of the top neighboring blocks or the left neighboring blocks of the first reference block, and the second set of one or more neighboring blocks may include one or more of the top neighboring blocks or the left neighboring blocks of the second reference block. Figure 7 As shown, one or more adjacent blocks of the current block 702 may include one or more of the top adjacent blocks Ni or the left adjacent blocks Ni of the current block, a first set of one or more adjacent blocks may include one or more of the top adjacent blocks P0 or the left adjacent blocks P0 of the first reference block 704, and a second set of one or more adjacent blocks may include one or more of the top adjacent blocks P1 or the left adjacent blocks P1 of the second reference block 706.

[0194] In some examples discussed above, the illumination compensation flag may be considered when constructing the candidate motion information list. For example, if it is determined that the illumination compensation flag is associated with the motion information of the current block, it may be determined whether to use the illumination compensation flag for comparing the motion information of the current block with the motion information stored in the motion information candidate list, wherein the motion information of the current block is added to the motion information candidate list based on the comparison.

[0195] In some examples, such as reference Fig.11 As discussed, at step 1104, process 1100 may include determining whether one or more motion vectors of the current block are obtained by modifying one or more motion vectors of one or more other blocks. At step 1106, process 1100 may include determining whether to apply illumination compensation to the current block based on whether one or more motion vectors of the current block are obtained by modifying one or more motion vectors of one or more other blocks.

[0196] In some embodiments, the processes (or methods) described herein may be performed by a computing device or apparatus (e.g. Figure 1 For example, it can be performed by the system 100 shown in FIG. Figure 1 and Fig.13 The encoding device 104, another video source side device or a video sending device, Figure 1 and Fig.14The decoding device 112 shown and / or another client device (e.g., a player device, a display, or any other client device) to perform the process. In some cases, the computing device or apparatus may include one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other components configured to perform the steps of processes 800, 900, 1000, and 1100. In some examples, the computing device may include a mobile device, a desktop computer, a server computer, and / or a server system, or other types of computing devices. The components of the computing device (e.g., one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other components) may be implemented in a circuit. For example, the component may include and / or may be implemented using an electronic circuit or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. In some examples, the computing device or apparatus may include a camera configured to capture video data (e.g., a video sequence) including video frames. In some examples, the camera or other capture device that captures the video data is separate from the computing device, in which case the computing device receives or obtains the captured video data. The computing device may include a network interface configured to transmit video data. The network interface may be configured to transmit data based on an Internet Protocol (IP) or other types of data. In some examples, the computing device or apparatus may include a display for displaying output video content (e.g., samples of a picture of a video bitstream).

[0197] The processes may be described with reference to logic flow diagrams (including processes 800, 900, 1000, and 1100), the operations of which represent a sequence of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0198] In addition, the process can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is jointly executed on one or more processors through hardware or a combination thereof. As described above, the code can be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including multiple instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0199] The coding and decoding techniques discussed herein can be implemented in an example video coding and decoding system (e.g., system 100). In some examples, the system includes a source device that provides encoded video data that will be decoded by a target device later. In particular, the source device provides video data to the target device via a computer-readable medium. The source device and the target device may include any of a variety of devices, including desktop computers, notebooks (i.e., laptops), tablet computers, set-top boxes, telephone handsets, such as so-called "smart" phones, so-called "smart" pads, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some cases, the source device and the target device may be equipped for wireless communication.

[0200] The target device can receive the encoded video data to be decoded by the computer-readable medium. The computer-readable medium can include any type of medium or device capable of moving the encoded video data from the source device to the target device. In one example, the computer-readable medium can include a communication medium so that the source device can directly send the encoded video data to the target device in real time. The encoded video data can be modulated according to a communication standard such as a wireless communication protocol and sent to the target device. The communication medium can include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form a part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium can include a router, a switch, a base station, or any other equipment that can be used to facilitate communication from the source device to the target device.

[0201] In some examples, the encoded data can be output from the output interface to the storage device. Similarly, the encoded data can be accessed from the storage device through the input interface. The storage device may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a DVD, a CD-ROM, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, the storage device may correspond to a file server or another intermediate storage device that can store the encoded video generated by the source device. The target device can access the stored video data from the storage device by streaming or downloading. The file server can be any type of server capable of storing encoded video data and sending the encoded video data to the target device. Example file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. The target device can access the encoded video data through any standard data connection (including an Internet connection). This can include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a DSL, a cable modem, etc.), or a combination of the two, which is suitable for accessing the encoded video data stored on the file server. The transmission of the encoded video data from the storage device may be streaming transmission, download transmission, or a combination thereof.

[0202] The technology of the present disclosure is not necessarily limited to wireless applications or settings. The technology can be applied to support video encoding and decoding of any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, Internet streaming video transmission, such as dynamic adaptive streaming over HTTP (DASH), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, the system can be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0203] In one example, the source device includes a video source, a video encoder and an output interface. The target device may include an input interface, a video decoder and a display device. The video encoder of the source device may be configured to apply the technology disclosed herein. In other examples, the source device and the target device may include other components or arrangements. For example, the source device may receive video data from an external video source (e.g., an external camera). Similarly, the target device may interface with an external display device, rather than including an integrated display device.

[0204] The example system above is just an example. The technology for parallel processing of video data can be performed by any digital video encoding and / or decoding device. Although the technology of the present disclosure is usually performed by a video encoding device, these technologies can also be performed by a video encoder / decoder (commonly referred to as a "codec"). In addition, the technology of the present disclosure can also be run by a video preprocessor. The source device and the target device are merely examples of such codec devices, wherein the source device generates codec video data to send to the target device. In some examples, the source device and the target device can operate in a substantially symmetrical manner so that each device includes a video encoding and decoding component. Therefore, the example system can support one-way or two-way video transmission between video devices, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0205] The video source may include a video capture device, such as a video camera, a video archive comprising previously captured video, and / or a video feed interface receiving video from a video content provider. As another alternative, the video source may generate computer graphics-based data as the source video, or may generate a combination of real-time video, archived video, and computer-generated video. In some cases, if the video source is a video camera, the source device and the destination device may form a so-called camera phone or video phone. However, as mentioned above, the technology described in the present disclosure may be generally applicable to video coding and decoding, and may be applicable to wireless and / or wired applications. In each case, captured, pre-captured, or computer-generated video may be encoded by a video encoder. The encoded video information may then be output to a computer-readable medium via an output interface.

[0206] As described above, computer-readable media may include transient media such as those transmitted via wireless broadcast or wired networks, or storage media (i.e., non-transient storage media) such as hard disks, flash drives, optical disks, digital video disks, Blu-ray disks, or other computer-readable media. In some examples, a network server (not shown) may, for example, receive encoded video data from a source device via a network transmission and provide the encoded video data to a target device. Similarly, a computing device of a media production facility such as a disk stamping device may receive encoded video data from a source device and generate a disk containing the encoded video data. Therefore, in various examples, computer-readable media may be understood to include one or more computer-readable media in various forms.

[0207] The input interface of the target device receives information from the computer-readable medium. The information of the computer-readable medium may include syntax information defined by the video encoder, which is also used by the video decoder, including syntax elements that describe the characteristics and / or processing of blocks and other coding and decoding units (e.g., groups of pictures (GOPs)). The display device displays the decoded video data to the user and may include any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device. Various embodiments of the present application have been described.

[0208] The details of the encoding device 104 and the decoding device 112 are respectively described in Fig.13 and Fig.14 Shown in. Fig.13 1 is a block diagram illustrating an example encoding device 104 that can implement one or more of the techniques described in the present disclosure. The encoding device 104 can, for example, generate the syntax structures described herein (e.g., syntax structures of VPS, SPS, PPS, or other syntax elements). The encoding device 104 can perform intra-frame prediction and inter-frame prediction codecs on video blocks within a video slice. As previously described, intra-frame codecs rely at least in part on spatial domain prediction to reduce or eliminate spatial domain redundancy within a given video frame or picture. Inter-frame codecs rely at least in part on temporal domain prediction to reduce or eliminate temporal domain redundancy within adjacent or surrounding frames of a video sequence. Intra-frame mode (I mode) may refer to any of several spatial domain-based compression modes. Inter-frame modes such as unidirectional prediction (P mode) or bidirectional prediction (B mode) may refer to any of several temporal domain-based compression modes.

[0209] The encoding device 104 includes a segmentation unit 35, a prediction processing unit 41, a filter unit 63, a picture memory 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 includes a motion estimation unit 42, a motion compensation unit 44, and an intra-frame prediction processing unit 46. For video block reconstruction, the encoding device 104 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62. The filter unit 63 is intended to represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter. Although the filter unit 63 is not limited to a single-loop filter, the filter unit 63 is not limited to a single-loop filter. Fig.13 6 is shown as an in-loop filter, but in other configurations, filter unit 63 may be implemented as a post-loop filter. Post-processing device 57 may perform additional processing on the encoded video data generated by encoding device 104. The techniques of the present disclosure may be implemented by encoding device 104 in some cases. However, in other examples, one or more techniques of the present disclosure may be implemented by post-processing device 57.

[0210] like Fig.13 As shown, the encoding device 104 receives video data, and the segmentation unit 35 segments the data into video blocks. The segmentation may also include segmentation into slices, slice segments, slices or other larger units, and video block segmentation according to, for example, a quadtree structure of LCUs and CUs. The encoding device 104 generally shows a component for encoding a video block within a video slice to be encoded. The slice may be divided into a plurality of video blocks (and may be divided into a set of video blocks called slices). The prediction processing unit 41 may select one of a plurality of possible codec modes for the current video block based on error results (e.g., codec rate and distortion level, etc.), such as one of one or more inter-frame prediction codec modes in a plurality of intra-frame prediction codec modes. The prediction processing unit 41 may provide the resulting intra-frame or inter-frame codec block to the adder 50 to generate residual block data, and to the adder 62 to reconstruct the coding block used as a reference picture.

[0211] Intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction encoding and decoding of the current video block relative to one or more neighboring blocks in the same frame or slice as the current block to be encoded to provide spatial compression. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction encoding and decoding of the current video block relative to one or more prediction blocks in one or more reference pictures to provide temporal compression.

[0212] Motion estimation unit 42 may be configured to determine an inter-prediction mode for a video slice according to a predetermined pattern for a video sequence. The predetermined pattern may specify a video slice in a sequence as a P slice, a B slice, or a GPB slice. Motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are shown separately for conceptual purposes. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors, which estimate the motion of a video block. For example, a motion vector may indicate the displacement of a prediction unit (PU) of a video block within a current video frame or picture relative to a prediction block within a reference picture.

[0213] A prediction block is a block that is found to closely match a PU of a video block to be coded in terms of pixel difference, which may be determined by a sum of absolute difference (SAD), a sum of squared difference (SSD), or other difference metrics. In some examples, encoding device 104 may calculate values ​​for sub-integer pixel positions of a reference picture stored in picture memory 64. For example, encoding device 104 may interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of a reference picture. Thus, motion estimation unit 42 may perform motion searches relative to full pixel positions and fractional pixel positions and output motion vectors with fractional pixel precision.

[0214] Motion estimation unit 42 calculates a motion vector for a PU of a video block in an inter-coded slice by comparing the position of the PU to the position of a prediction block of a reference picture. The reference picture may be selected from a first reference picture list (list 0) or a second reference picture list (list 1), each of which identifies one or more reference pictures stored in picture memory 64. Motion estimation unit 42 sends the calculated motion vector to entropy encoding unit 56 and motion compensation unit 44.

[0215] Motion compensation performed by the motion compensation unit 44 may involve obtaining or generating a prediction block based on a motion vector determined by motion estimation, possibly performing interpolation with sub-pixel precision. Upon receiving a motion vector for a PU of a current video block, the motion compensation unit 44 may locate the prediction block to which the motion vector points in a reference picture list. The encoding device 104 forms a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded and decoded, forming pixel difference values. The pixel difference values ​​form residual data for the block and may include both luminance and chrominance difference components. The adder 50 represents one or more components that perform the subtraction operation. The motion compensation unit 44 may also generate syntax elements associated with the video block and the video slice for use by the decoding device 112 when decoding the video block of the video slice.

[0216] The intra-prediction processing unit 46 may intra-predict the current block as an alternative to the inter-prediction performed by the motion estimation unit 42 and the motion compensation unit 44, as described above. Specifically, the intra-prediction processing unit 46 may determine an intra-prediction mode for encoding the current block. In some examples, the intra-prediction processing unit 46 may encode the current block using various intra-prediction modes (e.g., during separate encoding), and the intra-prediction processing unit 46 may select a suitable intra-prediction mode from the tested modes to use. For example, the intra-prediction processing unit 46 may calculate rate-distortion values ​​using rate-distortion analysis for various tested intra-prediction modes, and may select an intra-prediction mode with the best rate-distortion characteristics from the tested modes. The rate-distortion analysis typically determines the amount of distortion (or error) between the coded block and the original uncoded block that is encoded to generate the coded block, as well as the bit rate (i.e., the number of bits) used to generate the coded block. The intra-prediction processing unit 46 may calculate a ratio based on the distortion and rate of the various coded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0217] In any case, after selecting an intra-prediction mode for a block, intra-prediction processing unit 46 may provide information indicating the intra-prediction mode selected for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode. Encoding device 104 may include in the transmitted bitstream configuration data a definition of coding contexts for various blocks and an indication of the most probable intra-prediction mode for each context, an intra-prediction mode index table, and a modified intra-prediction mode index table. The bitstream configuration data may include a plurality of intra-prediction mode index tables and a plurality of modified intra-prediction mode index tables (also referred to as codeword mapping tables).

[0218] After the prediction processing unit 41 generates a prediction block for the current video block via inter-frame prediction or intra-frame prediction, the encoding device 104 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and applied to the transform processing unit 52. The transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform such as a discrete cosine transform (DCT) or a conceptually similar transform. The transform processing unit 52 may convert the residual video data from a pixel domain to a transform domain, such as a frequency domain.

[0219] Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of the matrix including the quantized transform coefficients. Alternatively, entropy encoding unit 56 may perform the scan.

[0220] After quantization, the entropy coding unit 56 performs entropy coding on the quantized transform coefficients. For example, the entropy coding unit 56 may perform context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding technique. After entropy coding by the entropy coding unit 56, the encoded bitstream may be sent to the decoding device 112, or archived for later transmission or retrieval by the decoding device 112. The entropy coding unit 56 may also entropy encode the motion vectors and other syntax elements of the current video slice being coded.

[0221] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain for later use as a reference block of a reference picture. Motion compensation unit 44 may calculate a reference block by adding the residual block to a prediction block of one of the reference pictures in a reference picture list. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values ​​for motion estimation. Adder 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to produce a reference block for storage in picture memory 64. The reference block may be used by motion estimation unit 42 and motion compensation unit 44 as a reference block to inter-frame predict a block in a subsequent video frame or picture.

[0222] In this way, Fig.13 The encoding device 104 of 104 represents an example of a video encoder configured to derive LIC parameters, adaptively determine the size of the template, and / or adaptively select the weights. The encoding device 104 can derive LIC parameters, adaptively determine the size of the template, and / or adaptively select the weight set, for example, as described above. For example, the encoding device 104 can perform any of the techniques described herein, including the above with respect to Figure 8 , Fig. 9 and Fig.10 In some cases, some techniques of the present disclosure may also be implemented by the post-processing device 57 .

[0223] Fig.14 1 is a block diagram illustrating an example decoding device 112. The decoding device 112 includes an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, a filter unit 91, and a picture memory 92. The prediction processing unit 81 includes a motion compensation unit 82 and an intra-frame prediction processing unit 84. In some examples, the decoding device 112 may perform operations generally related to Fig.14 The decoding process described by the encoding device 104 is the opposite of the encoding process.

[0224] During the decoding process, the decoding device 112 receives an encoded video bitstream representing video blocks and associated syntax elements of an encoded video slice sent by the encoding device 104. In some embodiments, the decoding device 112 may receive the encoded video bitstream from the encoding device 104. In some embodiments, the decoding device 112 may receive the encoded video bitstream from the network entity 79 (e.g., a server, a media-aware network element (MANE), a video editor / splicer, or other such device configured to implement one or more of the above-described techniques). The network entity 79 may or may not include the encoding device 104. Before the network entity 79 sends the encoded video bitstream to the decoding device 112, some of the techniques described in the present disclosure may be implemented by the network entity 79. In some video decoding systems, the network entity 79 and the decoding device 112 may be part of separate devices, while in other cases, the functions described with respect to the network entity 79 may be performed by the same device including the decoding device 112.

[0225] The entropy decoding unit 80 of the decoding device 112 entropy decodes the bitstream to generate quantized coefficients, motion vectors and other syntax elements. The entropy decoding unit 80 forwards the motion vectors and other syntax elements to the prediction processing unit 81. The decoding device 112 can receive syntax elements at the video slice level and / or the video block level. The entropy decoding unit 80 can process and parse both fixed length syntax elements and variable length syntax elements in one or more parameter sets such as VPS, SPS and PPS.

[0226] When the video slice is coded as an intra-frame codec (I) slice, the intra-frame prediction processing unit 84 of the prediction processing unit 81 can generate prediction data for the video block of the current video slice based on the signaled intra-frame prediction mode and data from the previously decoded block of the current frame or picture. When the video frame is coded as an inter-frame codec (i.e., B, P, or GPB) slice, the motion compensation unit 82 of the prediction processing unit 81 generates a prediction block for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 80. The prediction block can be generated from one of the reference pictures in the reference picture list. The decoding device 112 can construct the reference frame lists (list 0 and list 1) using a default construction technique based on the reference pictures stored in the picture memory 92.

[0227] Motion compensation unit 82 determines prediction information for video blocks of the current video slice by parsing the motion vectors and other syntax elements, and uses the prediction information to generate a prediction block for the current video block being decoded. For example, motion compensation unit 82 may use one or more syntax elements in the parameter set to determine a prediction mode (e.g., intra or inter prediction mode) for encoding or decoding video blocks of the video slice, an inter prediction slice type (e.g., B slice, P slice, or GPB slice), construction information for one or more reference picture lists for the slice, a motion vector for each inter-coded video block of the slice, an inter prediction state for each inter-coded video block of the slice, and other information for decoding video blocks in the current video slice.

[0228] The motion compensation unit 82 may also perform interpolation based on an interpolation filter. The motion compensation unit 82 may calculate interpolation values ​​for sub-integer pixels of the reference block using an interpolation filter used by the encoding device 104 during encoding of the video block. In this case, the motion compensation unit 82 may determine the interpolation filter used by the encoding device 104 from the received syntax elements and may use the interpolation filter to generate a prediction block.

[0229] The inverse quantization unit 86 inverse quantizes or dequantizes the quantized transform coefficients provided in the bitstream and decoded by the entropy decoding unit 80. The inverse quantization process may include determining a quantization degree using a quantization parameter calculated by the encoding device 104 for each video block in the video slice, and likewise determining a degree of inverse quantization that should be applied. The inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT or other suitable inverse transform), an inverse integer transform, or a conceptually similar inverse transform process to the transform coefficients to produce a residual block in the pixel domain.

[0230] After the motion compensation unit 82 generates a prediction block for the current video block based on the motion vector and other syntax elements, the decoding device 112 forms a decoded video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82. Adder 90 represents one or more components that perform this addition operation. If desired, a loop filter (in the codec loop or after the codec loop) can also be used to smooth pixel transitions or otherwise improve video quality. Filter unit 91 is intended to represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter. Although filter unit 91 is Fig.1491 is shown as an in-loop filter, but in other configurations, filter unit 91 may be implemented as a post-loop filter. The decoded video blocks in a given frame or picture are then stored in picture memory 92, which stores reference pictures for subsequent motion compensation. Picture memory 92 also stores the decoded video for later presentation on a display device, such as Figure 1 A video target device 122 is shown.

[0231] In this way, Fig.14 The decoding device 112 of 114 represents an example of a video decoder configured to derive LIC parameters, adaptively determine the size of the template, and / or adaptively select the weights. The decoding device 112 can, for example, derive LIC parameters, adaptively determine the size of the template, and / or adaptively select the weight set as described above. For example, the decoding device 112 can perform any of the techniques described herein, including the above with respect to Figure 8 , Fig. 9 and Fig.10 The process described.

[0232] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transitory media that can store data, and does not include carrier waves and / or temporary electronic signals that are wirelessly or wired for propagation. Examples of non-transitory media may include, but are not limited to, disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory, or storage devices. Computer-readable media may store thereon code and / or machine-executable instructions that may represent any combination of processes, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent in any suitable manner (including memory sharing, message passing, token passing, network sending, etc.).

[0233] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0234] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those of ordinary skill in the art that embodiments can be practiced without these specific details. For the sake of clarity of explanation, in some cases, the present technology may be presented as including separate functional blocks, which include functional blocks including devices, device components, steps or routines in a method embodied in software or a combination of hardware and software. In addition to the components shown in the figures and / or described herein, other components may also be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary details. In other cases, known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments.

[0235] Various embodiments may be described above as processes or methods, which are depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe an operation as a sequential process, many operations may be performed in parallel or concurrently. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but there may be other steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or a main function.

[0236] The processes and methods according to the above examples can be implemented using computer executable instructions stored in or otherwise obtained from a computer readable medium. These instructions may include, for example, instructions and data that enable or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Some of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary files, intermediate format instructions (e.g., assembly language, firmware, source code, etc.). Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the examples include disks or optical disks, flash memory, USB devices provided with non-volatile memory, network storage devices, etc.

[0237] The equipment according to these public implementation processes and methods may include hardware, software, firmware, middleware, microcode, hardware description language or any combination thereof, and may take any of various form factors. When implemented in software, firmware, middleware or microcode, the program code or code segment (e.g., computer program product) that performs the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical form factor examples include laptop computers, smart phones, mobile phones, tablet computer devices or other small personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein may also be embodied in peripheral devices or plug-in cards. By further example, this function may also be implemented on a circuit board between different chips, or implemented on different processes performed in a single device.

[0238] Instructions, media for transmitting such instructions, computing resources for executing the instructions, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0239] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative embodiments of the present application have been described in detail herein, it should be understood that the concepts of the present disclosure may be embodied and applied differently in other ways, and the appended claims are intended to be interpreted as including such variations, unless limited by the prior art. Various features and aspects of the above-mentioned application may be used individually or in combination. In addition, without departing from the broader spirit and scope of this specification, embodiments may be used in any number of environments and applications outside of those described herein. Therefore, the description and the accompanying drawings are considered to be illustrative rather than restrictive. For the purpose of illustration, the methods are described in a particular order. It should be understood that in alternative embodiments, these methods may be performed in an order different from the order described.

[0240] One of ordinary skill will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of this specification.

[0241] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor, or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0242] The phrase "coupled to" refers to any component that is physically connected directly or indirectly to another component, and / or that communicates with another component (e.g., connected to other components via a wired or wireless connection, and / or other suitable communication interfaces).

[0243] "At least one of" a set and / or "one or more of" a set recited in claim language or other language indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, "at least one of A and B" recited in claim language means A, B, or A and B. In another example, "one or more of A, B, and C" recited in claim language and "one or more of A, B, or C" recited in claim language means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, "at least one of A and B" recited in claim language may mean A, B, or A and B, and may additionally include items not listed in the A and B sets.

[0244] The various illustrative logic blocks, modules, circuits and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above with respect to their functionality. Whether these functions are implemented as hardware or software depends on the specific application and design constraints imposed on the entire system. The technician can implement the described functions in different ways for each specific application, but this implementation method decision should not be interpreted as causing deviation from the scope of the application.

[0245] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. These techniques may be implemented in any of a variety of devices (e.g., general-purpose computers, wireless communication device handsets, or integrated circuit devices with multiple uses, including applications in wireless communication device handsets and other devices). Any features described as modules or components may be implemented together in an integrated logic device, or individually as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium, which includes a program code, which includes instructions for executing one or more of the above methods when executed. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, etc. Additionally or alternatively, these techniques may be implemented at least in part by a computer-readable communication medium that carries or transmits program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer (e.g., a propagated signal or wave).

[0246] The program code can be run by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor can be configured to perform any of the techniques described in this disclosure. A general-purpose processor can be a microprocessor, but in an alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the technology described herein. In addition, in some aspects, the functions described herein may be provided in a dedicated software module or a hardware module configured for encoding and decoding, or incorporated into a combined video codec (CODEC).

[0247] Illustrative examples of the present disclosure include:

[0248] Example 1: A method for processing video data, the method comprising: obtaining video data; determining a bidirectional prediction value of a current block of the video data, the bidirectional prediction value being determined by performing bidirectional inter-frame prediction on the current block; and performing illumination compensation on the bidirectional prediction value.

[0249] Example 2: The method according to Example 1, wherein performing illumination compensation on the bidirectional predictive value comprises: deriving one or more illumination compensation parameters for the bidirectional predictive value; and applying the one or more illumination compensation parameters to the bidirectional predictive value.

[0250] Example 3: The method of Example 2, wherein one or more illumination compensation parameters for the bidirectional prediction value are derived using neighboring reconstruction samples.

[0251] Example 4: The method according to Example 3, wherein the adjacent reconstruction samples are from one or more of a first adjacent block of the current block and a second adjacent block of a reference block used for bidirectional inter prediction.

[0252] Example 5: The method according to Example 4, wherein the first neighboring block includes one or more of a top neighboring block or a left neighboring block of the current block, and wherein the second neighboring block includes one or more of a top neighboring block or a left neighboring block of the reference block.

[0253] Example 6: The method according to any one of Examples 3-5, wherein deriving one or more illumination compensation parameters for the bidirectional prediction value comprises minimizing a difference between a reconstructed sample of a first neighboring block and a reconstructed sample of a second neighboring block.

[0254] Example 7: The method of any of Examples 2-6, wherein the one or more illumination compensation parameters are derived using neighboring samples of prediction values ​​from a single reference picture list used during bidirectional inter prediction.

[0255] Example 8: The method according to any of Examples 2-6, wherein one or more illumination compensation parameters are derived using a combination of neighboring samples of prediction values ​​of two reference picture lists used during bidirectional inter prediction.

[0256] Example 9: The method according to any one of Examples 3-8, wherein adjacent reconstruction samples are not used when performing bidirectional prediction.

[0257] Example 10: The method according to any one of Examples 1-9, wherein illumination compensation is performed as a post-processing step after performing bidirectional inter prediction on the current block.

[0258] Example 11: The method according to any one of Examples 1-10, wherein the bidirectional inter-frame prediction of the current block comprises performing a first unidirectional prediction and a second unidirectional prediction on the current block.

[0259] Example 12: The method according to Example 11, wherein the bidirectional inter-frame prediction of the current block further includes applying a first weight to the first unidirectional prediction and applying a second weight to the second unidirectional prediction.

[0260] Example 13: The method of any of Examples 1-12, wherein the one or more local illumination compensation parameters include at least one scaling factor and at least one offset.

[0261] Example 14: The method of any of Examples 1-13, wherein performing illumination compensation on the bidirectional predictor results in an illumination compensated bidirectional predictor, and further comprising decoding the current block using the illumination compensated bidirectional predictor.

[0262] Example 15: The method of any of Examples 1-13, further comprising signaling one or more illumination compensation parameters in the encoded video bitstream.

[0263] Example 16: The method of any one of Examples 1-15, wherein the current block comprises a plurality of prediction units (PUs), and further comprising signaling an illumination compensation flag for each prediction unit (PU) of the block.

[0264] Example 17: The method of any of Examples 1-15, wherein illumination compensation is not performed on the block of video data based on the block having more than one prediction unit.

[0265] Example 18: The method according to any one of Examples 1-15, wherein the current block comprises a plurality of prediction units (PUs), and wherein illumination compensation parameters of the PUs of the current block are derived from neighboring samples sharing a boundary with the PUs.

[0266] Example 19: An apparatus comprising a memory and a processor, the memory being configured to store video data, the processor being configured to process the video data according to any one of claims 1-18.

[0267] Example 20: The apparatus of claim 19, wherein the apparatus comprises a decoder.

[0268] Example 21: An apparatus according to Example 19, wherein the apparatus comprises an encoder.

[0269] Example 22: The apparatus of any of Examples 16-21, wherein the apparatus is a mobile device.

[0270] Example 23: The apparatus of any of Examples 16-22, wherein the apparatus comprises a display configured to display video data.

[0271] Example 24: The apparatus of any of Examples 16-23, wherein the apparatus comprises a camera configured to capture one or more pictures.

[0272] Example 25: A computer readable medium having stored thereon instructions that, when executed by a processor, perform the method of any of Examples 1 to 18.

[0273] Example 26: A method for processing video data, the method comprising: obtaining video data; determining whether bidirectional prediction is enabled for a current block of the video data; determining a bidirectional prediction value for the current block, the bidirectional prediction value being determined by performing bidirectional inter-frame prediction on the current block, wherein illumination compensation is disabled for the current block based on the bidirectional prediction being enabled.

[0274] Example 27: The method of Example 26, wherein the illumination compensation flag is not signaled for the current block.

[0275] Example 28: The method of Example 26, wherein for bidirectional motion vector candidates in the motion vector candidate list, an illumination compensation flag is set to false.

[0276] Example 29: The method of Example 26, wherein an illumination compensation flag is maintained using a bidirectional prediction value, wherein when the illumination compensation flag is true and when bidirectional inter-frame prediction is performed on the current block, illumination compensation is not performed on the current block.

[0277] Example 30: The method according to any one of Examples 26-29 further includes decoding the current block using the bidirectional prediction value.

[0278] Example 31: An apparatus comprising a memory and a processor, the memory being configured to store video data, the processor being configured to process the video data according to any of Examples 26-30.

[0279] Example 32: An apparatus according to Example 31, wherein the apparatus comprises a decoder.

[0280] Example 33: An apparatus according to Example 31, wherein the apparatus comprises an encoder.

[0281] Example 34: An apparatus according to any of Examples 31-33, wherein the apparatus is a mobile device.

[0282] Example 35: An apparatus according to any of Examples 31-34, wherein the apparatus comprises a display configured to display video data.

[0283] Example 36: An apparatus according to any of Examples 31-35, wherein the apparatus comprises a camera configured to capture one or more pictures.

[0284] Example 37: A computer readable medium having stored thereon instructions that when executed by a processor perform the method of any of Examples 26 to 30.

[0285] Article 1: A method for decoding video data, the method comprising: obtaining an encoded video bitstream including the video data; obtaining a current block of a picture of the video data from the encoded video bitstream; determining whether to apply bidirectional prediction to inter-frame prediction of the current block; and after determining whether to apply bidirectional prediction to inter-frame prediction of the current block, determining whether to apply illumination compensation to the current block.

[0286] Clause 2: The method according to clause 1, further comprising: determining whether to apply illumination compensation to the current block based on whether bidirectional prediction is to be applied to inter-frame prediction of the current block.

[0287] Clause 3: The method according to Clause 2, further comprising: determining that bidirectional prediction is to be applied to inter-frame prediction of the current block; and determining not to apply illumination compensation to the current block based on determining that bidirectional prediction is to be applied to inter-frame prediction of the current block.

[0288] Clause 4: The method according to any one of Clauses 2-3 further includes: determining that bidirectional prediction will not be applied to inter-frame prediction of the current block; and applying illumination compensation to the current block based on determining that bidirectional prediction will not be applied to inter-frame prediction of the current block.

[0289] Clause 5: The method of clause 4, wherein applying illumination compensation to the current block comprises: deriving one or more illumination compensation parameters for the current block using one or more neighboring blocks of the current block and one or more neighboring blocks of a reference block used for inter-frame prediction of the current block.

[0290] Clause 6: A method according to any one of clauses 2-5, wherein whether to apply bi-directional prediction to inter-frame prediction of the current block is determined based on an illumination compensation flag associated with the motion information of the current block.

[0291] Clause 7: The method of clause 6, wherein the value of the illumination compensation flag set to false indicates that bi-directional prediction is to be applied for inter prediction of the current block and illumination compensation is not to be applied for the current block.

[0292] Clause 8: According to the method of any of clauses 6-7, a value of the illumination compensation flag set to true indicates that bi-directional prediction is not to be applied to inter-frame prediction of the current block, and illumination compensation is to be applied to the current block.

[0293] Article 9: The method according to any one of Articles 2-8 also includes: obtaining signaling information from an encoded video bitstream; determining whether the signaling information includes an illumination compensation flag associated with the motion information of the current block; and determining whether to apply bidirectional prediction to inter-frame prediction of the current block based on determining whether the signaling information includes an illumination compensation flag associated with the motion information of the current block.

[0294] Article 10: The method according to Article 9 also includes: determining that the signaling information does not include an illumination compensation flag; and based on the signaling information not including the illumination compensation flag, determining that bidirectional prediction will be applied to inter-frame prediction of the current block and illumination compensation will not be applied to the current block.

[0295] Article 11: The method according to any one of Articles 9-10 also includes: applying one of bidirectional prediction of inter-frame prediction to the current block or illumination compensation to the current block; and reconstructing the samples of the current block based on the inter-frame prediction applied to the current block or the illumination compensation applied to the current block based on the bidirectional prediction.

[0296] Clause 12: The method of clause 11, wherein reconstructing the samples of the current block further comprises applying residual values.

[0297] Article 13: The method according to any one of Articles 1-12 also includes determining that bidirectional prediction will be applied to inter-frame prediction of the current block; determining that illumination compensation will be applied to the current block; and deriving illumination compensation parameters of the current block using one or more neighboring blocks of the current block and one or more of the following: a first set of one or more neighboring blocks of a first reference block for inter-frame prediction of the current block, a second set of one or more neighboring blocks of a second reference block for inter-frame prediction of the current block, or a combination of the first set of one or more neighboring blocks and the second set of one or more neighboring blocks.

[0298] Article 14: A method according to Article 13, wherein the one or more neighboring blocks of the current block include one or more of the top neighboring blocks or the left neighboring blocks of the current block, the first set of one or more neighboring blocks includes one or more of the top neighboring blocks or the left neighboring blocks of the first reference block, and the second set of one or more neighboring blocks includes one or more of the top neighboring blocks or the left neighboring blocks of the second reference block.

[0299] Article 15: The method according to any one of Articles 1 to 14 also includes: determining that an illumination compensation flag is associated with the motion information of the current block; and determining whether to use the illumination compensation flag for comparing the motion information of the current block with the motion information stored in the motion information candidate list, wherein the motion information of the current block is added to the motion information candidate list based on the comparison.

[0300] Article 16: The method according to any one of Articles 1 to 15 also includes: determining whether one or more motion vectors of the current block are obtained by modifying one or more motion vectors of one or more other blocks; and determining whether to apply illumination compensation to the current block based on whether one or more motion vectors of the current block are obtained by modifying one or more motion vectors of one or more other blocks.

[0301] Article 17: An apparatus for decoding video data, the apparatus comprising: a memory; and a processor implemented in a circuit and configured to: obtain an encoded video bitstream comprising video data; obtain a current block of a picture of the video data from the encoded video bitstream; determine whether to apply bidirectional prediction to inter-frame prediction of the current block; and after determining whether to apply bidirectional prediction to inter-frame prediction of the current block, determine whether to apply illumination compensation to the current block.

[0302] Clause 18: The apparatus of clause 17, wherein the processor is further configured to determine whether to apply illumination compensation to the current block based on whether bi-directional prediction is to be applied to inter-frame prediction of the current block.

[0303] Clause 19: The apparatus of clause 18, wherein the processor is further configured to: determine that bi-prediction is to be applied to inter-prediction of the current block; and determine not to apply illumination compensation to the current block based on determining that bi-prediction is to be applied to inter-prediction of the current block.

[0304] Clause 20: The apparatus of clause 19, wherein the processor is further configured to: determine that bi-prediction will not be applied to inter-prediction of the current block; and apply illumination compensation to the current block based on determining that bi-prediction will not be applied to inter-prediction of the current block.

[0305] Clause 21: An apparatus according to any one of Clauses 18-20, wherein whether to apply bi-directional prediction to inter-frame prediction of the current block is determined based on an illumination compensation flag associated with the motion information of the current block.

[0306] Clause 22: The apparatus of clause 21, wherein the value of the illumination compensation flag set to false indicates that bi-directional prediction is to be applied for inter prediction of the current block and illumination compensation is not to be applied for the current block.

[0307] Clause 23: An apparatus according to any of Clauses 21-22, wherein the value of the illumination compensation flag set to true indicates that bi-directional prediction is not to be applied to inter-frame prediction of the current block and illumination compensation is to be applied to the current block.

[0308] Article 24: An apparatus according to any one of Articles 18-23, wherein the processor is further configured to: obtain signaling information from an encoded video bitstream; determine whether the signaling information includes an illumination compensation flag associated with motion information of the current block; and determine whether to apply bidirectional prediction to inter-frame prediction of the current block based on determining whether the signaling information includes an illumination compensation flag associated with motion information of the current block.

[0309] Article 25: An apparatus according to Article 24, wherein the processor is further configured to: determine that the signaling information does not include an illumination compensation flag; and based on the signaling information not including the illumination compensation flag, determine that bidirectional prediction will be applied to inter-frame prediction of the current block and illumination compensation will not be applied to the current block.

[0310] Article 26: An apparatus according to any one of Articles 24-25, wherein the processor is further configured to: apply one of bidirectional prediction of inter-frame prediction to the current block or illumination compensation to the current block; and reconstruct the samples of the current block based on the inter-frame prediction applied to the current block or the illumination compensation applied to the current block based on the bidirectional prediction.

[0311] Clause 27: An apparatus according to any of Clauses 17-26, wherein the apparatus comprises a mobile device having a camera for capturing one or more pictures.

[0312] Article 28: Apparatus according to any of Articles 17-27, further comprising a display for displaying one or more images.

[0313] Article 29: A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: obtain an encoded video bitstream comprising video data; obtain a current block of a picture of the video data from the encoded video bitstream; determine whether to apply bidirectional prediction to inter-frame prediction of the current block; and after determining whether to apply bidirectional prediction to inter-frame prediction of the current block, determine whether to apply illumination compensation to the current block.

[0314] Article 30: An apparatus for encoding video data, the apparatus comprising: a memory; and a processor implemented in a circuit and configured to: obtain a current block of a picture of the video data; determine whether to apply bidirectional prediction to inter-frame prediction of the current block; after determining whether to apply bidirectional prediction to inter-frame prediction of the current block, determine whether to apply illumination compensation to the current block; and generate an encoded video bitstream including at least a portion of the current block.

[0315] Article 31: An apparatus comprising a memory and a processor, wherein the memory is configured to store video data, and the processor is configured to process the video data according to any one of Articles 1-16.

[0316] Clause 32: Apparatus according to clause 31, wherein the apparatus comprises a decoder.

[0317] Clause 33: Apparatus according to clause 31, wherein the apparatus comprises an encoder.

[0318] Article 34: A device according to any of Articles 31-33, wherein the device is a mobile device.

[0319] Clause 35: The apparatus of any of Clauses 31-34, wherein the apparatus comprises a display configured to display video data.

[0320] Clause 36: A device according to any of Clauses 31-35, wherein the device comprises a camera configured to capture one or more pictures.

[0321] Clause 37: A computer readable medium having stored thereon instructions which, when executed by a processor, perform the method of any one of Clauses 26 to 30.

Claims

1. A method for decoding video data, the method comprising: obtaining an encoded video bitstream comprising video data; obtaining a current block of a picture of the video data from the coded video bitstream; determining to inherit a value of an illumination compensation flag associated with an additional block of the video data for at least one motion vector candidate of the current block, the value of the illumination compensation flag associated with the additional block indicating that illumination compensation is to be performed on the current block; determining whether at least one motion vector candidate of the current block is obtained by modifying one or more motion vectors of one or more other blocks of the video data; as well as Based on whether the at least one motion vector candidate is obtained by modifying the one or more motion vectors, it is determined whether to apply illumination compensation to the current block according to the illumination compensation flag.

2. The method according to claim 1, further comprising: determining whether at least one motion vector candidate of the current block is obtained using bidirectional prediction; as well as Whether to apply illumination compensation to the current block according to the illumination compensation flag is determined based on whether the at least one motion vector candidate is obtained using bi-directional prediction.

3. The method according to claim 2, further comprising: Based on determining that at least one motion vector candidate for the current block is obtained using bi-directional prediction, determining not to apply illumination compensation to the current block.

4. The method according to claim 3, further comprising determining to perform illumination compensation on the current block based on the value of the illumination compensation flag of the at least one motion vector candidate, wherein Performing illumination compensation on the current block includes: One or more illumination compensation parameters for the current block are derived using one or more neighboring blocks of the current block and one or more neighboring blocks of a reference block used for inter-frame prediction of the current block.

5. The method according to claim 2, further comprising: Based on determining that the at least one motion vector candidate of the current block is obtained using bi-directional prediction, setting a value of at least one illumination compensation flag of the at least one motion vector candidate to a false value, the false value indicating that the illumination compensation is not to be applied.

6. The method according to claim 1, further comprising: determining that illumination compensation is to be applied to the current block; as well as An illumination compensation parameter for the current block is derived using one or more neighboring blocks of the current block and one or more of the following: a first set of one or more neighboring blocks of a first reference block used for inter-frame prediction of the current block, a second set of one or more neighboring blocks of a second reference block used for inter-frame prediction of the current block, or a combination of the first set of one or more neighboring blocks and the second set of one or more neighboring blocks.

7. The method according to claim 6, wherein: The one or more adjacent blocks of the current block include one or more of the top adjacent blocks or the left adjacent blocks of the current block, the first set of one or more adjacent blocks includes one or more of the top adjacent blocks or the left adjacent blocks of the first reference block, and the second set of one or more adjacent blocks includes one or more of the top adjacent blocks or the left adjacent blocks of the second reference block.

8. The method according to claim 1, further comprising: Determining whether to use the illumination compensation flag for comparing the motion information of the current block with motion information stored in a motion information candidate list, wherein the motion information of the current block is added to the motion information candidate list based on the comparison.

9. The method according to claim 1, further comprising: Based on determining that at least one motion vector candidate of the current block is not obtained by modifying one or more motion vectors of the one or more other blocks, it is determined that illumination compensation is to be applied to the current block according to the illumination compensation flag.

10. The method according to claim 9, further comprising determining to perform illumination compensation on the current block based on the value of the illumination compensation flag of the at least one motion vector candidate, wherein Performing illumination compensation on the current block includes: One or more illumination compensation parameters for the current block are derived using one or more neighboring blocks of the current block and one or more neighboring blocks of a reference block used for inter-frame prediction of the current block.

11. The method according to claim 1, further comprising: Based on determining that the at least one motion vector candidate of the current block is obtained by modifying the one or more motion vectors of the one or more other blocks, setting the value of at least one illumination compensation flag of the at least one motion vector candidate to a false value, the false value indicating that illumination compensation is not to be applied.

12. The method according to claim 11, wherein: The one or more motion vectors of the one or more other blocks are paired motion vector candidates.

13. An apparatus for decoding video data, the apparatus comprising: Memory; as well as A processor implemented in circuitry and configured to: obtaining an encoded video bitstream comprising video data; obtaining a current block of a picture of the video data from the coded video bitstream; determining to inherit a value of an illumination compensation flag associated with an additional block of the video data for at least one motion vector candidate of the current block, the value of the illumination compensation flag associated with the additional block indicating that illumination compensation is to be performed on the current block; determining whether at least one motion vector candidate of the current block is obtained by modifying one or more motion vectors of one or more other blocks of the video data; as well as Based on whether the at least one motion vector candidate is obtained by modifying the one or more motion vectors, it is determined whether to apply illumination compensation to the current block according to the illumination compensation flag.

14. The device according to claim 13, wherein: The processor is further configured to: determining whether at least one motion vector candidate of the current block is obtained using bidirectional prediction; and Whether to apply illumination compensation to the current block according to the illumination compensation flag is determined based on whether the at least one motion vector candidate is obtained using bi-directional prediction.

15. The device according to claim 14, wherein: The processor is further configured to: Based on determining that at least one motion vector candidate for the current block is obtained using bi-directional prediction, determining not to apply illumination compensation to the current block.

16. The device according to claim 15, wherein: The processor is further configured to determine, based on the value of the illumination compensation flag of the at least one motion vector candidate, whether to perform illumination compensation on the current block, wherein, in order to perform illumination compensation on the current block, the processor is further configured to: One or more illumination compensation parameters for the current block are derived using one or more neighboring blocks of the current block and one or more neighboring blocks of a reference block used for inter-frame prediction of the current block.

17. The device according to claim 14, wherein: The processor is further configured to: Based on determining that the at least one motion vector candidate of the current block is obtained using bi-directional prediction, setting a value of at least one illumination compensation flag of the at least one motion vector candidate to a false value, the false value indicating that the illumination compensation is not to be applied.

18. The apparatus of claim 13, further comprising a camera for capturing one or more pictures and a display for displaying the one or more pictures.

19. The device according to claim 13, wherein: The processor is configured to: determining that illumination compensation is to be applied to the current block; and Illumination compensation parameters for the current block are derived using one or more neighboring blocks of the block of the video data and one or more of: a first set of one or more neighboring blocks of a first reference block used for inter-frame prediction of the current block, a second set of one or more neighboring blocks of a second reference block used for inter-frame prediction of the current block, or a combination of the first set of one or more neighboring blocks and the second set of one or more neighboring blocks.

20. The device according to claim 19, wherein The one or more adjacent blocks of the current block include one or more of the top adjacent blocks or the left adjacent blocks of the current block, the first set of one or more adjacent blocks includes one or more of the top adjacent blocks or the left adjacent blocks of the first reference block, and the second set of one or more adjacent blocks includes one or more of the top adjacent blocks or the left adjacent blocks of the second reference block.

21. The device according to claim 13, wherein: The processor is configured to: Determining whether to use the illumination compensation flag for comparing the motion information of the current block with motion information stored in a motion information candidate list, wherein the motion information of the current block is added to the motion information candidate list based on the comparison.

22. The device according to claim 13, wherein: The processor is further configured to: Based on determining that at least one motion vector candidate of the current block is not obtained by modifying one or more motion vectors of the one or more other blocks, it is determined that illumination compensation is to be applied to the current block according to the illumination compensation flag.

23. The device according to claim 22, wherein: The processor is further configured to determine, based on the value of the illumination compensation flag of the at least one motion vector candidate, whether to perform illumination compensation on the current block, wherein, in order to perform illumination compensation on the current block, the processor is configured to: One or more illumination compensation parameters for the current block are derived using one or more neighboring blocks of the current block and one or more neighboring blocks of a reference block used for inter-frame prediction of the current block.

24. The device according to claim 13, wherein: The processor is further configured to: Based on determining that the at least one motion vector candidate of the current block is obtained by modifying the one or more motion vectors of the one or more other blocks, setting the value of at least one illumination compensation flag of the at least one motion vector candidate to a false value, the false value indicating that illumination compensation is not to be applied.

25. The device according to claim 24, wherein: The one or more motion vectors of the one or more other blocks are paired motion vector candidates.

26. A non-transitory computer readable medium having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to: obtaining an encoded video bitstream comprising video data; obtaining a current block of a picture of the video data from the coded video bitstream; determining to inherit a value of an illumination compensation flag associated with an additional block of the video data for at least one motion vector candidate of the current block, the value of the illumination compensation flag associated with the additional block indicating that illumination compensation is to be performed on the current block; determining whether at least one motion vector candidate of the current block is obtained by modifying one or more motion vectors of one or more other blocks of the video data; as well as Based on whether the at least one motion vector candidate is obtained by modifying the one or more motion vectors, it is determined whether to apply illumination compensation to the current block according to the illumination compensation flag.

27. The non-transitory computer readable medium of claim 26, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: Determining whether the current block is obtained using bidirectional prediction; and Whether to apply illumination compensation to the current block according to the illumination compensation flag is determined based on whether the at least one motion vector candidate is obtained using bi-directional prediction.

28. The non-transitory computer readable medium of claim 27, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: Based on determining that at least one motion vector candidate for the current block is obtained using bi-directional prediction, determining not to apply illumination compensation to the current block.

29. The non-transitory computer-readable medium of claim 28, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to determine to perform illumination compensation on the current block based on the value of the illumination compensation flag of the at least one motion vector candidate, wherein To perform illumination compensation on the current block, the instructions, when executed by the one or more processors, cause the one or more processors to: One or more illumination compensation parameters for the current block are derived using one or more neighboring blocks of the current block and one or more neighboring blocks of a reference block used for inter-frame prediction of the current block.

30. The non-transitory computer-readable medium of claim 27, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: Based on determining that the at least one motion vector candidate of the current block is obtained using bi-directional prediction, setting a value of at least one illumination compensation flag of the at least one motion vector candidate to a false value, the false value indicating that the illumination compensation is not to be applied.

31. The non-transitory computer-readable medium of claim 26, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: determining whether to use the illumination compensation flag for comparing the motion information of the current block with the motion information stored in the motion information candidate list, wherein: The motion information of the current block is added to the motion information candidate list based on the comparison.

32. The non-transitory computer-readable medium of claim 26, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: Based on determining that at least one motion vector candidate of the current block is not obtained by modifying one or more motion vectors of the one or more other blocks, inheriting the value of the illumination compensation flag for the at least one motion vector candidate, the value of the illumination compensation flag indicating that illumination compensation is to be applied.

33. The non-transitory computer-readable medium of claim 32, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to determine to perform illumination compensation on the current block based on the value of the illumination compensation flag of the at least one motion vector candidate, wherein To perform illumination compensation on the current block, the instructions, when executed by the one or more processors, cause the one or more processors to: One or more illumination compensation parameters for the current block are derived using one or more neighboring blocks of the current block and one or more neighboring blocks of a reference block used for inter-frame prediction of the current block.

34. The non-transitory computer-readable medium of claim 26, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: Based on determining that the at least one motion vector candidate of the current block is obtained by modifying the one or more motion vectors of the one or more other blocks, setting the value of at least one illumination compensation flag of the at least one motion vector candidate to a false value, the false value indicating that illumination compensation is not to be applied.

35. The non-transitory computer readable medium of claim 34, wherein: The one or more motion vectors of the one or more other blocks are paired motion vector candidates.

36. An apparatus for encoding video data, the apparatus comprising: Memory; as well as A processor implemented in circuitry and configured to: Get a current block of a picture of video data; determining to inherit a value of an illumination compensation flag associated with an additional block of the video data for at least one motion vector candidate of the current block, the value of the illumination compensation flag associated with the additional block indicating that illumination compensation is to be performed on the current block; determining whether at least one motion vector candidate of the current block is obtained by modifying one or more motion vectors of one or more other blocks of the video data; as well as determining whether to apply illumination compensation to the current block according to the illumination compensation flag based on whether the at least one motion vector candidate is obtained by modifying the one or more motion vectors; as well as An encoded video bitstream is generated that includes at least a portion of the current block.

Images