Local illumination compensation model merge mode

Abstract

Apparatuses and methods are disclosed including techniques for encoding and decoding video data. Techniques disclosed provide for coding and decoding of an inter-predicted video block in a local illumination compensation (LIC) model merge mode, including constructing a list of LIC model candidates, selecting an LIC model from the list of LIC model candidates, and applying the selected LIC model to a reference block of the video block to generate an illumination-compensated prediction block.

Description

[0001] LOCAL ILLUMINATION COMPENSATION MODEL MERGE MODE CROSS REFERENCE TO RELATED APPLICATIONS [1] This application claims the benefit of European Application No. 23306575.4, filed on September 22, 2023, which is incorporated herein by reference in its entirety. BACKGROUND [2] In predictive video coding, a currently coded (or decoded) video block from a video picture may be predicted based on a reference video block from another (reference) video picture that is temporally positioned before or after the current video picture. To compensate for changes in illumination across video pictures, the prediction of the current video block can be corrected via a local illumination compensation (LIC) process. In this LIC process, an LIC model is applied to the reference video block to generate an illumination-compensated prediction block. As implemented in recent video codecs, the LIC model is estimated based on template regions in the immediate neighborhoods of the current video block and its reference video block. However, if the correlation between corresponding samples from these template regions is not representative of the correlation between corresponding samples from the video block and its reference block, the estimated LIC model is limited in its capability to compensate for changes in illumination, and so applying it may impair coding efficiency. SUMMARY [3] Aspects disclosed in the present disclosure describe methods for encoding video data. The methods include coding a video block of the video data into a bitstream. The coding of the video block comprises constructing a list of LIC model candidates, selecting an LIC model from the list of LIC model candidates, and applying the selected LIC model to a reference block of the video block to generate a prediction block for the video block. Aspects disclosed in the present disclosure also describe methods for decoding video data. The methods include decoding a video block of the video data from a bitstream. The decoding of the video block comprises constructing a list of LIC model candidates, selecting an LIC model from the list of LIC model candidates, and applying the selected LIC model to a reference block of the video block to generate a prediction block for the video block. [4] Aspects disclosed in the present disclosure describe an apparatus for encoding video data. The apparatus comprises at least one processor and memory storing instructions. The instructions, when executed by the at least one processor, cause the apparatus to code a video block of the video data into a bitstream. The coding of the video block comprises constructing a list of LIC model candidates, selecting an LIC model from the list of LIC model candidates, and applying the selected LIC model to a reference block of the video block to generate a prediction block for the video block. Aspects disclosed in the present disclosure also describe an apparatus for decoding video data. The apparatus comprises at least one processor and memory storing instructions. The instructions, when executed by the at least one processor, cause the apparatus to decode a video block of the video data from a bitstream. The decoding of the video block comprises constructing a list of LIC model candidates, selecting an LIC model from the list of LIC model candidates, and applying the selected LIC model to a reference block of the video block to generate a prediction block for the video block. [5] Further aspects disclosed in the present disclosure describe a non-transitory computer- readable medium comprising instructions executable by at least one processor to perform methods for encoding video data. The methods include coding a video block of the video data into a bitstream. The coding of the video block comprises constructing a list of LIC model candidates, selecting an LIC model from the list of LIC model candidates, and applying the selected LIC model to a reference block of the video block to generate a prediction block for the video block. Aspects disclosed in the present disclosure also describe a non-transitory computer-readable medium comprising instructions executable by at least one processor to perform methods for decoding video data. The methods include decoding a video block of the video data from a bitstream. The decoding of the video block comprises constructing a list of LIC model candidates, selecting an LIC model from the list of LIC model candidates, and applying the selected LIC model to a reference block of the video block to generate a prediction block for the video block. [6] This Summary is provided to introduce a selection of concepts in a simplified form that is further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to limitations that solve any or all disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS [7] FIG. 1 is a block diagram of an example system, according to which aspects of the present embodiments can be implemented. [8] FIG.2 is a block diagram of an example video encoder, according to which aspects of the present embodiments can be implemented. [9] FIG. 3 is a block diagram of an example video decoder, according to which aspects of the present embodiments can be implemented.

[0010] FIG.4 is a diagram illustrating the derivation of an LIC model when operating in a bi- prediction mode, according to which aspects of the present embodiments can be implemented.

[0011] FIG. 5 is a diagram illustrating the derivation of an LIC model when operating in a subblock mode, according to which aspects of the present embodiments can be implemented.

[0012] FIG. 6 is a diagram illustrating positions of spatially adjacent neighboring blocks, according to which aspects of the present embodiments can be implemented.

[0013] FIG.7 is a diagram illustrating positions of spatially non-adjacent neighboring blocks, according to which aspects of the present embodiments can be implemented.

[0014] FIG.8 is a diagram illustrating positions of temporal neighboring blocks, according to which aspects of the present embodiments can be implemented.

[0015] FIG. 9 is a diagram illustrating positions of shifted temporal neighboring blocks, according to which aspects of the present embodiments can be implemented.

[0016] FIG.10 is a diagram illustrating positions of spatially non-adjacent neighboring blocks, according to which aspects of the present embodiments can be implemented.

[0017] FIG.11 is a flowchart illustrating the application of an LIC mode, according to which aspects of the present embodiments can be implemented.

[0018] FIG. 12 is a flowchart illustrating the application of an LIC model merge mode, according to which aspects of the present embodiments can be implemented.

[0019] FIG. 13 is a diagram illustrating a refinement of an LIC model, according to which aspects of the present embodiments can be implemented.

[0020] FIG. 14 is a diagram illustrating another refinement of an LIC model, according to which aspects of the present embodiments can be implemented.

[0021] FIG. 15 is a flowchart of an example method for encoding video data, according to which aspects of the present embodiments can be implemented.

[0022] FIG. 16 is a flowchart of an example method for decoding video data, according to which aspects of the present embodiments can be implemented. DETAILED DESCRIPTION

[0023] Apparatus and methods are disclosed in the present disclosure for encoding and decoding of video data. An LIC model merge mode is introduced herein, providing for the inheritance of LIC models from neighboring blocks of a video block being predicted in an inter-prediction mode (i.e., an inter-predicted block). Traditional systems and methods for predictive video coding are described next in reference to FIGS.1-3, followed by a description of aspects of the present disclosure, described in reference to FIGS.4-16.

[0024] FIG. 1 illustrates a block diagram of an example system 100. System 100 can be embodied as a device and can be configured to perform one or more of the aspects described in this application. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 100, singly or in combination, can be embodied in an integrated circuit, multiple integrated circuits, and / or discrete components. For example, in at least one embodiment, the processing 110 and encoder / decoder 130 elements of system 100 are distributed across multiple integrated circuits and / or discrete components. In various embodiments, the system 100 is communicatively coupled to other systems, or to other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports.

[0025] The system 100 includes at least one processor 110 that can be configured to execute instructions loaded therein for implementing, for example, the various aspects described in this application. Processor 110 can include embedded memory, input and output interfaces, and various other circuitries as known in the art. The system 100 includes at least one memory 120, such as a volatile memory device and / or a non-volatile memory device. System 100 includes a storage device 140, which can include non-volatile memory and / or volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives. The storage device 140 can be an internal storage device, an attached storage device, and / or a network accessible storage device, for example.

[0026] System 100 includes an encoder / decoder module 130 configured to process data to provide encoded video data or decoded video data. The encoder / decoder module 130 can include its own processor and memory. The encoder / decoder module 130 can be implemented as a separate element of system 100 or can be incorporated within processor 110 as a combination of hardware and / or software as known to those skilled in the art. Additionally, the encoder / decoder module 130 represents module(s) that can be implemented in a separate device to perform encoding and / or decoding functions.

[0027] Program code that is to be loaded into processor 110 or into encoder / decoder 130 to perform the various aspects described in this application can be stored in a storage device 140 and subsequently loaded into memory 120 for execution by processor 110. In accordance with various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 can store one or more of various items during the performance of the processes described in this application. Such stored items can include, but are not limited to, the input video, the decoded video or portions of the decoded video, the bitstream, matrices, variables, operational logic, and intermediate or final results from the processing of equations, formulas, operations.

[0028] In several embodiments, memory inside of the processor 110 and / or the encoder / decoder module 130 is used to store instructions and to provide working memory for processing functions that are needed during encoding or decoding. In other embodiments, however, memory external to the processing device (where, for example, the processing device can be either the processor 110 or the encoder / decoder module 130) can be used for one or more of these functions. The external memory can be the memory 120 and / or the storage device 140 that may comprise, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for video coding and decoding operations.

[0029] The input to the elements of system 100 can be provided through various input devices as indicated in block 105. Such input devices include, but are not limited to, (i) an RF portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Composite input terminal (COMP), (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0030] In various embodiments, the input devices of block 105 have associated respective input processing elements as known in the art. For example, the RF portion can be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select, for example, a signal frequency band which can be referred to as a channel in certain embodiments, (iv) demodulating the down converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements that perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner that performs some of these functions, including, for example, down-converting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to a baseband. In one set-top box embodiment, the RF portion and its associated input processing element receive an RF signal transmitted over a wired (for example, cable) medium, and perform frequency selection by filtering, down-converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Added elements can include inserting elements in between existing elements, for example, inserting amplifiers and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna.

[0031] Additionally, the USB and / or HDMI terminals can include respective interface processors for connecting system 100 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed- Solomon error correction, can be implemented, for example, within a separate input processing integrated circuit or within processor 110 as necessary. Similarly, aspects of USB or HDMI interface processing can be implemented within separate interface integrated circuits or within processor 110 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 110, and encoder / decoder 130 operating in combination with the memory and storage elements to process the data stream as necessary for presentation on an output device.

[0032] Various elements of system 100 can be provided within an integrated housing. Within the integrated housing, the various elements can be interconnected and transmit data therebetween using a suitable connection arrangement 115, for example, an internal bus as known in the art, including the I2C bus, wiring, and printed circuit boards.

[0033] The system 100 includes communication interface 150 that enables communication with other devices via communication channel 190. The communication interface 150 can include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 190. The communication interface 150 can include, but is not limited to, a modem or network card. The communication channel 190 can be implemented, for example, within a wired and / or a wireless medium.

[0034] Data can be streamed to the system 100, in various embodiments, using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signal of these embodiments is received over the communication channel 190 and the communication interface 150 which can be adapted for Wi-Fi communications. The communication channel 190 of these embodiments is typically connected to an access point or router that provides access to outside networks including the Internet for allowing streaming applications and other over-the-top communications. In other embodiments, data can be streamed to the system 100 using a set-top box that delivers the data over the HDMI connection of the input block 105 or data can be streamed to the system 100 using the RF connection of the input block 105.

[0035] The system 100 can provide an output signal to various output devices, including a display device 165, an audio device (e.g., speaker(s)) 175, and other peripheral devices 185. The other peripheral devices 185 include, in various examples of embodiments, one or more of a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide a function based on the output of the system 100. In various embodiments, control signals are communicated between the system 100 and the display device 165, the audio device 175, or the other peripheral devices 185 using signaling such as AV.link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, the output devices can be connected to system 100 using the communication channel 190 via the communication interface 150. The display device 165 and the audio device 175 can be integrated in a single unit with the other components of system 100 in an electronic device, for example, a television. In various embodiments, the display interface 160 includes a display driver, for example, a timing controller (T Con) chip.

[0036] Alternatively, the display device 165 and the audio device 175 can be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments in which the display device 165 and the audio device 175 are external components, the output signal can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0037] FIG. 2 illustrates a functional block diagram of an example video encoder 200. The video encoder 200 can be employed by the system 100 described in reference to FIG. 1. For example, the video encoder 200 can be an encoder that operates according to coding standards such as Advanced Video Coding (AVC, H.264 / MPEG-4 | ISO / IEC 14496-10), High Efficiency Video Coding (HEVC, ITU-T H.265 | ISO / IEC 23008-2), or Versatile Video Coding (VVC, Standard ITU-T H.266, ISO / IEC 23090-3, 2020).

[0038] Prior to undergoing encoding, the video data can be pre-processed by a precoding processor (not shown). Such pre-processing can include applying a color model transform to the color components of the input video frames (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0) or mapping the color components of the input video frames to obtain a signal distribution that is more resilient to compression (for instance, applying a histogram equalizer and / or a denoising filter to one or more of the video frames’ color components). The pre-processing can also include associating metadata with the video data that can be attached to the coded video bitstream.

[0039] In the encoder 200, a video frame is encoded by the encoder elements as generally described below. A picture (frame) of the original video to be encoded is partitioned into coding units (namely, original blocks) by an image partitioner 202. Typically, a coding unit (CU) contains a luminance block and respective chroma blocks, and so, generally, operations described herein as applied to a CU are applied to the luminance block and to the respective chroma blocks. Following partition 202, each CU can be encoded using an intra-prediction mode or an inter-prediction mode. In an intra-prediction mode, a prediction of the CU is performed by an intra-predictor 260. In the intra-prediction mode, the content of a CU in a frame is predicted based on content from one or more other CUs of the same frame, using the other CUs’ reconstructed version (available from the adder 255 output). In an inter-prediction mode, motion estimation and motion compensation are performed by a motion estimator 275 and a motion compensator 270, respectively. In the inter-prediction mode, the content of a CU in a frame is predicted based on content from one or more other CUs of neighboring frames, using the other CUs’ reconstructed versions (available from the reference picture buffer 280). The encoder decides 205 which prediction result (one obtained through operations in the intra- prediction mode 260 or one obtained through operations in the inter-prediction mode 270, 275) to use for encoding a CU, and indicates the selected prediction mode by a prediction mode flag, for example. The selected prediction result may then be enhanced (e.g., filtered) by a prediction enhancer 285, outputting a respective prediction block. Once a prediction block is generated for each CU, a respective residual block is calculated, for example, by subtracting 210 the predicted CU (i.e., prediction block) from the CU (i.e., original block).

[0040] A CU’s respective residual block or a partition thereof (i.e., a transform block) is then transformed into a coefficient block by a transformer 220 – that is, residual samples of the transform block are transformed into transform coefficients of the coefficient block. The resulting coefficient block is quantized by a quantizer 230. An entropy encoder 245 is next employed to entropy-encode the quantized coefficient block and respective coding parameters (e.g., syntax elements including motion vectors and other control data). Hence, the entropy- encoded quantized coefficient blocks and respective encoding parameters associated with each video frame of the original video are packed into the bitstream of the coded video data.

[0041] Along with the coding of the original video blocks, as described above, the encoder 200 reconstructs the coded original blocks to provide references for future predictions. Accordingly, quantized coefficient blocks (provided by the quantizer 230) are de-quantized, by an inverse quantizer 240, and then inverse transformed, by an inverse transformer 250, to reconstruct (decode) the residual blocks of respective original blocks. Adding 255 the reconstructed residual blocks to respective prediction blocks results in respective reconstructed original blocks. In-loop filters 265 can then be applied to the reconstructed picture (formed by the reconstructed original blocks), performing, for example, deblocking filtering and / or sample adaptive offset (SAO) filtering to reduce encoding artifacts. The filtered reconstructed picture can then be stored in the reference picture buffer 280, available for future predictions in an inter-prediction mode. Thus, the encoder 200 also performs decoding operations 240, 250 through which the encoded pictures (frames) are reconstructed. The reconstructed pictures can then be stored in the reference picture buffer 280 and be used to facilitate motion estimation 275 and compensation 270, as explained above.

[0042] FIG. 3 illustrates a functional block diagram of an example video decoder 300. The video decoder 300 can be employed by the system 100 described in reference to FIG. 1. Generally, operational aspects of the video decoder 300 are reciprocal to operational aspects of the video encoder 200. In the decoder 300, the bitstream of coded video data, generated by the video encoder 200, is first entropy-decoded by an entropy decoder 330, decoding from the bitstream the quantized coefficient blocks and various coding parameters. The quantized coefficient blocks are de-quantized, by an inverse quantizer 340, and then are inverse transformed, by an inverse transformer 350, to decode (reconstruct) respective residual blocks. Adding 355 the reconstructed residual blocks to respective prediction blocks results in respective reconstructed original blocks. Depending on the selected prediction mode, a predicted original block can be obtained 370 from an intra-predictor 360 or from a motion compensator 375 and may then be enhanced (e.g., filtered) by a prediction enhancer 390, generating a prediction block. In-loop filters 365 can be applied to the reconstructed picture (formed by the reconstructed original blocks), outputting a reconstructed (decoded) video frame. The filtered reconstructed picture is also stored in a reference picture buffer 380 to facilitate motion compensation 375.

[0043] A post-decoding processor (not shown) can further process the reconstructed video. For example, post-decoding processing can include an inverse color model transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or an inverse mapping to reverse the mapping process performed by the pre-encoding processor. The post-decoding processor can use metadata that were derived by the pre-encoding processor and / or were signaled in the video bitstream.

[0044] Aspects disclosed herein are described in reference to video blocks (i.e., blocks) of a video picture (i.e., frame). However, the described aspects are similarly applicable to any region of the video frame to which coding tools may be applied by an encoder 200 or by a decoder 300. Generally, aspects described herein may be applied to a video data region, formed by a video partition, of any shape or size. The video region may be a Coding Tree Unit (CTU), or a CU (for which an inter-prediction or an intra-prediction can be applied), or a partition thereof, each can include a luma component, Y, and chroma components, Cr and Cb (denoted herein also by C).

[0045] LIC has been introduced into recent video codecs. LIC aims at compensating for local differences in illumination across frames that can cause discrepancies between the values of corresponding pixels in a block currently being coded or decoded (i.e., a current block) and a reference block that serves for its prediction (i.e., a prediction block). Particularly, for a current block that is being predicted in an inter-prediction mode, LIC, when applied, modifies the respective reference block into an illumination-compensated prediction block by applying an LIC model to the pixel values of the reference block to compensate for illumination differences. Typically, the parameters of an LIC model are estimated based on neighboring reconstructed samples. That is, for a current block, the parameters of the LIC model are estimated based on reconstructed samples in the neighborhood of the current block (namely, a current template) and corresponding reconstructed samples in the neighborhood of the reference block (namely, a reference template). To obtain an LIC model that accurately models the illumination discrepancies between a current block and its reference block, more than one LIC model should be considered. In doing so, compression efficiency can be improved (that is, the bitrate can be reduced while maintaining the same quality, or equivalently the quality can be improved while maintaining the same bitrate). LIC, as applied in recent video codecs (when relying on one LIC model), is described next.

[0046] In the inter-prediction mode of operation, as implemented by the exploratory coding model (ECM), an LIC is employed to compensate for temporal changes in illumination that may occur across pictures (see, “CE4-3.1a and CE4-3.1b: Unidirectional local illumination compensation with affine prediction,” Document JVET-O0066, 15th Meeting: Gothenburg, SE, 3–12 Jul. 2019). Therein, the LIC model is a linear model that is applied to the samples of a reference block to predict the values of corresponding samples of the current block. The linear model can be expressed as follows: ^^^ ^^, ^^^ ൌ ^^ ൈ ^^^൫^^ ^ ^^௫, ^^ ^ ^^௬൯^ ^^ (1)where ^^^ ^^, ^^^ is a where ^^^൫ ^^ ^ ^^௫, ^^ ^ ^^௬൯ is a pixel value of a corresponding sample at location ൫ ^^ ^ ^^௫, ^^ ^ in which൫^^௫, ^^௬൯is the motion vector obtained by inter-prediction;and where ^ ^^, ^^^ are the linear in which ^^ denotes a scaling parameter and ^^ denotes an offset parameter. The derivation of the LIC model – that is, the estimation of the linear model’s parameters ^ ^^, ^^^ – can be done by using a least-mean-square-error (LMSE) estimation method. In the LMSE method, the estimates for parameters ^ ^^, ^^^ are those parameters, ^ ^^^, ^^^^, that minimize the LMSE. The LMSE can be expressed as follows:^^ ^^ ^^ ^^ ൌ ∑ே^ୀ^^ ^^^ ^^^, ^^^^ െ൫^^ ൈ ^^^൫^^^^ ^^௫, ^^^^ ^^௬൯^ ^^൯^ଶ , (2)where ^^ error in equation (2) can be computed as follows: ಿேൈ^ ் ௫^,௬^ ൈ ^൫௫^ା௩^,௬^ା௩^൯ି ^ಿ ಿ^ ^ ்^௫^,௬^^ ൈ^ ^൫௫ ା௩ ,௬ ା௩ ൯^సభ ^సభ ^సభ ^ ^ ^ ^^^^ ൌ and where sample associated with the current block (i.e., the current template) at location ^ ^^^, ^^^^; and where ^^^൫ ^^^^ ^^௫, ^^^^ ^^௬൯ is a pixel value of a template sample associated with the reference at location ൫ ^^^^ ^^௫, ^^ ^ ^^ ൯. Note that ^^^ ^^ , ^^ ^ spatially ^ ௬ ^ ^corresponds to ^^^൫ ^^^^ ^^௫, ^^^^ ^^௬൯ via the vector ൫ ^^௫, ^^௬൯.

[0047] Hence, when an encoder 200) respect to a current block, an LIC flag is signaled to indicate to a decoder (e.g., decoder 300) that a prediction block of the current block (i.e., its reference block) is modified to compensate for illumination changes. If LIC is not applicable for the current block, the LIC flag is not signaled. When the LIC flag is signaled, it can be added to the motion information of the current block and it can be context- coded with a single context. Notice that the LIC model can be independently derived by the decoder in the same manner it is derived by the encoder. This is because the parameters of the LIC model are estimated based on corresponding reconstructed samples from the current template T and from the reference template Tr, as discussed above with respect to equations (1)-(4). Therefore, the encoder need not signal the LIC model itself (e.g., parameters ^^^ and ^^^) to the decoder.

[0048] LIC can also be applied to bi-predicted blocks (see, "EE2-Test2.7: Improvements on local illumination compensation", Document JVET-AD0213, 30th Meeting, Antalya, TR, 21– 28 Apr.2023, hereinafter “JVET-AD0213”), as next described with reference to FIG.4.

[0049] FIG.4 is a diagram illustrating the derivation of an LIC model when operating in a bi- prediction mode 400. In the example of FIG.4, a currently bi-predictively encoded (or decoded) block 425 of a current picture 420 is shown together with a template region T 450, including reconstructed samples located along the block’s left boundary and reconstructed samples located along the block’s top boundary. Reference pictures, L0410 and L1430, are also shown, including respective reference blocks 415, 435, used to predict the current block 425 in an inter-prediction mode. T0440 and T1460 are template regions in respective reference frames L0410 and L1430, that correspond to the template region T 450 of the current picture. In this bi-predictive coding of the current block 425, two LIC models are derived. The first model’s parameters, ^ ^^^, ^^^^, are estimated based on the samples in T and their corresponding samples in T0, pointed to by the motion vector ^^ ^^^ൌ ^ ^^௫^, ^^௬^^ 470 and the second model’s parameters, ^ ^^^, ^^^^, are estimated based on the samples in T and their corresponding samples in T1, pointed to by the motion vector ^^ ^^^ൌ ^ ^^௫^, ^^௬^^ 480. Hence, by minimizing the LMSE error in equation (2), the parameter estimates of the first model or the second model, denoted^ ^^^^ / ଶ, ^^^^ / ଶ^ for simplicity, can be computed as follows:ಿேൈ^ ் ௫^,௬^ ൈ்బ / భ^௫^ା௩బ / భ,௬^ା௩బ ^ି ^ಿ ಿ^ ^ / భ ்^௫^,௬^^ ൈ^ ்బ / ^௫ ା௩బ / భ,௬ ା௩బ / భ^^^^ ൌ^సభ ^ ^ ^సభ ^సభ భ ^ ^ ^ ^and sample associated with the current block at location ^ ^^^, ^^^^; and where ^^^ / ^^ / ^൫^^^^ ^^௫, ^^^^ ^^^ / ^is a pixel value of a template sample at location ൫ ^^^^ ^^^ / ^ ^ / ^௫ , ^^^^ ^^௬൯. Note that ^^^ ^^^, ^^^^ spatially corresponds to ^^^ / ^൫ ^^^^ ^^^ / ^, ^^ ^ ^^^ / ^ ^ / ^௬ vector ൫ ^^௫, ^^௬

[0050] The derivation of the LIC models (e.g., according to equations (5) and (6)) may be computationally demanding. To decrease computational complexity, the number of used template samples can be reduced by subsampling T 450, T0440, and T1460. For example, for a block size larger than 8, the template samples may be subsampled using a sampling rate of 2:1, that is, only the patterned samples in FIG. 4 are used to estimate ^^ and ^^. Note that the LIC model (independently or combinedly) can be derived from the color components of the video block – e.g., the luma component, Y, and the chroma components, Cr and Cb.

[0051] Once the two LIC models are derived, they can be applied to the reference blocks ^^^^415 and ^^^^435 (in respective reference pictures L0410 and L1430) that are pointed to by respective motion vectors ^^ ^^^ൌ ^ ^^௫^, ^^௬^^ 470 and ^^ ^^^ൌ ^ ^^௫^, ^^௬^^ 480, as follows: ^^^^ ^^, ^^^ ൌ ^^^^ൈ ^^^^൫^^ ^ ^^௫^, ^^ ^ ^^௬^൯^ ^^^^(7) where ^^^ and ^^^^are and ^^^^and ^^^^are the estimates of the scaling and the offset parameters of the second LIC model. Next, the bi-prediction block ^^^ ^^, ^^^ is obtained by combining the two illumination- compensated prediction blocks ^^^^ ^^, ^^^ and ^^^^ ^^, ^^^, as follows: ^^^ ^^, ^^^ ൌ ^1 െ ^^^ ൈ ^^^^ ^^, ^^^ ^ ^^ ൈ ^^^^ ^^, ^^^ (9) where ^^ denotes a weight parameter. When applying bi-prediction with a CU-based weighting (BCW) the weight is indicated by the BCW index.

[0052] The estimation of the parameters ^ ^^^, ^^^^ and ^ ^^^, ^^^^, to derive ^ ^^^^, ^^^^^ and ^ ^^^^, ^^^^^,can be done iteratively, for example, as described next with respect to equations (10)-(13). Ina first iteration, the parameters ^ ^^^^, ^^^^^ are computed according to equations (5) and (6), andthen the pixel values of the template samples in T are updated by subtracting the corresponding samples in ^^^′, which is derived using the computed parameters as follows: ^^′^ ^^, ^^^ ൌ ^ ^^ െ ^1 െ ^^^ ൈ ^^^′൫^^ ^ ^^௫^, ^^ ^ ^^௬^൯^⁄ ^^ , (10)^^^′൫ ^^ ^ ^^௫^, ^^ ^ ^^௬^൯ ൌ ^^^^ ൈ ^^^൫ ^^ ^ ^^௫^, ^^ ^ ^^௬^൯ ^ ^^^^ . (11)In a second using the updated template T’, and then the pixel values of the template samples in T are updated again by subtracting the corresponding samples in ^^^′, which is derived using the computed parameters as follows: ^^′′^ ^^, ^^^ ൌ ^ ^^ െ ^^ ൈ ^^^′൫^^ ^ ^^௫^, ^^ ^ ^^௬^൯^⁄ ^1 െ ^^^ , (12)Then, in a using the updated T’’.

[0053] As described in JVET-AD0213, according to the current LIC design, when using bi- prediction with an AMVP mode, one flag is signaled to indicate an LIC mode; while, when using bi-prediction with a merge mode, the flag is inherited from the used merge candidate. Note that an LIC operation is disabled when certain tools are applied, such as decoder-side motion vector refinement (DMVR) (including multi-pass DMVR, adaptive DMVR and affine DMVR) and bi-directional optical flow (BDOF).

[0054] Similarly, LIC can be applied to subblocks of the current block (e.g., when operating in a subblock mode such as an affine mode). In this case, the LIC model is derived from template regions associated with respective subblocks, as further described with respect to FIG. 5.

[0055] FIG. 5 is a diagram illustrating the derivation of an LIC model when operating in a subblock mode 500. As shown in the example of FIG.5, a current block 515 of a current picture 510 is partitioned to subblocks, of which subblocks A, B, C, D, E, F, and G are denoted. These subblocks are associated with respective reference subblocks of a reference picture 520, of which reference subblocks AREF, BREF, CREF, DREF, EREF, FREF, and GREF are denoted. The reference subblocks are pointed to by respective motion vectors (illustrated by the arrows). The gray areas to the left and on the top of the current block 515 indicate the current template, and the gray areas to the left and on the top of the reference subblocks indicate the reference template. In this case for example, the template samples above subblock A of the current block 515 correspond to the template samples above subblock AREF; the template samples to the left of subblock A of the current block 515 correspond to the template samples to the left of subblock AREF; the template samples above subblock B of the current block 515 correspond to the template samples above subblock BREF; the template samples to the left of subblock E of the current block 515 correspond to the template samples to the left of subblock EREF; and so on. The corresponding samples in the reference template and the current template can be used to estimate the LIC model (e.g., according to equations (3) and (4)).

[0056] Furthermore, LIC can be applied also to an intra block copy (IBC) mode (see, "EE2- 3.6: IBC-CIIP, IBC-GPM, and IBC-LIC", Document JVET-AC0112, 29th Meeting, by teleconference, 11–20 Jan.2023). The application of IBC with LIC (i.e., in an IBC-LIC mode) aims at compensating for illumination variations within the same picture – that is, illumination variations between the CU being coded and its IBC prediction block. In this case, the LIC model is derived in the same manner as described above for inter-prediction, except that the reference template is associated with a reference block that is pointed to by the block vector given by the IBC-LIC mode. IBC-LIC can be applied together with an IBC AMVP mode and with an IBC merge mode. For the IBC AMVP mode, an IBC-LIC flag is signaled to indicate the application of LIC. For the IBC merge mode, the IBC-LIC flag is inferred from a merge candidate.

[0057] In recent video codecs, template-based techniques were introduced to predict chroma samples from corresponding luma samples – the so-called cross-component prediction (CCP). Various CCP models are adopted in ECM, including a cross-component linear model (CCLM), a multi-model CCLM (MMLM), a convolutional cross-component model (CCCM), and a gradient linear model (GLM). These CCP models exploit correlation that is likely to exist between corresponding luma and chroma regions associated with a current block. To increase efficiency (e.g., by reducing signaling overhead), cross-component merge (CCMerge) mode has been introduced as a new CCP mode (see, “EE2-1.6: Non-local cross-component prediction and cross-component merge mode,” Document JVET-AD0188, 30th Meeting, Antalya, TR, 21–28 Apr. 2023, hereinafter “JVET-AD0188”). The CCMerge mode can be used to inherit one of the CCP models used by spatial neighboring blocks (JVET-AD0188) or by temporal neighboring blocks (see, “EE2-2.7: Cross-component prediction merge mode with temporal candidates,” Document JVET-AE0043, 31st Meeting, Geneva, CH, 11–19 Jul. 2023, hereinafter “JVET-AE0043”), as further described below.

[0058] In a CCMerge mode, the CCP model that is used for predicting a chroma block can be 1) inherited from that block’s spatially adjacent or non-adjacent neighboring blocks; 2) a history-based model; or 3) a default model. To that end, a list of candidates is created for a current block, which can include CCP models such as CCLM, MMLM, CCCM, GLM, chroma fusion, and CCMerge modes. To construct this list, first spatially adjacent and non-adjacent neighboring blocks are considered and (if available) their respective CCP models are added to the list. Next, if the list has not yet reached its maximum size, CCP models listed in a history- based table are added into the list, followed by default CCP models. To avoid having redundant CCP models in the list, pruning operations can be applied. After the list is constructed, the CCP models in the list can be reordered depending on the cost of applying these models to a template region of the current block (e.g., using sum of absolute differences (SAD) metric).

[0059] FIG.6 is a diagram illustrating positions of spatially adjacent neighboring blocks 600. The spatially adjacent blocks are those that overlap the positions indicated by B0, B1, B2, A0, and A1, shown in FIG.6. These spatially adjacent blocks are considered in the following order: B1, A1, B0, A0, and then B2; and, if a CCP model is used by each of these blocks, it can be added to the list as a CPP model candidate.

[0060] FIG. 7 is a diagram illustrating positions of spatially non-adjacent neighboring blocks 700. In the example of FIG. 7, 59 positions are shown, indicated by dots arranged relative to the current block, at distances that can be determined based on the width and the height of the current block. The spatially non-adjacent blocks are those blocks that overlap these positions. Thus, if the list of CCP model candidates is not yet full after adding the CCP models of the spatially adjacent neighboring blocks, the CCP models (if available) of the spatially non- adjacent neighboring blocks can be added to the list. The order in which the spatially non- adjacent neighboring blocks are considered can be based on their distance from the current block and / or their frequency of occurrence.

[0061] As mentioned above, CCP model candidates can be history-based candidates that are drawn from a history-based table. A history-based table (e.g., of size six) can be maintained to include the recently used CCP models, and the table can be periodically reset (e.g., at the beginning of each CTU row). If the list of CCP model candidates is not yet full after adding the CCP model candidates of the spatially adjacent neighboring blocks and of the non-adjacent neighboring blocks, the CCP models in the history-based table can be added into the list.

[0062] Next, if the CCP model candidate list is still not full (after adding available CCP models of the spatially adjacent neighboring blocks and of the non-adjacent neighboring blocks as well as history-based CCP models), default candidates can be added. Accordingly, if the list does not include a CCLM candidate, default CCLM candidates can be added to the list with respective scaling parameters of 0, 1 / 8, -1 / 8, 2 / 8, -2 / 8, 3 / 8, -3 / 8, 4 / 8, -4 / 8, 5 / 8, -5 / 8, and 6 / 8. Otherwise, default CCLM candidates can be added to the list with a scaling parameter of 0, and with scaling parameters that equal to the scaling parameter of the first CCLM candidate in the list plus 1 / 8, -1 / 8, 2 / 8, -2 / 8, 3 / 8, -3 / 8, 4 / 8, -4 / 8, 5 / 8, -5 / 8, and 6 / 8. The offset parameter can be derived as shown below in equation (14).

[0063] Thus, a chroma block (of a current block) can be predicted based on a CCP model that is inherited from a neighboring block that is coded in CCLM, CCCM, GLM, chroma fusion, or CCMerge modes (or the chroma block can be predicted based on multiple CCP models that are inherited from a neighboring block that is coded in MMLM or multi model CCCM). The inheritance is performed as follows. ^ The chroma block can inherit the following information from a candidate in the list of CCP model candidates: the model’s parameters, the GLM pattern index (when the CCP model is GLM), and the classification threshold (when the CPP is MMLM or multi model CCCM). ^ The offset parameter (if defined for the CCP model) is not inherited, but is derived as follows: ^^ ൌ ^^^௩^െ ^^^^^^^^௧^ௗൈ ^^^௩^, (14) where ^^^^^^^^௧^ௗis the reconstructed luma sample values, and where ^^^௩^is the average of neighboring reconstructed chroma sample values.

[0064] Hence, to indicate whether a CCMerge mode is used to predict the chroma samples of a current block, a flag can be signaled (e.g., after the signaling of cclm_mode_flag syntax element). If this flag is set to true (i.e., a CCMerge mode is used), an index is further signaled to indicate which candidate from the list is used by the current block. However, if the maximum number of allowed candidates is modified to 1, the index need not to be signaled. Currently, the maximum number of allowed candidates is set to 12 as a default. The signaled index can be shared by the Cb and Cr color components. Each bin of an index is context-coded with a separate context. Note that a CCMerge mode is not allowed for the intra-prediction of a chroma block when the CU (including the chroma block) is coded by intra sub-partitions (ISP) with a single tree, or when the size of the chroma block is less than or equal to 16.

[0065] As mentioned above, CCP models of temporal neighboring blocks can also be added to the list of CCP model candidates (see, JVET-AE0043). In this case, CCP models of temporal neighboring blocks and shifted temporal neighboring blocks can be added into the list. CCP models of temporal neighboring blocks can be added to the list after those of spatially adjacent neighboring blocks have been added, while CCP models of shifted temporal neighboring blocks can be added to the list after those of the spatially non-adjacent neighboring blocks have been added. CCP model candidates that are entered into the list can then be reordered based on the cost of applying them to predict chroma samples in a template region associated with the current block (e.g., using an SAD metric), thereby, further improving signaling efficiency. The candidates can be stored on an 8x8 grid and provided for later prediction. For a 16x16 prediction block, only candidate information on the top-left 8x8 grid is stored for access. Temporal and shifted temporal neighboring blocks are further described with respect to FIG.8 and FIG.9.

[0066] FIG.8 is a diagram illustrating positions of temporal neighboring blocks 800. Temporal neighboring blocks are selected from the collocated picture 810 relative to the collocated position of the current block 820. A collocated picture 810 is defined by the index of the collocated reference picture to be used for derivation of the collocated CU, and the index is explicitly signaled in the slice header (as in ECM-8.0). In the example of FIG. 8, two groups of positions are highlighted: C0 that are denoted by black boundaries and C1 that are denoted by gray boundaries. These positions of temporal neighboring blocks are also numbered to indicate the order in which respective CCP model candidates (if available) are added to the list of CCP model candidates. The order is defined as: C01, C02, … C010. If C0i is not available (i.e., the position of C0iis outside of picture / slice boundary, or outside of the current CTU row) and C1iis available, C1iwill be used to replace C0i, where 1 ^ ^^ ^ 10. Otherwise (both C0iand C1i are not available), the next position C0i+1 is checked.

[0067] FIG. 9 is a diagram illustrating positions of shifted temporal neighboring blocks 900. These positions are selected from the collocated picture 910 relative to the collocated position of the current block 920. However, in this case, the position of the collocated block 920 is shifted by a selected neighboring motion vector (as denoted by the arrows). Consequently, the positions of C0i and C1i are also shifted by the same motion vector. The order in which the CCP models of the shifted temporal neighboring blocks are added to the list of CCP model candidates is the same as that of the temporal neighboring blocks. The motion vector is selected from the motion vectors of blocks that neighbor the current chroma block 930 – that is, neighboring blocks that overlap positions A0, A1, B0, B1, and B2. L0 motion vector 940 and / or L1 motion vector 950 may be associated with each of these neighboring blocks. The order in which the motion vectors of the neighboring blocks are checked is as follows: L0B1, L1B1, L0A1, L1A1, L0B0, L1B0, L0A0, L1A0, L0B2, and L1B2. The first motion vector that uses the collocated picture 910 as the reference picture is selected. If no such motion vector is found, no CCP models of shifted temporal neighboring blocks are added to the list.

[0068] A merge mode is also applied when IBC-LIC is used, namely, an IBC-LIC model merge mode. In the IBC-LIC model merge mode, the IBC-LIC model parameters are inherited from previously coded blocks (see, Non-EE2: IBC-LIC Model Merge mode", Document JVET- AE0073, 31st Meeting, Geneva, CH, 11–19 Jul.2023). More specifically, for a current block, an LIC model can be obtained as follows. a) A list of LIC model candidates is constructed. This list consists of LIC models used by spatially adjacent and non-adjacent neighbors, of history-based models, and of default models. The size of the candidate list is predetermined (e.g., twelve). Similar to the CCMerge mode, LIC models are collected from blocks (previously coded in an IBC- LIC mode and in an IBC-LIC model merge mode) that are located at adjacent and non- adjacent positions (e.g., positions of spatially non-adjacent neighboring blocks 1000 are shown in FIG. 10). Additionally, a history IBC-LIC model table (e.g., with a size of six) is maintained. The LIC models from spatial neighbors and from the history IBC- LIC model table are added to the list of IBC-LIC model merge candidates. If the list is not full, default models can then be added to the list. To avoid redundant models, a pruning operation can be applied. b) The offset parameter of an IBC-LIC model candidate can be adjusted. The adjustment, ^^ᇱ, can be calculated as follows: ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ൌ∑ே^ୀ^^ ^^ ^^ ^^^െ ^ ^^ ∗ ^^ ^^ ^^^^ ^^^^ (15) where ^^ ^^ ^^^and ^^ ^^ ^^^are pixel values of reconstructed samples from the templates of the current block and the reference block, respectively; and where ^^ is the total number of samples. The offset ^^ is then adjusted by ^^ᇱ, that is, ^^ ൌ ^^ ^ ^^ᇱ. c) An IBC-LIC model is selected from the candidate list and its index is signaled in the bitstream.

[0069] A flag is signaled to indicate whether the IBC-LIC model merge mode is applied or not. If this flag is set to true (i.e., the IBC-LIC model merge mode is applied), an index is further signaled to indicate which candidate from the list is used by the current block. In the IBC- AMVP mode, this flag is signaled when the flag indicating operation in IBC-LIC mode is set to true. In the IBC-Merge mode, this flag is only signaled when the current block is not coded in IBC-CIIP, IBC-GPM, TM-Merge, or skip modes. If the IBC-LIC model merge flag is set to true, the regular inheritance of an IBC-LIC flag will not be applied, and the current block is treated as a regular IBC-LIC by other blocks.

[0070] Aspects disclosed herein enhance coding efficiency of inter-coded blocks that use LIC. In the state-of-the-art LIC design, when LIC is used, only a single LIC model can be applied – that is the model that is estimated based on template regions in the immediate neighborhood of a current block and its reference block. If that estimated model poorly compensates for illumination discrepancies, then applying it to correct the prediction of the current block can introduce additional errors that may penalize coding efficiency. Particularly, when a merge mode is used, the LIC flag of the current block is inherited from a merge candidate. Regardless of whether it is optimal, the prediction of the current block is then corrected using the LIC model of that merge candidate. As described above, in a CCMerge mode an inherited CCP model is applied to predict chroma samples of a block, and a similar concept has been applied when using an IBC-LIC mode. However, the concept of inheriting an LIC model from previously coded blocks is not applied to an inter-predicted block.

[0071] According to aspects, an LIC model merge mode is used for inter-predicted blocks. In an LIC model merge mode, an LIC model is inherited from previously coded blocks to compensate for illumination discrepancies. To that end, an LIC model merge mode can be explicitly signaled as a sub-mode of an LIC mode both when using the AMVP mode or the merge mode. When the LIC model merge mode is used, an LIC model candidate list is constructed; then, an LIC model is selected from the list, and (optionally) refined before it is applied to compensate for illumination changes. Additionally, the LIC model merge mode can be used in combination with a subblock mode and a bi-predictive block mode. Described next are: the general operation of the disclosed LIC model merge mode; the construction of an LIC model candidate list; the adaptions of the LIC model merge mode to a subblock mode and to a bi-predictive block mode; and the refinement of the inherited LIC model parameters.

[0072] LIC model merge mode.

[0073] FIG.11 is a flowchart illustrating the application of an LIC mode 1100. In the example of FIG.11, an LIC model (e.g., a linear model) is derived and applied to the coding of an inter- predicted block. Accordingly, in step 1110, it is determined whether the LIC flag, lic_flag (that is set and signaled in the bitstream by the encoder and that is decoded from the bitstream by the decoder), is set to true for a current block that is predicted in an inter-prediction mode. When LIC is enabled for an inter-predicted block, in step 1120, the current template (including reconstructed neighboring samples of the current block) and the reference template (including reconstructed neighboring samples of the reference block) are obtained. Based on these templates, in step 1130, an LIC model is derived – e.g., the parameters ( ^^, ^^) of a linear model are derived (i.e., estimated) as described with respect to equations (1)-(4). Then, in step 1140, to compensate for illumination changes, the derived LIC model is applied to samples in the reference block, generating therefrom an illumination-compensated prediction block that serves as a prediction of the current block.

[0074] As illustrated in FIG. 11, a single LIC model is applied based on the current and the reference templates. However, such an LIC model may not be sufficient to correct the illumination changes experienced by the current block. This is because this LIC model is derived based on a limited number of corresponding samples from the current and the reference templates. Moreover, the correlation between the corresponding samples from the reference template and from the current template may not be representative of the correlation between corresponding samples from the reference and the current block.

[0075] To mitigate these shortcomings, an LIC model merge mode is introduced herein, where an LIC model can be inherited from previously coded blocks for which the LIC flag is enabled. In this mode, multiple LIC models can be collected from the previously coded blocks and be considered as candidates for the current block’s LIC model.

[0076] The disclosed herein LIC model merge mode can be explicitly signaled as a sub-mode of an LIC mode both when using the AMVP mode or the merge mode. A flag, lic_model_merge_flag, can be signaled for an inter-predicted block, conditioned on the LIC flag, to indicate whether the LIC model merge mode is applied or not. If the lic_model_merge_flag is set to true, an index is further signaled to indicate which candidate LIC model is used by the current block. However, in an aspect, the index can be implicitly derived (e.g., a predetermined candidate can be used such as the first available candidate in the list of LIC model candidates) – and so the index need not be signaled. In the AMVP mode, the flag lic_model_merge_flag is signaled when the LIC flag is true. In the merge mode, the flag lic_model_merge_flag is only signaled when the LIC flag of the current block inferred from the merge candidate is true.

[0077] FIG. 12 is a flowchart illustrating the application of an LIC model merge mode 1200. In the example of FIG.12, in step 1210, it is determined whether the LIC model merge mode flag, lic_model_merge_flag (that is set and may be signaled in the bitstream by the encoder and that may be decoded from the bitstream by the decoder), is set to true for a current block being coded in an inter-prediction mode. The lic_model_merge_flag indicates whether the LIC model merge mode is applied to an inter-predicted block. If the lic_model_merge_flag is set to false, in steps 1220 and 1230, a regular LIC process is performed (as in FIG. 11). That is, in step 1220, the current template and the reference template are obtained. Based on these templates, in step 1230, the LIC model is derived – e.g., the parameters ( ^^, ^^) of a linear model are derived (i.e., estimated) as described with respect to equations (1)-(4). If lic_model_merge_flag is set to true, the LIC model merge mode is applied in steps 1240 and 1250. An LIC model candidate list is constructed in step 1240. Then, in step 1250, an LIC model is selected from the list of LIC model candidates. The selected LIC model is indicated by an index (that is determined and may be signaled in the bitstream by the encoder and that may be decoded from the bitstream by the decoder). Then, in step 1260, to compensate for illumination changes, the LIC model (derived in step 1230 or selected from the list of LIC model candidates in step 1250) is applied to samples in the reference block, generating therefrom an illumination-compensated prediction block that serves as a prediction of the current block.

[0078] According to aspects, when an AMVP mode is applied, the LIC model merge mode can be signaled before the signaling of the LIC flag, and the LIC flag is only signaled if the LIC model merge mode is false. In this case, if lic_model_merge_flag is true, the proposed LIC model merge mode is applied to the current block, and the LIC flag need not be signaled; if lic_model_merge_flag is false, then the LIC flag is signaled, and the LIC process is performed on the current block if the LIC flag is true; if both the lic_model_merge_flag and the LIC flag are false, the LIC process is not performed on the current block. When a merge mode is applied, the flag lic_model_merge_flag is only signaled when the current block is not coded in CIIP, GPM, TM-Merge, or skip modes, regardless of the inferred LIC flag from the merge candidate. In the merge mode, if the LIC model merge flag, lic_model_merge_flag, is true, the regular inheritance of an LIC flag from merge candidates is not applied, meaning that the current block performs the LIC process even if the LIC flag inherited from its merge candidate is false.

[0079] The lic_model_merge_flag may be derived implicitly. For example, it can be assumed to be always true or to be equal to the LIC flag. The lic_model_merge_flag can also be derived based on coding parameters or other conditions, such as the current block size or the number of neighboring blocks (or reconstructed samples) with LIC enabled. For example, the LIC model merge mode can be used only for the merge mode or can be used only for the AMVP mode. In another example, the LIC model merge mode can be disabled for blocks of a certain size, for example, blocks with less (or more) than N number of luma samples (e.g., N=32).

[0080] In an aspect, when either the LIC model merge mode or a regular LIC mode is applied to a current block, the LIC flag is stored as true in the motion information of this block, and can be used for later prediction. The current block is treated as using a regular LIC mode by future blocks that would use the LIC model of the current block as a candidate in their list of LIC model candidates.

[0081] Construction of the LIC model candidate list.

[0082] The list of LIC model candidates can be constructed out of: LIC models of spatially adjacent neighboring blocks; LIC models of spatially non-adjacent neighboring blocks; LIC models of temporal neighboring blocks; LIC models of shifted temporal neighboring blocks; history-based LIC models; default LIC models; or a combination thereof. The order in which the candidates may be added to the list may differ. For example, LIC models from shifted temporal neighboring blocks may be included in the list after history-based LIC models. In an aspect, the maximum number of candidates for each candidate type can be defined. For example, the maximum number of LIC models of spatial neighboring blocks can be set to 5 and the maximum number of LIC models of temporal neighboring blocks can be set to 2.

[0083] To avoid including redundant LIC models in the list, pruning operations can be applied. For example, an LIC model candidate may be added to the list only if such an LIC model is not already in the list. Once the list of LIC model candidates is constructed, the LIC models in the list can be reordered based on the cost of applying these models to a template region of the currently coded block (e.g., using a SAD metric).

[0084] The size of the merge list can be signaled in the bitstream. For example, the list size can be signaled in a sequence parameter set (SPS), a view parameter set (VPS), a picture parameter set (PPS), an adaptation parameter set (APS), or a picture header (PH) of the encoded video data. Alternatively, the size of the merge list can be implicitly determined. That is, the size of the merge list can be predetermined or can be determined based on coding parameters or other conditions, such as the block size, color components, prediction modes, QP values, or sequence class and configuration. In an aspect, a maximum allowed size of the merge list can be predetermined (e.g., 12).

[0085] For each block coded using an LIC model merge mode, an index of the selected (and used) LIC model, in the list of LIC model candidates, can be encoded (signaled) by the encoder in the bitstream (e.g., using truncated unary binarization). Alternatively, the index of the selected LIC model can be implicitly determined by the decoder. When signaled, the first bin of the index can be coded with context and bypass coding can be used for the other bins; or each bin of the index can be coded with a separate context using context-based adaptive binary arithmetic coding (CABAC).

[0086] The list of LIC model candidates can include LIC models from spatially adjacent and / or non-adjacent neighboring blocks that are coded in an LIC mode. For the spatially adjacent neighboring blocks, multiple positions of the spatially adjacent blocks can be used (e.g., as shown in FIG. 6) and their respective LIC models (if available) can be added to the list in the following order: B1, A1, B0, A0, and B2. For the spatially non-adjacent neighboring blocks, various positional patterns can be used (e.g., as shown in FIG. 7 and FIG. 10). The distances between the non-adjacent neighboring blocks and the current coding block can be based on the width and the height of the current coding block. The order in which the LIC models of the spatial adjacent and non-adjacent neighboring candidates are added to the list may be according to a predetermined criterion (e.g., distance and / or statistical metrics). For example, relative to the current block, the LIC model of the closest neighboring block may be added first and the LIC model of the farthest neighboring block may be added last; or, the neighboring blocks can be divided into two sets so that LIC models of blocks in the first set are added to the list first, and then, if the list is not full, LIC models of blocks in the second set are added.

[0087] The list of LIC model candidates can include LIC models from temporal and / or shifted temporal neighboring blocks that are coded in an LIC mode. These temporal and shifted temporal neighboring blocks are described, respectively, in reference to FIG.8 and FIG.9.

[0088] History-based LIC models, maintained in an history-based table, can also be added to the list of LIC model candidates after LIC models from the spatial and temporal neighboring blocks (if available) are added. Redundancy checks can be applied to any history-based LIC model that is to be added to the list. The history-based table includes LIC models of blocks that were previously coded in an LIC mode. Multiple LIC models (e.g., 6) can be maintained in the history-based table during the encoding / decoding process. The history-based table can be reset (emptied) when a new CTU row is encountered. Whenever a block is coded in an LIC mode, the used LIC model may be added to the last entry of the table as a new history-based LIC model. When inserting a new LIC model to the table, a first-in-first-out (FIFO) rule may be utilized, where a redundancy check is first applied to find whether there is already an identical LIC model in the table. If found, the identical LIC model is removed from the table and all the history-based LIC models after it are moved forward in the table and the new LIC model is inserted into the last entry of the table.

[0089] If the list of LIC model candidates is not full – even after adding the LIC models from the spatial neighboring blocks, temporal neighboring blocks, and history-based table – default candidates can be added to the list until the list is full (i.e., the maximum LIC model candidate number has been reached). For example, the default scaling parameters of the default LIC models can be determined based on the scaling parameter of the first LIC model candidate plus 0, 1 / 8, -1 / 8, 2 / 8, -2 / 8, 3 / 8, -3 / 8, 4 / 8, -4 / 8, 5 / 8, -5 / 8, and / or 6 / 8. The offset parameter can be derived as follows: ^^ ൌഥT െ ^^ ൈ ^ഥ^^ , (17)where ^^ is the default scaling parameter, where Tഥis the average of samples in the current template, and where ^ഥ^^is the average of samples in the reference template.

[0090] The LIC model merge mode, disclosed herein, can also be applied when using a subblock mode. In the subblock mode, each subblock has its own motion vector and so an LIC model can be derived based on the current and the reference templates associated with the subblocks, as described in reference to FIG. 5. When using an LIC model merge mode in combination with a subblock mode, the list of LIC model candidates may include LIC models from various reference blocks pointed to by the motion vectors associated with each of the subblocks of the current block. For example, FIG.5 illustrates 16 subblocks in a current block 515. The LIC models (if they exist) of reference blocks associated with these subblocks can be added to the merge list. For example, LIC models used by respective reference blocks AREF, BREF, CREF,…, and GREF can be added in that order. Thus, a reference block pointed to by the motion vector of a respective subblock can be checked: if the reference block (e.g., AREF) is coded in an LIC mode and the corresponding LIC model has not yet been added to the list, then this corresponding LIC model can be added to the list. In an aspect, an added LIC model can be that of a reference block that is pointed to by a motion vector that is derived based on one or more motion vectors associated with respective subblocks of the video block. In another aspect, that LIC model may be derived from a reference block pointed to by the motion vector representing the center of the current block.

[0091] The LIC model merge mode, disclosed herein, can be applied when using bi-prediction to predict the current block (namely, a bi-predictive LIC model merge mode). Thus, LIC can be extended to bi-predictive blocks, where two LIC models are applied to generate two prediction blocks that are then combined to generate the final bi-prediction block, as described with respect to equations (5)-(9).

[0092] To indicate operation in a bi-predictive LIC model merge mode, one flag, bic_model_merge_flag (instead of the lic_model_merge_flag), can be signaled. If the bic_model_merge_flag is set to true, two LIC models can be selected from the list of LIC model candidates, and the associated index can be signaled in the bitstream. As before, the list may contain LIC models from spatial and temporal neighboring blocks, from the history-based table, or default LIC models. For example, the list can include pairs of LIC models, that is, pairs of an L0 LIC model and an L1 LIC model. These LIC model pairs can be derived from spatial and temporal neighboring blocks that are coded in a bi-predictive LIC mode, and can be added to the list if these LIC model pairs have not yet been added to the list.

[0093] In an aspect, the two LIC models can be selected from different pairs of LIC models in the list of LIC model candidates, using two respective indexes that can be signaled in the bitstream. For example, a first LIC model (e.g., an L0 LIC model) can be selected out of a first pair of LIC models in the list using a first index and a second LIC model (e.g., an L1 LIC model) can be selected out of a second pair of LIC models in the list using a second index.

[0094] In another aspect, the pair of LIC models can be selected separately from two lists of LIC model candidates, and thus two corresponding indexes can be signaled into the bitstream. For example, a list of L0 LIC models and a list of L1 LIC models can be generated and used for deriving a pair of L0 LIC model and L1 LIC model, respectively. In another example, for a temporal neighboring block, a pair of LIC models may be: 1) an LIC model from the reference block pointed to by the corresponding ^^ ^^^^in reference picture list L0; and 2) an LIC model from the reference block pointed to by the corresponding ^^ ^^^^in reference picture list L1.

[0095] In yet another aspect, one LIC model (e.g., an L0 LIC model) is selected from a list, while the other LIC model (e.g., an L1 LIC model) is derived from the current and reference templates (as in FIG. 11). In this case, two flags, bic_model_merge_flag_L0 and bic_model_merge_flag_L1, can be signaled for a bi-predictive LIC block, to respectively indicate whether a merge mode is used to derive the L0 LIC model and the L1 LIC model. For example, if bic_model_merge_flag_L0 is set to true, then the L0 LIC model can be selected from a list of L0 LIC model candidates based on an L0 LIC index that can be signaled in the bitstream. And, if bic_model_merge_flag_L1 is set to false, then the L1 LIC model can be derived from the current and reference templates (as in FIG.11).

[0096] Alternatively, one LIC model can be selected from the list of LIC model candidates, and this single LIC model can then be applied to the two prediction blocks (e.g., ^^^^ ^^, ^^^ and ^^^^^^, ^^^in equations (7)-(8)). For example, for LIC models from the temporal neighboring blocks, a single LIC model can be derived by blending two LIC models inherited from two reference blocks in reference picture lists L0 and L1. The blending process can be simple averaging or can be weighted averaging (e.g., with the weights indicated by the BCW index).

[0097] Note that aspects of the bi-predictive LIC model merge mode described above can be similarly applied to bi-predictive IBC blocks.

[0098] Refinement of the inherited LIC model parameters.

[0099] An inherited LIC model can be refined to improve the local illumination compensation performance. To that end, the parameters of the inherited LIC model can be adjusted – e.g., a refined scaling parameter and / or a refined offset parameter can be calculated from the templates associated with the respective video block. For example, the inherited scaling parameter can be used as is, while the inherited offset parameter can be refined based on the current template and the reference template (respectively, associated with the current block and its reference block), as follows: ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ൌ∑ே^ୀ^^ ^^ ^^ ^^^െ ^ ^^ ∗ ^^ ^^ ^^^^ ^^^^ (18) ^^ᇱൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ / ^^ (19) where ^^ ^^ ^^^and ^^ ^^ ^^^are pixel values of reconstructed samples from the current and reference templates, respectively, and where N is the total number of samples. The offset ^^ is then adjusted by ^^ᇱ, that is, ^^ ൌ ^^ ^ ^^ᇱ.

[0100] When bi-prediction is applied, the inherited L0 LIC model and L1 LIC model can be refined using the following steps: 1) applying the inherited L0 LIC model ^ ^^^, ^^^^ to samples of the L0 template, ^^^, obtaining: ^^^ᇱൌ ^^^^^^^ ^^^; 2) subtracting samples of ^^^ᇱfrom corresponding samples of template ^^ of the current block, obtaining ^^ᇱൌ ^ ^^ െ ^1 െ ^^^ ൈ ^^^ᇱ^⁄ ^^ ; 3) refining inherited L1 LIC offset ^ ^^^^ based on minimum difference between templates ^^ᇱand ^^^: ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ൌ ∑ே^ୀ^൫ ^^ᇱെ ^ ^^^∗ ^^^^ ^^^^൯, (20) 4) applying the inherited L1 LIC model to samples of the L1 template, ^^^, obtaining ^^^ᇱൌ ^^^^^^^ ^^^ᇱ; and 5) refining the inherited L0 LIC offset^^^^^by subtracting samples of ^^^ᇱfrom corresponding samples of ^^ and recomputing the offset ^^^ᇱ. In a variant, only the L0 LIC model is inherited, and the L1 LIC model and (optionally) the L0 LIC model are iteratively refined. For example, the L0 LIC model ( ^^^, ^^^) is inherited and the L1 LIC model ( ^^^, ^^^) is derived using the following steps: 1) applying the inherited L0 LIC model ^ ^^^, ^^^^ to the samples of the L0 template, ^^^, obtaining: ^^^ᇱൌ ^^^^^^^ ^^^; 2) subtracting samples of ^^^ᇱfrom corresponding samples of the template ^^ of the current block, obtaining updated template ^^ᇱൌ ^ ^^ െ ^1 െ ^^^ ൈ ^^^ᇱ^⁄ ^^ ; 3) deriving the L1 LIC model ^ ^^^, ^^^^ according to equations (5) and (6) using the updated T’ and L1 template ^^^.

[0101] In the case where the LIC model is inherited from a temporal candidate, the scaling parameter ^^ can be refined based on the picture order count (POC) distance and the offset ^^ can be refined, for example, according to equations (20)-(21) using the refined scaling parameter. This refinement is further described with respect to FIG.13.

[0102] FIG.13 is a diagram illustrating a refinement of an LIC model 1300. In the example of FIG. 13, a current block 1315 (of a current picture 1310) refers to reference block A 1325 (of reference picture 1320) that may contain a temporal neighboring block. The LIC model of that temporal neighboring block may be applied to reference block B 1335 to generate therefrom a prediction block. In an aspect, when the current block 1315 inherits the LIC model from that temporal neighboring block, the inherited LIC model’s scaling parameter can be refined based on the distances ^^ ^^ 1340 and ^^ ^^ 1350, where ^^ ^^ is the POC difference between reference picture 1320 and the current picture 1310 and ^^ ^^ is the POC difference between reference picture 1330 and reference picture 1320. The refinement of the scaling parameter can be normalized to constrain its range. For example, the scaling parameter ^^ can be refined into ^^ᇱas followed: ^^ᇱൌ^^^ െ 1^∗்^்^ ^ 1 (22)

[0103] FIG. 14 is a diagram example of FIG.14, the scaling parameter ^^ of the inherited LIC model is refined or adjusted into ^^ᇱusing an adjustment parameter, denoted by ^^, as follows: ^^ᇱൌ ^^ ^ ^^, and (23) ^^ᇱൌ ^^ െ ^^ ൈ ^^^, (24) where ^^^denotes an average value of the reference samples. As demonstrated in FIG.14, this refinement tilts or rotates the LIC model around ^^^. The adjustment parameter ^^ can be a value within a predefined range. In an aspect, the encoder can signal the adjustment parameter ^^ in the bitstream.

[0104] FIG.15 is a flowchart of an example method for encoding video data 1500. According to aspects described herein, method 1500 can be applied to the coding of a video block (of the video data) into a bitstream. The coding of the video block 1510 may begin, in step 1520, by constructing a list of LIC model candidates. In step 1530, an LIC model can be selected from the list of LIC model candidates. The selected LIC model, in step 1540, can then be applied to a reference block of the video block to generate therefrom a prediction block for the video block. As described herein, the list of LIC model candidates can be constructed out of LIC models, including: LIC models of spatially adjacent neighboring blocks; LIC models of spatially non-adjacent neighboring blocks; LIC models of temporal neighboring blocks; LIC models of shifted temporal neighboring blocks; history-based LIC models; default LIC models; or, a combination thereof. In an aspect, the constructed list of LIC model candidates can also include LIC models of respective reference blocks, each of which is pointed to by a motion vector associated with a subblock of the video block. Alternatively, the list of LIC model candidates can include an LIC model of a reference block that is pointed to by a motion vector derived from one or more motion vectors associated with respective subblocks of the video block.

[0105] FIG.16 is a flowchart of an example method for decoding video data 1600. According to aspects described herein, method 1600 can be applied to the decoding of a video block (of the video data) from a bitstream. The decoding of the video block 1610 may begin, in step 1620, by constructing a list of LIC model candidates. In step 1630, an LIC model can be selected from the list of LIC model candidates. The selected LIC model, in step 1640, can then be applied to a reference block of the video block to generate therefrom a prediction block for the video block. The selection of the LIC model from the list of LIC model candidates can be based on an index that is decoded from the bitstream. Alternatively, the selection of the LIC model from the list of LIC model candidates can be based on an implicitly derived index.

[0106] In an aspect, a flag is decoded from the bitstream, indicating whether a current video block is to be decoded in a bi-predictive LIC model merge mode. If the flag is set to true, the construction (in steps 1520 and 1620), the selection (in steps 1530 and 1630), and the application (in steps 1540 and 1640) performed by methods 1500 and 1600 can include: constructing a list of pairs of LIC model candidates; selecting a pair of LIC models from the list; and then applying the pair of LIC models to respective reference blocks of the video block to generate a bi-prediction block for the video block. The selected pair of LIC models from the list of LIC model candidates can be based on an index that may be decoded from the bitstream. In another aspect, a first LIC model from a pair of LIC models in the list is selected based on a first index decoded from the bitstream, and a second LIC model from a pair of LIC models in the list is selected based on a second index that may be decoded from the bitstream.

[0107] According to further aspects, the construction (in steps 1520 and 1620), the selection (in steps 1530 and 1630), and the application (in steps 1540 and 1640) performed by methods 1500 and 1600 can include: constructing a first and a second lists of LIC model candidates; selecting a first LIC model from the first list and a second LIC model from the second list, and then applying the first and the second LIC models to respective reference blocks of the video block to generate a bi-prediction block for the video block. Alternatively, the construction, the selection, and the application can include: 1) constructing a first list and a second list of LIC model candidates; 2) for each model (of a first LIC model and of a second LIC model) and its respective list (of the constructed first list and second list), decoding from the bitstream a flag indicating whether the model is to be selected from the respective list, and if the flag is set to true, selecting the model from the respective list, otherwise deriving the model form template regions of the video block and its reference block; and then 3) applying the first and second LIC models to respective reference blocks of the video block to generate a bi-prediction block for the video block.

[0108] We have described several aspects and embodiments in the present disclosure. These aspects and embodiments provide at least the following outputs and results, including all combinations, across different claim categories and types: ^ Encoding, into coded video data, syntax elements that can enable the decoder to decode the coded video data, according to any of the aspects described herein. ^ A bitstream that includes one or more of the described syntax elements, or variations thereof. A bitstream can be any set of data whether transmitted, stored, or otherwise made available. ^ Creating, transmitting, receiving, and / or decoding of the bitstream. ^ An electronic device (e.g., a TV, a set-top box, a cell phone, or a tablet) that tunes (e.g., using a tuner) a channel to receive the bitstream or that receives (e.g., using an antenna) the bitstream over the air. The electronic device decodes the syntax elements from the bitstream, and, optionally, displays (e.g., using a monitor, screen, or any other type of display) a resulting image. Various other generalized, as well as particularized, outputs, results, implementations, and claims are also supported and contemplated throughout this disclosure.

[0109] Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for the proper operation of the method, the order and / or use of specific steps and / or actions can be modified or combined. Additionally, terms such as “first”, “second”, etc. can be used in various embodiments to modify an element, component, step, operation, etc., for example, a “first decoding” and a “second decoding”. Use of such terms does not imply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and can occur, for example, before, during, or in an overlapping time period with the second decoding.

[0110] Various methods and other aspects described in this application can be used to modify modules, for example, the modules of the video encoder 200 and the video decoder 300 as shown in FIG.2 and FIG.3. Moreover, the present aspects are not limited to a specific standard (such as VVC or HEVC) and can be applied, for example, to other standards and recommendations, as well as extensions of any such standards and recommendations. Unless indicated otherwise, or technically precluded, the aspects described in this application can be used individually or in combination.

[0111] Various numeric values are used in the present application. The specific values are for example purposes and the aspects described are not limited to these specific values.

[0112] Various implementations involve decoding. “Decoding,” as used in this application, can encompass all or part of the processes performed, for example, on a received encoded sequence in order to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, for example, entropy decoding, inverse quantization, inverse transformation, and differential decoding. Whether the phrase “decoding process” is intended to refer specifically to a subset of operations or generally to the broader decoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.

[0113] Various implementations involve encoding. In an analogous way to the above discussion about “decoding”, “encoding” as used in this application can encompass all or part of the processes performed, for example, on an input video data in order to produce an encoded bitstream. Additionally, the terms “reconstructed” and “decoded” can be used interchangeably, the terms “encoded” or “coded” can be used interchangeably, and the terms “image,” “picture,” and “frame” can be used interchangeably. Usually, but not necessarily, the term “reconstructed” is used on the encoder side while the term “decoded” is used on the decoder side.

[0114] Note that the syntax elements as used herein are descriptive terms. As such, they do not preclude the use of other syntax element names.

[0115] This disclosure has described various pieces of information, such as for example syntax, that can be transmitted or stored, for example. This information can be packaged or arranged in a variety of manners, including, for example, manners that are common in video standards such as putting the information into an SPS, a PPS, a NAL unit, a header (for example, a NAL unit header, or a slice header), or an SEI message. Other manners are also available, including, for example, manners common for system level or application level standards such as signaling the information into one or more of the following: a. SDP (session description protocol), a format for describing multimedia communication sessions for the purposes of session announcement and session invitation, for example, as described in RFCs and used in conjunction with RTP (Real-time Transport Protocol) transmission. b. DASH MPD (Media Presentation Description) Descriptors, for example, as used in DASH and transmitted over HTTP. A descriptor is associated with a Representation or collection of Representations to provide additional characteristics to the content Representation. c. RTP header extensions, for example, as used during RTP streaming. d. ISO Base Media File Format, for example, as used in OMAF and using boxes which are object-oriented building blocks defined by a unique type identifier and length (also known as 'atoms' in some specifications). e. HLS (HTTP live Streaming) manifest transmitted over HTTP. A manifest can be associated, for example, with a version or collection of versions of content to provide the characteristics of the version or collection of versions.

[0116] The implementations and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed can also be implemented in other forms (for example, an apparatus or program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, an apparatus, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, for example, computers, cell phones, portable / personal digital assistants (PDAs), and other devices that facilitate communication of information between end-users.

[0117] Reference to “one / an aspect” or “one / an embodiment” or “one / an implementation,” as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the aspect / embodiment / implementation is included in at least one embodiment. Thus, the appearances of the phrase “in one / an aspect” or “in one / an embodiment” or “in one / an implementation,” as well any other variations, appearing in various places throughout this application, are not necessarily all referring to the same embodiment.

[0118] Additionally, this application can refer to “determining” various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0119] Further, this application may refer to “accessing” various pieces of information. Accessing the information can include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0120] Additionally, this application may refer to “receiving” various pieces of information. Receiving is, as with “accessing,” intended to be a broad term. Receiving the information can include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0121] It is to be appreciated that the use of any of the following “ / ”, “and / or”, and “at least one of”, for example, in the cases of “A / B”, “A and / or B” and least one of A and B,” is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This can be extended, as is clear to one of ordinary skill in this and related arts, for as many items as are listed.

[0122] Also, as used herein, the word “signal” refers to, among other things, indicating something to a corresponding decoder. For example, in certain embodiments the encoder signals a quantization parameter for de-quantization. In this way, in an embodiment the same parameter is used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as well as others, then signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual data, a bit savings is realized in various embodiments. It is to be appreciated that signaling can be accomplished in a variety of ways. For example, one or more syntax elements, flags, and so forth are used to signal information to a corresponding decoder in various embodiments. While the preceding relates to the verb form of the word “signal”, the word “signal” can also be used herein as a noun.

[0123] As will be evident to one of ordinary skill in the art, implementations can produce a variety of signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal can be formatted to carry the bitstream of a described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

Claims

CLAIMS 1. A method for decoding video data, comprising: decoding, from a bitstream, a video block of the video data, the decoding comprises: constructing a list of local illumination compensation (LIC) model candidates, selecting an LIC model from the list of LIC model candidates, and applying the selected LIC model to a reference block of the video block to generate a prediction block for the video block.

2. The method according to claim 1, wherein the constructing comprises constructing a list of pairs of LIC model candidates; the selecting comprises selecting a pair of LIC models from the list of pairs of LIC model candidates; and the applying comprises applying the pair of LIC models to respective reference blocks of the video block to generate a bi-prediction block for the video block.

3. The method according to claim 2, further comprises: decoding from the bitstream an index, wherein the selecting of the pair LIC models from the list of LIC model candidates is based on the decoded index.

4. The method according to claim 2, further comprises: decoding from the bitstream a first index and a second index, wherein the selecting of the pair of LIC models comprises selecting a first LIC model and a second LIC model: the first LIC model is selected from a pair of LIC models in the list of pairs of LIC model candidates based on the decoded first index, and the second LIC model is selected from another pair of LIC models in the list of pairs of LIC model candidates based on the decoded second index.

5. The method according to claim 1, wherein the constructing comprises constructing a first and a second lists of LIC model candidates; the selecting comprises selecting a first LIC model from the first list and a second LIC model from the second list; andthe applying comprises applying the first and the second LIC models to respective reference blocks of the video block to generate a bi-prediction block for the video block.

6. The method according to claims 1, wherein the constructing comprises constructing a first and a second lists of LIC model candidates; and; the selecting comprises, for each model of a first and a second LIC models and a respective list of the constructed first and second lists, decoding from the bitstream a flag, indicating whether the model is to be selected from the respective list, and if the flag indicates that the model is to be selected from the respective list, selecting the model from the respective list, otherwise deriving the model; and the applying comprises applying the first and second LIC models to respective reference blocks of the video block to generate a bi-prediction block for the video block.

7. The method according to any one of claims 2-6, further comprising: decoding from the bitstream a flag, indicating whether the video block is to be decoded in a bi-predictive LIC model merge mode.

8. A method for encoding video data, comprising: coding, into a bitstream, a video block of the video data, the coding of the video block comprises: constructing a list of LIC model candidates, selecting an LIC model from the list of LIC model candidates, and applying the LIC model to a reference block of the video block to generate a prediction block for the video block.

9. The method according to claim 8, further comprising: refining at least one parameter of the selected LIC model based on templates regions in the neighborhood of the video block and the reference block.

10. The method according to claim 8 or 9, further comprising: refining at least one parameter of the selected LIC model based on a first distancebetween a picture containing the video block and a picture containing the reference block and based on a second distance between a picture containing the reference block and a picture containing a reference block of the reference block.

11. The method according to any one of claims 8-10, further comprising: refining at least one parameter of the selected LIC model based on an adjustment parameter; and coding the adjustment parameter into the bitstream.

12. An apparatus for decoding video data, comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the apparatus to: decode, from a bitstream, a video block of the video data, the decoding comprises: constructing a list of LIC model candidates, selecting an LIC model from the list of LIC model candidates, and applying the selected LIC model to a reference block of the video block to generate a prediction block for the video block.

13. The apparatus according to claim 12, wherein the instructions further cause the apparatus to: decode from the bitstream a flag, indicating whether the video block is to be decoded in an LIC model merge mode.

14. The apparatus according to claim 12, wherein a flag, indicating whether the video block is to be decoded in an LIC model merge mode, is implicitly derived.

15. The apparatus according to any one of claims 12-14, wherein the instructions further cause the apparatus to: decode from the bitstream an index, wherein the selecting of the LIC model from the list of LIC model candidates is based on the decoded index.

16. The apparatus according to any one of claims 12-14, wherein the selecting of the LIC model from the list of LIC model candidates is based on an implicitly derived index.

17. An apparatus for encoding video data, comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the apparatus to: code, into a bitstream, a video block of the video data, the coding of the video block comprises: constructing a list of LIC model candidates, selecting an LIC model from the list of LIC model candidates, and applying the LIC model to a reference block of the video block to generate a prediction block for the video block.

18. The apparatus according to claim 17, wherein the list of LIC model candidates is constructed out of LIC models, including: LIC models of spatially adjacent neighboring blocks; LIC models of spatially non- adjacent neighboring blocks; LIC models of temporal neighboring blocks; LIC models of shifted temporal neighboring blocks; history-based LIC models; default LIC models; or a combination thereof.

19. The apparatus according to claim 17 or 18, wherein the list of LIC model candidates is constructed out of LIC models, including: LIC models of respective reference blocks each pointed to by a motion vector associated with a subblock of the video block.

20. The apparatus according to any one of claims 17-19, wherein the list of LIC model candidates is constructed out of LIC models, including: an LIC model of a reference block pointed to by a motion vector derived from one or more motion vectors associated with respective subblocks of the video block.

21. The apparatus according to any one of claims 17-20, wherein candidates inthe list of LIC model candidates are reordered based on respective costs of applying the candidates to a template region of the video block.

22. The apparatus according to any one of claims 17-21, wherein the instructions further cause the apparatus to: code into the bitstream a syntax element, indicating the size of the list of LIC model candidates, the coding of the syntax element is in a sequence parameter set (SPS), a view parameter set (VPS), a picture parameter set (PPS), an adaptation parameter set (APS), or a picture header (PH) of the encoded video data.

23. The apparatus according to any one of claims 17-21, wherein the size of the list of LIC model candidates is implicitly determined.

24. A non-transitory computer-readable medium comprising instructions executable by at least one processor to perform a method for decoding video data, the method comprising: decoding, from a bitstream, a video block of the video data, the decoding comprises: constructing a list of LIC model candidates, selecting an LIC model from the list of LIC model candidates, and applying the selected LIC model to a reference block of the video block to generate a prediction block for the video block.

25. A non-transitory computer-readable medium comprising instructions executable by at least one processor to perform a method for encoding video data, the method comprising: coding, into a bitstream, a video block of the video data, the coding of the video block comprises: constructing a list of LIC model candidates, selecting an LIC model from the list of LIC model candidates, and applying the LIC model to a reference block of the video block to generate a prediction block for the video block.

Images