Method and apparatus for video encoding and decoding

Abstract

A method and apparatus for local illumination compensation for video encoding and decoding are disclosed. A local illumination compensation system for video encoding and decoding uses a memory to store illumination compensation parameters and does not require access to reconstructed pixels of neighboring blocks. A set of illumination compensation parameters is stored in a dedicated buffer that has a limited size and is separate from coding unit level storage of information. The buffer contains a set of illumination compensation parameters, which may, for example, be calculated on-the-fly (or in some other manner determined) or predetermined (e.g., obtained from a video signal or from a device).

Description

[0001] This application is a divisional application of the Chinese invention patent application with the application date of September 11, 2019, application number 201980060982.4, and invention name “Local illumination compensation for video encoding and decoding using stored parameters”. Technical Field

[0002] At least one embodiment of the present invention relates generally to the field of video compression. At least one embodiment is particularly directed to local illumination compensation, and more particularly to local illumination compensation using stored parameter sets. Background Art

[0003] In order to achieve high compression efficiency, image and video coding schemes usually use prediction and transform to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlation, and then the difference between the original block and the predicted block, which is usually represented as a prediction error or prediction residual, is transformed, quantized, and entropy decoded. To reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy decoding, quantization, transform, and prediction. Summary of the invention

[0004] One or more embodiments of the present invention are directed to a local illumination compensation (LIC) system that uses stored illumination compensation (IC) parameters and does not require access to reconstructed pixels of neighboring blocks.

[0005] According to a first aspect of at least one embodiment, a video encoding method for encoding picture data of a block in a picture includes: encoding the picture data using signaling information, wherein the signaling information at least includes information indicating the use of a local illumination compensation function and an index of a local illumination compensation parameter to be used.

[0006] According to a second aspect of at least one embodiment, a video decoding method for encoding picture data of a block in a picture includes: obtaining an index of local illumination compensation parameters to be used; and decoding the picture data, and when the use of the local illumination compensation function is signaled in signaling information corresponding to the block, using a local illumination compensation function having the local illumination compensation parameters obtained according to the index.

[0007] According to a third aspect of at least one embodiment, a device includes an encoder for encoding picture data for at least one block in a picture or a video, wherein the encoder is configured to encode the picture data using signaling information, the signaling information comprising at least information indicating the use of a local illumination compensation function and an index of a local illumination compensation parameter to be used.

[0008] According to a fourth aspect of at least one embodiment, a device includes a decoder for decoding picture data for at least one block in a picture or video, wherein the decoder is configured to obtain an index of local illumination compensation parameters to be used from the buffer; and decode the picture data, and when the use of local illumination compensation is signaled in signaling information corresponding to the block, use a local illumination compensation function having the local illumination compensation parameters obtained from the buffer according to the index.

[0009] According to a fifth aspect of at least one embodiment, a computer program comprising program code instructions executable by a processor is proposed, which computer program implements the steps of the method according to at least the first or second aspect.

[0010] According to an eighth aspect of at least one embodiment, a computer program product is proposed, which is stored on a non-transitory computer-readable medium and includes program code instructions executable by a processor, and the computer program product implements the steps of the method according to at least the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A block diagram of an example of a video encoder 100 , such as a High Efficiency Video Coding (HEVC) encoder, is shown.

[0012] Figure 2 A block diagram of an example of a video decoder 200 , such as a HEVC decoder, is shown.

[0013] Figure 3 A block diagram of an example of a system in which various aspects and embodiments are implemented is shown.

[0014] Figure 4A An example of a coding tree unit and a coding tree in the compressed domain is shown.

[0015] Figure 4B An example of dividing a CTU into a coding unit, a prediction unit, and a transform unit is shown.

[0016] Figure 5 A sample for reconstruction of a picture block in inter mode is shown.

[0017] Figure 6 The motion vector between the current block and the reference block is shown.

[0018] Figure 7 An example of bidirectional prediction using two references is shown.

[0019] Figure 8 An example of an overall coding process that signals coding parameters for a CU is shown.

[0020] Fig. 9 An example of the overall entropy decoding process of a CU is shown.

[0021] Fig.10 An example of coding inter prediction information associated with a CU in AMVP mode is shown.

[0022] Fig.11 An example of a process for decoding motion information associated with an AMVP coding unit is shown.

[0023] Fig.12 A first exemplary embodiment is shown, in which the LIC parameters are calculated after the CU is reconstructed at the decoder, and the LIC parameters are calculated based on the right sub-part of the CU.

[0024] Fig.13 shows the evolution of the LIC parameter buffer, as long as the Fig.12 An embodiment of the present invention is when a CU within a given CTU is being processed by a decoder.

[0025] Fig.14 A second exemplary embodiment is shown, in which the LIC parameters are calculated after the CU is reconstructed at the decoder, and the LIC parameters are calculated based on a lower horizontal sub-part of the CU.

[0026] Fig.15 shows the evolution of the LIC parameter buffer, as long as the Fig.14 An embodiment of the present invention is when a CU within a given CTU is being processed by a decoder.

[0027] Fig.16 A typical example of switching from a CTU to the next CTU is shown.

[0028] Fig.17 A third exemplary embodiment is shown, in which the LIC parameters are calculated after the CU is reconstructed at the decoder, and the LIC parameters are calculated over the vertical part on the right side of the CU and then over the horizontal lower part of the CU.

[0029] Fig.18 The evolution of the LIC parameters contained in the two buffers is shown as long as a CU within a given CTU is being processed by the decoder when the third embodiment is used.

[0030] Fig.19 An example of LIC parameter memory storage as proposed in the first embodiment is shown.

[0031] Fig. 20 A schematic diagram showing a decoding procedure of an encoding unit according to a first embodiment of the present invention is shown.

[0032] Fig.21An example process for decoding inter prediction parameters of a CU coded in AMVP mode according to the proposed first embodiment is shown.

[0033] Fig. 22 A fifth exemplary embodiment is shown which aims to speed up the encoding and decoding process.

[0034] Fig.23 An exemplary variant embodiment is shown in which the pre-fetched map indicates whether the LIC parameters have to be calculated.

[0035] Fig.24 The conditional calculation of LIC parameters using prefetch in CTU is shown.

[0036] Fig.25 An inter-CU decoding process according to the sixth embodiment is shown.

[0037] Fig.26 A schematic diagram showing an inter-frame prediction parameter decoding process according to the sixth embodiment.

[0038] Fig. 27 An exemplary embodiment of rate-distortion optimized coding mode selection for a CU in an inter slice is shown.

[0039] Fig.28 A flow chart for LIC parameter determination in accordance with at least one embodiment of the present application is described.

[0040] Fig.29 A flow chart for use of LIC parameters in accordance with at least one embodiment of the present application is described. DETAILED DESCRIPTION

[0041] Various embodiments are directed to an LIC system that uses a memory to store illumination compensation (IC) parameters and does not require access to reconstructed pixels of neighboring blocks. In at least one embodiment, a set of LIC parameters is stored in a dedicated buffer that has a finite size and is decoupled from the coding unit level storage of information. The buffer contains the set of LIC parameters, which can be calculated on the fly (or determined in some other way) or predetermined (e.g., obtained from a video signal or from a device such as an encoder), for example, by a decoder (and encoder).

[0042] With respect to the coding syntax of a coding unit (CU), a flag indicates that an illumination compensation function is used for the coding unit to which LIC can be applied. In addition, according to at least one embodiment, a LIC parameter index is signaled to indicate which LIC parameters from the buffer or LIC parameter set under consideration are used for the current CU. In the case of LIC parameters calculated on the fly, the LIC parameter set of at least one embodiment includes LIC parameters associated with the right edge portion of one or more previously coded CUs. These parameters are calculated or determined in some other way based on the vertical right sub-portion of one or more coded / decoded CUs. The parameter set collects LIC parameters over a limited spatial height, which is generally equal to the height of a coding tree unit (CTU). In another embodiment, an additional buffer may also store LIC parameters associated with the horizontal lower portion of the coded / decoded CU within the current CTU. Advantageously, this embodiment combined with the previous embodiment can increase coding efficiency at the expense of a limited increase in memory load. Another form of on-the-fly LIC parameter calculation may be to calculate the illumination change in the current CU being coded / decoded and store the resulting LIC parameters in a first-in-first-out (FIFO) buffer. The buffer is used as the LIC parameter set for one or more subsequent CUs in the slice under consideration. In at least one embodiment using this form of on-the-fly LIC parameter computer for the LIC in the CU, an index indicating the LIC parameters used in the FIFO is signaled in the bitstream. In addition, the FIFO concept is typically accompanied by a strategy for updating / maintaining the LIC parameters contained in the FIFO, which is described in detail later in this document. Another embodiment uses a predetermined set of LIC parameters. The predetermined LIC parameters can be defined before the application is used and may not change. The predetermined LIC parameters may also be adjustable so that they can be changed and therefore can be obtained before their use and updated during their use. In at least one embodiment, for at least one CU in which the use of LIC is indicated as true, an index signals the predetermined LIC parameters to be used during temporal prediction.

[0043] Furthermore, although principles related to specific drafts of VVC (Universal Video Coding) or HEVC (High Efficiency Video Coding) specifications are described, aspects of the present invention are not limited to VVC or HEVC and may be applied, for example, to other standards and recommendations (whether pre-existing or developed in the future) and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, the aspects described in this application may be used alone or in combination.

[0044] Figure 1 A block diagram of an example of a video encoder 100 , such as a HEVC encoder, is shown. Figure 1Encoders in which improvements are made to the HEVC standard or encoders that employ techniques similar to HEVC may also be described, such as the JEM (Joint Exploration Model) encoder for VVC being developed by the JVET (Joint Video Exploration Team).

[0045] Before being encoded, the video sequence may be subjected to a pre-encoding process (101). This is performed, for example, by applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0046] In HEVC, to encode a video sequence with one or more pictures, the pictures are partitioned (102) into one or more slices, where each slice may include one or more slice segments. The segments are organized into coding units, prediction units, and transform units. The HEVC specification distinguishes between "blocks" and "units", where a "block" addresses a specific area in the sample array (e.g., luma Y), and a "unit" contains a concatenated block of all coded color components (Y, Cb, Cr, or monochrome), syntax elements, and prediction data associated with the block (e.g., motion vectors).

[0047] For decoding in HEVC, pictures are divided into coding tree blocks (CTBs) of square shape with configurable size, and continuous sets of coding tree blocks are grouped into slices. A coding tree unit (CTU) contains CTBs of coded color components. CTB is the root of a quadtree partition into coding blocks (CBs), and a coding block can be partitioned into one or more prediction blocks (PBs) and forms the root of a quadtree partition into transform blocks (TBs). Corresponding to coding blocks, prediction blocks, and transform blocks, a coding unit (CU) includes a tree structure set of prediction units (PUs) and transform units (TUs), PUs include prediction information for all coding block color components, and TUs include a residual coding syntax structure for each color component. The sizes of CBs, PBs, and TBs for luminance components apply to the corresponding CUs, PUs, and TUs. In the present application, the term "block" may be used to represent, for example, any of CTUs, CUs, PUs, TUs, CBs, PBs, and TBs. Additionally, "block" may also be used to refer to macroblocks and partitions as specified in H.264 / AVC or other video coding standards, and more generally to data arrays of various sizes.

[0048] In an example of encoder 100, a picture is encoded by an encoder element as described below. The picture to be encoded is processed in units of CUs. Each CU is encoded using intra or inter mode. When a CU is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of intra mode or inter mode to use to encode the CU, and indicates the intra / inter decision through a prediction mode flag. The prediction residual is calculated by subtracting (110) the predicted block from the original image block.

[0049] A CU in intra mode is predicted from reconstructed neighboring samples within the same slice. A set of 35 intra prediction modes is available in HEVC, including DC, planar, and 33 angular prediction modes. The intra prediction reference is reconstructed from rows and columns adjacent to the current block. The reference is extended by twice the block size in the horizontal and vertical directions using available samples from previously reconstructed blocks. When an angular prediction mode is used for intra prediction, the reference samples can be copied along the direction indicated by the angular prediction mode.

[0050] Different options may be used to code the applicable luma intra prediction mode for the current block. If the applicable mode is contained in a constructed list of six most probable modes (MPMs), the mode is signaled by an index in the MPM list. Otherwise, the mode is signaled by a fixed-length binarization of the mode index. The six most probable modes are derived from the intra prediction modes of the top and left neighboring blocks.

[0051] For an inter CU, motion information (eg, a motion vector and a reference picture index) may be signaled in various methods (eg, “merge mode” and “advanced motion vector prediction (AMVP)”).

[0052] In merge mode, the video encoder or decoder assembles a candidate list based on the coded blocks, and the video encoder signals the index of one of the candidates in the candidate list. At the decoder side, the motion vector (MV) and reference picture index are reconstructed based on the signaled candidates.

[0053] In AMVP, a video encoder or decoder assembles a candidate list based on motion vectors determined from coded blocks. The video encoder then signals an index into the candidate list to identify the motion vector predictor (MVP) and signals the motion vector difference (MVD). At the decoder side, the motion vector (MV) is reconstructed as MVP+MVD. Applicable reference picture indexes are also explicitly coded in the CU syntax for AMVP.

[0054] The prediction residual is then transformed (125) and quantized (130), including at least one embodiment for adapting the chrominance quantization parameters, as described below. The transform is typically based on a separable transform. For example, a DCT transform is first applied in the horizontal direction and then in the vertical direction. In recent codecs such as JEM, the transforms used in the two directions may be different (e.g., DCT in one direction and DST in the other), which results in a wide variety of 2D transforms, whereas in previous codecs the variety of 2D transforms for a given block size was typically limited.

[0055] The quantized transform coefficients are entropy decoded (145) along with motion vectors and other syntax elements to output a bitstream. The encoder may also skip the transform and apply quantization directly to the untransformed residual signal on a 4x4 TU basis. The encoder may also bypass the transform and quantization, i.e., directly decode the residual without applying a transform or quantization process. In direct PCM decoding, prediction is not applied, and the coding unit samples are decoded directly into the bitstream.

[0056] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. An in-loop filter (165) is applied to the reconstructed picture, for example, to perform deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0057] Figure 2 A block diagram of an example of a video decoder 200 (e.g., a HEVC decoder, etc.) is shown. In the example of the decoder 200, a bitstream is decoded by the decoder elements as described below. The video decoder 200 generally performs the same Figure 1 The encoding pass described in is reciprocal to the decoding pass, which performs video decoding as part of encoding the video data. Figure 2 Also shown are decoders in which improvements are made to the HEVC standard or decoders that employ techniques similar to HEVC, such as the JEM decoder.

[0058] In particular, the input to the decoder comprises a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, picture partitioning information and other coding information. The picture partitioning information indicates the size of the CTU, and the way in which the CTU is divided into CUs and, if applicable, PUs. The decoder can therefore divide (235) the picture into CTUs and each CTU into CUs based on the decoded picture partitioning information. The transform coefficients are dequantized (240), including at least one embodiment for adapting the chroma quantization parameters described below, and inverse transformed (250) to decode the prediction residual.

[0059] The decoded prediction residual is combined with the prediction block (255) to reconstruct the image block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). As described above, AMVP and merge mode techniques can be used to derive motion vectors for motion compensation, which can use interpolation filters to calculate interpolated values ​​of sub-integer samples of the reference block. An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0060] The decoded picture may further undergo post-decoding processing (285), such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or performing an inverse remapping of the remapping process performed in the pre-encoding process (101). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0061] Figure 3A block diagram of an example of a system in which various aspects and embodiments are implemented is shown. System 300 may be implemented as a device including various components described below, and is configured to perform one or more of the various aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, encoders, code converters, and servers. The elements of system 300 may be implemented individually or in combination in a single integrated circuit, multiple integrated circuits, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 300 are distributed across multiple integrated circuits and / or discrete components. In various embodiments, the elements of system 300 are coupled communicatively via an internal bus 310. In various embodiments, system 300 is coupled communicatively to other similar systems or other electronic devices via, for example, a communication bus or by dedicated input and / or output ports. In various embodiments, system 300 is configured to implement one or more of the aspects described in this document, such as the video encoder 100 and video decoder 200 described above and modified as described below.

[0062] System 300 includes at least one processor 301, which is configured to execute instructions loaded therein, for implementing various aspects described herein, for example. Processor 301 may include embedded memory, input-output interface, and various other circuits known in the art. System 300 includes at least one memory 302 (e.g., volatile memory device and / or non-volatile memory device). System 300 includes storage device 304, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drive, and / or optical disk drive. As a non-limiting example, storage device 304 may include internal storage device, attached storage device, and / or network accessible storage device.

[0063] System 300 includes an encoder / decoder module 303, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 303 may include its own processor and memory. The encoder / decoder module 303 represents a module (one or more) that may be included in a device to perform encoding and / or decoding functions. As known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 303 may be implemented as a separate element of the system 300, or may be incorporated into the processor 301 as a combination of hardware and software known to those skilled in the art.

[0064] Program code to be loaded onto the processor 301 or encoder / decoder 303 to perform various aspects described in this document may be stored in the storage device 304 and subsequently loaded onto the memory 302 for execution by the processor 301. According to various embodiments, one or more of the processor 301, memory 302, storage device 304, and encoder / decoder module 303 may store one or more of various items during the execution of the processes described herein. These stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0065] In several embodiments, memory inside the processor 301 and / or the encoder / decoder module 303 is used to store instructions and to provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 301 or the encoder / decoder module 303) is used for one or more of these functions. The external memory may be a memory 302 and / or a storage device 304, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations such as MPEG-2, HEVC, or VVC.

[0066] Input to the elements of system 300 may be provided through various input devices as shown in block 309. Such input devices include, but are not limited to, (i) an RF section that receives an RF signal transmitted over the air, for example, by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0067] In various embodiments, the input device of block 309 has associated corresponding input processing elements known in the art. For example, the RF portion may be associated with elements for implementing the following necessary functions: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select a signal frequency band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF portion of various embodiments includes one or more elements to perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF portion may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near baseband frequency) or baseband. In a set-top box embodiment, the RF part and its relevant input processing element receive the RF signal transmitted by wired (for example, cable) medium, and filter to the frequency band of expectation again by filtering, down-conversion and perform frequency selection.Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements of similar or different functions.Adding element can include inserting element between existing element, for example inserting amplifier and analog-to-digital converter.In various embodiments, the RF part comprises antenna.

[0068] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 300 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented as needed, for example, in a separate input processing integrated circuit or in the processor 301. Similarly, various aspects of USB or HDMI interface processing may be implemented as needed, in a separate interface integrated circuit or in the processor 301. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements (including, for example, the processor 301 and the encoder / decoder 303), which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on an output device.

[0069] The various components of system 300 may be disposed within an integrated housing. Within the integrated housing, the various components may be interconnected and transmit data between using a suitable connection arrangement (eg, an internal bus including an I2C bus, wiring, and a printed circuit board as known in the art).

[0070] The system 300 includes a communication interface 305 that enables communication with other devices via a communication channel 320. The communication interface 305 may include, but is not limited to, a transceiver configured to send and receive data through the communication channel 320. The communication interface 305 may include, but is not limited to, a modem or a network card, and the communication channel 320 may be implemented, for example, within a wired and / or wireless medium.

[0071] In various embodiments, a Wi-Fi network such as IEEE 802.11 is used to stream data to the system 300. The Wi-Fi signals of these embodiments are received through a communication channel 320 and a communication interface 305 suitable for Wi-Fi communication. The communication channel 320 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to the system 300, which transmits data through the HDMI connection of the input block 309. Still other embodiments use the RF connection of the input block 309 to provide streaming data to the system 300.

[0072] The system 300 can provide output signals to various output devices, including a display 330, a speaker 340, and other peripherals 350. In various examples of embodiments, the other peripherals 350 include one or more of a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide functions based on the output of the system 300. In various embodiments, control signals are transmitted between the system 300 and the display 330, the speaker 340, or other peripherals 350 using signaling such as AV. Link, CEC, or other communication protocols, which enables device-to-device control with or without user intervention. The output device can be communicatively coupled to the system 300 via a dedicated connection through the corresponding interfaces 306, 307, and 308. Alternatively, the output device can be connected to the system 300 via the communication interface 305 using the communication channel 320. The display 330 and the speaker 340 can be integrated into a single unit in an electronic device (e.g., a television) along with other components of the system 300. In various embodiments, the display interface 306 includes a display driver (e.g., a timing controller (T Con) chip).

[0073] For example, if the RF part of input 309 is part of a separate set-top box, the display 330 and the speaker 340 can be alternatively separated from one or more of the other components. In various embodiments where the display 330 and the speaker 340 are external components, the output signal can be provided via a dedicated output connection, which includes, for example, an HDMI port, a USB port, or a COMP output. The implementation described herein can be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of the discussed feature can also be implemented in other forms (for example, a device or a program). For example, the device can be implemented with appropriate hardware, software, and firmware. The embodiment can be performed by a computer software implemented by a combination of processor 301 or hardware, or hardware and software. As a non-limiting example, the embodiment can be implemented by one or more integrated circuits. The memory 302 can be any type suitable for a technical environment, and can be implemented using any appropriate data storage technology, as a non-limiting example, such as an optical memory device, a magnetic memory device, a semiconductor-based memory device, a fixed memory, and a removable memory. Processor 301 may be of any type suitable for the technical environment, and may include one or more of a microprocessor, a general purpose computer, a special purpose computer, and a processor based on a multi-core architecture as non-limiting examples. Processors also include communication devices such as computers, cellular phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate information communication between end users.

[0074] Figure 4A and 4B shows examples of various structures used to represent images. In the HEVC video compression standard, motion compensated temporal prediction is used to exploit the redundancy that exists between consecutive pictures of a video. To do this, a motion vector is associated with each prediction unit (PU), which we now introduce. Each coding tree unit (CTU) is represented by a coding tree in the compressed domain. This is Figure 4A The quadtree partitioning of the CTU shown in Figure 1 is shown in Figure 2, where each leaf is called a coding unit (CU). Each CU is then given some intra-frame or inter-frame prediction parameters (prediction information). To this end, each CU is spatially split into one or more prediction units (PUs), each of which is assigned some prediction information. Figure 4B As shown, intra or inter coding modes are assigned on CU level. Specifically, one motion vector is assigned to each PU in HEVC. This motion vector is used for motion compensated temporal prediction of the considered PU. Therefore, the motion model linking the prediction block with its reference block simply consists in translation.

[0075] Figure 5Samples for reconstruction of a picture block in inter-frame mode are shown. In inter-frame mode, local illumination compensation (LIC) allows to correct block prediction samples obtained via motion compensation (MC) by taking into account spatial or temporal local illumination variations. Local illumination compensation (LIC) is based on a first-order linear model of illumination variations using a scaling factor a and an offset b. By subtracting the pixels located in the neighborhood V 当前 The set of reconstructed samples around the current block (“current blk”) in (Vcur) and the neighborhood V of the reference block (“ref blk”) located in the reference picture 参考 The reconstructed sample sets in (Vref)(MV) are compared to estimate the LIC parameters (a and b).

[0076] Figure 6 shows the motion vector between the current block and the reference block. 当前 and V 参考 (MV) exist in samples located in an L shape around the current block and the reference block (at the top, left, and upper-left side), respectively.

[0077] The LIC parameters are chosen to minimize V 当前 The sampling and V 参考 The mean square error difference between the correction samples in (MV). In general, the LIC model is linear: LIC(x) = a.x + b.

[0078]

[0079] Where s and r correspond to V 当前 and V 参考 (MV) is the pixel position in the image.

[0080] Hereinafter, a pixel position is represented by a set (eg, s or r) or a corresponding coordinate (eg, (x, y) or (i, j)).

[0081] Figure 7 An example of bidirectional prediction using two references is shown. In the case of bidirectional prediction, the two references V 参考 0(MV0) and V 参考 1(MV1) derives the LIC parameters (a0, b0) and (a1, b1). Therefore, LIC is based on the calculation of local distortion, which is defined as follows:

[0082]

[0083] Once the encoder or decoder obtains the LIC parameters for the current CU, the prediction pred(current_block) of the current CU is as follows (unidirectional prediction case):

[0084] pred(current_block)=a×ref_block+b

[0085] Wherein, current_block is the current block to be predicted, pred(current_block) is the prediction of the current block, and ref_block is the reference block used for temporal prediction of the current block.

[0086] Figure 8 An example of an overall decoding process that signals decoding parameters of a CU is shown.At least one embodiment focuses on conventional inter mode, also known as AMVP (Adaptive Motion Vector Prediction).

[0087] Fig. 9 An example of the overall entropy decoding process of a CU is shown. The case of AMVP inter mode is emphasized. This process occurs in the case of an inter CU that does not adopt merge mode.

[0088] Fig.10 An example of decoding inter prediction information associated with a CU in AMVP mode is shown. In this example, a single IC flag is decoded for each AMVP CU. Several motion fields are encoded for an AMVP CU. One or two reference pictures are used to predict the current inter CU. For each reference picture used, the motion field is encoded in the form of a reference frame index, an index identifying a candidate for predicting the motion vector of the current CU, and a motion vector difference (MVd) between the motion vector of the current CU and the motion vector predictor (MVP) used.

[0089] Finally, additional temporal prediction parameters may be encoded, including the following:

[0090] The -iMv flag indicates whether the motion vectors of the current CU are coded with a reduced level of precision compared to the usual quarter-pel precision.

[0091] The -OBMC flag indicates whether the temporal prediction of the current CU includes a known overlapped block motion compensation step size.

[0092] - A single IC flag associated with each inter CU in AMVP mode.

[0093] Fig.11 An example of a process for decoding motion information associated with an AMVP coding unit is shown. Fig.10 After the inverse of the decoding process of , the motion field of each reference image used to predict the current CU is decoded. Note that in AMVP mode, a single flag is decoded once for each inter CU.

[0094] The principle of LIC as described above requires access to reconstructed samples of the top and left neighboring blocks that have already been reconstructed. This aspect may be undesirable as it introduces some latency in the motion compensation process and potentially disrupts conventional decoder pipeline architectures.

[0095] This disclosure describes a number of embodiments designed with the foregoing in mind.

[0096] Fig.12 A first exemplary embodiment is shown, in which the LIC parameters are calculated after the CU is reconstructed at the decoder, and the LIC parameters are calculated based on the right sub-portion of the CU. In at least one exemplary embodiment, the LIC parameters are calculated based on the right sub-portion of the decoded CU under consideration and stored in a buffer. In at least one embodiment, this buffer of LIC parameters comprises an array of LIC parameters (a, b) indexed as a function of the vertical position of the sub-blocks to which they refer. In other words, it is proposed to store a tuple (a, b) of LIC parameters for each y position of a sub-block within a CTU. This results in storing less total number of tuples (a, b) than in the case of storing (a, b) parameters for each CU of a picture.

[0097] Fig.13 shows the evolution of the LIC parameter buffer, as long as the Fig.12 In the embodiment of the present invention, a CU within a given CTU is being processed by the decoder. In fact, at least one advantage of calculating the LIC parameters based on the right part of each CU is that it ensures that at any time, when decoding a coding unit, some LIC parameters corresponding to the picture area adjacent to the current CU are available. The LIC parameters are then used to reconstruct the current CU. Fig.13 In the figure, the grey areas represent all regions for which LIC parameters have been calculated and currently stored in memory at a given stage of the decoding process. It can be seen that for each CU being decoded, some LIC parameters representing the brightness variation in the left neighboring regions of the current CU are available and can potentially be used to apply LIC in the current CU. In other words, the LIC parameters associated with the already decoded CU are discarded and replaced by the LIC parameters for other CUs. Note that in Fig.13 In general, before starting decoding of a CU contained in a current CTU, the LIC parameters contained in the buffer correspond to the LIC parameters previously calculated in the right part of the CU located to the right of the previous CTU. This is determined by Fig.13 The first stage of the process is shown in the vertical grey rectangle on the left side of the CTU.

[0098] According to at least one variant embodiment, LIC may be used only for blocks whose area is strictly above a certain value (e.g. 32). This corresponds to blocks at least as large as 8×8. In this case, if the CTU size is 128×128, the size of the buffer for LIC parameters may be reduced by 2 by storing a pair of parameters (a, b) for each y position of an 8×8 sub-block of the CTU, which results in a total of 16 pairs of parameters (a, b). Note that according to the variants given above, the buffer size is 32 or 16 for the entire picture.

[0099] Fig.14 A second exemplary embodiment is shown, in which the LIC parameters are calculated after the CU is reconstructed at the decoder, and the LIC parameters are calculated based on a horizontal sub-portion of the lower edge of the CU. In at least one exemplary embodiment, the LIC parameters are calculated on the horizontal sub-portion of the lower edge of the decoded CU under consideration and stored in a buffer. At the CU level, this process is close to Fig.12 The process is coupled with a memory buffer of (W_CTU / 4) pairs of LIC parameters (a_i, b_i), where W_CTU represents the width of CTU (usually 128).

[0100] Fig.15 shows the evolution of the LIC parameter buffer, as long as the Fig.14 When a CU within a given CTU is being processed by the decoder. Fig.12 A weakness of this embodiment compared to the embodiment of FIG. 1 is that at a certain point, no LIC parameters corresponding to the spatial region adjacent to the current CU are available. This occurs when switching from a CTU to the next CTU during decoding. However, in other situations that occur more frequently, this embodiment proves to be beneficial.

[0101] Fig.16 A typical example of switching from a CTU to the next CTU is shown. In this figure, the light grey CU in the second CTU uses the LIC parameters calculated in the lower part of the previous CTU.

[0102] In at least one variant embodiment, the LIC may be used only for blocks whose area is strictly above a certain value, such as 32. This corresponds to blocks at least as large as 8x8. In this case, if the CTU size is 128x128, the size of the buffer for the LIC parameters may be reduced by 2 by storing a pair of parameters (a, b) for each y position of an 8x8 sub-block of the CTU, which results in a total of 16 pairs of parameters (a, b).

[0103] Fig.17A third exemplary embodiment is shown, in which the LIC parameters are calculated after the CU is reconstructed at the decoder, and the LIC parameters are calculated on the vertical part to the right of the CU and then on the horizontal lower part of the CU. This embodiment combines the two first embodiments. Thus, during the decoding process, two buffers corresponding to the vertical part and the horizontal part are maintained. The sizes of these buffers are the same as explained in the previous two parts.

[0104] Fig.18 The evolution of the LIC parameters contained in the two buffers is shown as long as the CU within a given CTU is being processed by the decoder when the third embodiment is used. The grey parts represent all areas where LIC parameters have been calculated and currently stored in memory at a given stage of the decoding process. Note that the horizontal part used to calculate the LIC parameters of the previous CTU does not appear here to keep the figure as clear as possible.

[0105] Fig.19 An example of LIC parameter memory storage as proposed in the first embodiment is shown. It shows a memory buffer for storing LIC parameters during the decoding process, represented as a grey rectangle.

[0106] Fig. 20 FIG. 1 is a schematic diagram of a CU decoding process according to a first embodiment of the present invention. Except for the steps highlighted in bold, the decoding process is conventional. First, according to the following Fig.21 Next, two other major modifications are the LIC parameter calculation and LIC parameter storage steps, which occur in Fig. 20 The end of the CU decoding process.

[0107] The calculation of the LIC parameters (a, b) is performed conventionally as described above, except that it is performed on the vertical right portion of the current CU. This step then results in determining the LIC parameters (a, b) of the first-order linear model for local illumination compensation. Then, as Fig. 20 As shown in the last step of , these parameters are stored in a buffer. As mentioned above, the buffer of LIC parameters simply consists of an array of tuples (a, b), indexed by the vertical spatial position referred to by the (a, b) parameter under consideration relative to the top of the CTU under consideration. Therefore, the buffer can be represented as the following array:

[0108] {(a i ,b i ),i=0,…,N-1},N=(H CTU >>2)

[0109] Among them, H CTUis the height of the CTU under consideration, and for each index i included between 0 and (N-1), the parameter (a i ,b i ) denotes the LIC parameters associated with the vertical spatial position i. Since motion information is usually stored at the 4x4 block level in the current picture, a pair of LIC parameters is associated with each index i (which is associated with each vertical position) considering a 4-pixel one-dimensional (1D) grid at the CTU height. This is Fig.19 , which shows a memory buffer for storing LIC parameters during the decoding process.

[0110] Therefore, to store the LIC parameters (a, b) associated with a given CU, for each 4-subblock based vertical position i contained in the current CU, the parameters (a_i, b_i) in the memory buffer are set equal to the values ​​(a, b).

[0111] Fig.21 FIG. 1 shows an example process for decoding inter prediction parameters of a CU coded in AMVP mode according to the first proposed embodiment. In addition to the newly introduced step of LIC parameter selection for the current CU, for a CU with the LIC flag equal to true, the process is the same as Fig.11 The same as in.

[0112] This step allows determining which LIC parameters are used for the current CU among the LIC parameters stored in the dedicated memory buffer, and proceeds as described below. The LIC parameters corresponding to the closest neighbor block having the same reference picture index as the current CU are selected. The closest neighbor is, for example, the LIC parameter having the closest vertical position to the vertical position of the current CU, since the memory buffer is organized by vertical spatial position. Therefore, this step comprises looping over the LIC parameters currently stored in the memory buffer, from the current closest vertical position to the vertical position that is less close to the current CU vertical position. Here, as an exemplary way of implementing the loop, the position of the current CU is defined as the position (x, y) of the upper left corner of the current CU. Therefore, the vertical position of the current CU is the y coordinate of the upper left corner of the current CU. According to a variant, the starting position in the loop corresponds to the LIC parameter associated with the lowest vertical position. Therefore, it corresponds to the LIC parameter issued from the encoding / decoding of the coding unit containing the 4x4 sub-block located to the left of the lower left corner of the current CU. Then, the loop iterates from this starting point towards the LIC parameter associated with the highest vertical position. According to another variation, the loop iteration is also performed on the LIC parameters associated with vertical positions beyond the horizontal boundary of the current CU to make it more likely that the process will find some LIC parameters with the same reference picture index as the current CU.

[0113] To obtain the reference picture index associated with a candidate pair of LIC parameters in the buffer, one retrieves the reference picture index of the last decoded CU at the considered vertical position (i.e. the reconstructed CU that led to the calculation and storage of the considered candidate LIC parameters). According to a variant, and in order to make the decoding process simpler, the reference picture index is stored together with the LIC parameters in the considered memory buffer.

[0114] If no neighboring block has the same reference picture index as the current block, then according to an embodiment, the LIC parameters referring to the position closest to the position of the current block may be used for the current CU. However, note that, before considering candidate LIC parameters that refer to blocks with reference picture indexes different from the current block, priority is given to LIC parameters emitted from decoded blocks with the same reference picture index as the current block.

[0115] According to a variant, the LIC flag may be inferred to be false if there are no stored LIC parameters issued from a block having the same reference picture as the current block.

[0116] Finally, note that in all proposed embodiments, the LIC parameters corresponding to the trivial linear model (i.e. a=1, b=0) are not considered and are not stored in the dedicated memory buffer. In this case, the LIC is not used at all and is signaled accordingly.

[0117] The detailed CU decoding process according to the second embodiment is similar to the above description of the first embodiment. The difference is that, instead of the vertical area for calculating the LIC parameters, the horizontal lower sub-part of each reconstructed CU is used to calculate the LIC parameters, and then the LIC parameters are stored in a dedicated memory buffer ( Fig.14 , 15and 16). With regard to the selection / determination of the LIC parameters for the current CU, a process similar to the step "Selecting LIC parameters in the memory buffer for the current CU" is called. It therefore consists in a loop on the LIC parameters stored in the memory buffer under consideration. Likewise, the loop may use different starting points (and therefore also end points) in the LIC parameters stored in the memory buffer. According to a first variant, the loop iterates starting from the LIC parameters corresponding to the horizontal CU position on the left (i.e., the LIC parameters associated with the horizontal position equal to the left position of the current CU). In this variant, the loop iterates from this position to the stored LIC parameters corresponding to the horizontal position equal to the horizontal position of the right corner of the current CU. According to another variant, the order is reversed. This means that the loop iterates from the right position (equal to the position of the right corner of the current CU) to the left position (equal to the horizontal position of the left corner of the current CU). According to another variant, the loop iterates also on the LIC parameters associated with the horizontal position beyond the boundary of the current CU to make it more likely that the process will find some LIC parameters with the same reference picture index as the current CU.

[0118] The third embodiment is a combination of the first and second embodiments, and the LIC parameter calculation and storage are the same as those of the first and second methods. The significant difference from the two first embodiments is that assuming two memory buffers (storing LIC parameters based on vertical and horizontal areas respectively), the LIC parameters are selected to be applied to the current CU buffer. In fact, a pair of LIC parameters is selected in these two buffers for the current CU. A simple way to select it is to select a pair of LIC parameters in each buffer, as done in the first and second embodiments. Next, among the two selected LIC parameter pairs (a, b), the LIC parameter pair corresponding to the position closest to the current CU is selected. If their respective distances to the current CU position are the same, priority is given to one direction, such as the vertical direction.

[0119] In at least a fourth embodiment, a new criterion for pruning LIC parameter candidates in two memory buffers (vertical and horizontal) is used. An iterative loop is performed on all LIC parameters (vertical, horizontal, or both) stored in the memory buffer(s) to find all LIC parameters with the same reference picture index as the current CU, and finally select the LIC parameters calculated with neighbors (N) whose reconstructed motion vector values ​​(MVx, MVy) are closest to the motion vector of the current CU. "Closest" can be, for example, a function based on the L1 norm (sum of absolute differences) or the L2 norm.

[0120] In some conventional codecs, motion vectors and reference indices are stored in a motion information buffer that can be used to decode other CUs. Such a structure can be used to obtain reference picture indices and motion vector values ​​associated with LIC parameters stored in memory.

[0121] In a variant embodiment, if the current CU is coded in merge mode, which indicates that some of the current CU coding parameters have been derived from (or directly merged with) a top (or left) neighboring CU (N), then the LIC parameters calculated with this top (or left) CU neighbor stored in the memory buffer (one or more) are selected for the current CU.

[0122] In a variation of the fourth embodiment, a LIC flag indicating whether LIC applies to the current CU may be inferred or coded (from this neighboring CU (N)).

[0123] Fig. 22 A fifth exemplary embodiment is shown which is intended to speed up the encoding and decoding process. In this embodiment, the LIC parameters for the partitions located at the bottom of the current CTU or to the right of the picture are not calculated because they will not be used for decoding other CUs. These partitions are Fig. 22 The black rectangle in .

[0124] In a variant embodiment, during the decoding phase, during parsing of coding parameters and construction of the CTU partition tree, a prefetch map may be constructed indicating whether LIC parameters must be calculated after reconstruction of each CU. For example, for a CU coded intra or inter with an explicitly coded LIC flag (i.e., not inferred from reconstructed neighbors), the map would contain the following LIC flag:

[0125] Intra-coded CU: Map LIC flag = false

[0126] Inter-coded CU with explicitly decoded LIC flag: Map LIC Flag = LIC Flag Inter-coded CU with inferred LIC flag, inferred LIC flag: Map LIC Flag = True

[0127] Next, after reconstructing a current CU with inter-frame decoding, if mapping indicates that all right CUs (Cur) and lower CUs (Cub) in the current CTU do not use LIC (Map LIC Flag = False), LIC parameters are not calculated for the current CU.

[0128] Fig.23 An exemplary variant embodiment is shown in which the pre-fetched map indicates whether LIC parameters must be calculated. In this figure, partitions with a white background use LIC, while partitions with a grey background do not use LIC, and for the remaining partitions with a cross-dashed background, LIC usage is inferred from reconstructed neighbors.

[0129] Fig.24 Conditional calculation of LIC parameters using pre-fetching in a CTU is shown. In the first loop over the CU tree in the CTU, the CU parameters are decoded and a map of LIC flags is constructed as described above. In the second loop over the CU tree in the CTU, the CU is reconstructed and the LIC flag is checked to decide if the LIC parameters should not be calculated.

[0130] Fig.25 A sixth exemplary embodiment is shown. This embodiment includes the following features:

[0131] - For the 3 first embodiments, a memory buffer of limited size is used to store LIC parameters which can be used to apply illumination changes later during motion compensated temporal prediction of some CUs in the picture being decoded.

[0132] - For the 3 first embodiments, the LIC parameters for each reconstructed CU are calculated. They can be calculated on the entire reconstructed CU, rather than on the right or bottom part of the CU as in the first 3 embodiments.

[0133] - Changing the strategy of storing LIC parameters into a dedicated memory buffer compared to previous embodiments: storing LIC parameters only when they bring some change with respect to already stored parameters, in order to provide subsequent CUs with a good set of candidate LIC parameters for coding efficiency issues.

[0134] - Storing some LIC parameters is combined with discarding some LIC parameters to keep the memory buffer at a desired size.

[0135] - The LIC parameters used in the prediction of the current CU are identified by an index indicating which LIC parameters stored in the buffer are used in the current coding unit. If the LIC flag of the considered CU is equal to true, this index is entropy decoded from the bitstream.

[0136] Fig.25 The inter-CU decoding process according to the sixth embodiment is shown. In this embodiment, the LIC parameters of the CU (after it has been reconstructed) are calculated similarly to the above, except that the area on which the LIC parameters (a, b) are calculated is located within the reconstructed CU. Once calculated, the LIC parameters are compared with those already stored in the buffer. If it is detected that the distance between the newly calculated parameters and the already stored parameters is sufficient (for example, greater than a threshold), the newly calculated parameters are stored in a FIFO (first in first out) buffer for each stored parameter pair. The buffer has a limited size, so if it is full, the first element already stored in the buffer is removed.

[0137] Fig.26 A schematic diagram showing the inter-frame prediction parameter decoding process according to the sixth embodiment is shown. Fig.25 The calculation / storage algorithm of LIC parameters is as follows. If the LIC flag is valid for the current inter CU (e.g., the flag is true), the process is the same as before, except for the decoding step that includes an additional syntax element. This syntax element corresponds to the index of the LIC parameters to be used for the current CU during motion compensated temporal prediction of the current CU. The decoded index identifies the LIC parameters from the FIFO buffer to be assigned to the current CU.

[0138] The seventh embodiment uses a predetermined set of LIC parameters. This includes the following features.

[0139] - The use of LIC is again signaled by the CU level LIC flag, as in the previous embodiments.

[0140] -LIC parameters are no longer calculated based on picture content. Instead, they are predetermined.

[0141] - As in the previous embodiment, the predetermined LIC parameters are stored in a dedicated, limited memory buffer.

[0142] - For each CU for which LIC is valid, an index is decoded from the bitstream. This index identifies the LIC parameters from the buffer that are used during inter prediction for the current coding unit.

[0143] Note that this embodiment is similar to the generalized bidirectional prediction concept, but is applied to the case of unidirectional temporal prediction. Basically, the unidirectional prediction takes the form of a linear function as follows: Pred(x) = a.ref + b. The particularity of this embodiment is that here (a, b) is a pair of values ​​selected from a set of predefined pairs and identified by the transmitted pair index. Therefore, this embodiment is similar to the embodiment in the previous section. The difference is that Fig.25 LIC parameter calculation steps. Memory buffer ( Fig.26 ) remains unchanged. Furthermore, the memory buffer for the LIC parameters is not a FIFO as in the previous embodiment, but a fixed set of predetermined fixed LIC parameters.

[0144] Fig. 27An exemplary embodiment of rate-distortion optimized decoding mode selection for CUs in inter-frame slices is shown. In fact, in addition to the decoding of flags related to illumination compensation, the rate-distortion optimization of the AMVP decoding mode is also modified. As shown in the figure, all possible inter-frame modes are evaluated, and then if the performance of the INTER mode best found in the decoding of the CU considered is insufficient, the intra-frame mode is evaluated. The tested inter-frame modes include, for example, affine merging, merge mode, FRUC merge mode, and inter-frame AMVP mode. In another embodiment, all available modes are tested. In AMVP mode, the rate-distortion search includes a loop over all decoding parameters, including iterations of the IC flag. Therefore, each possible value of the IC flag is evaluated from the rate-distortion point of view for the current CU.

[0145] At least one embodiment of the present application is implemented on the encoder side, for example, in the device 100 or 300 in the following manner:

[0146] - As long as a coding unit of a CTU is being encoded, maintain exactly the same memory buffer as at the decoder to store candidate LIC parameters for subsequent coding units for encoding.

[0147] - In merge mode, the same LIC flag and LIC parameter derivation process as at the decoder side is performed. This involves:

[0148] o In the case of the five first embodiments, the LIC parameter pairs in a dedicated memory buffer are propagated from the RD-evaluated merge candidate CUs, in exactly the same way as on the decoder side.

[0149] o In the case of the sixth and seventh embodiments, the LIC parameter index is propagated from the RD-evaluated merge candidate CU.

[0150] -In the AMVP model, it involves Fig.24 RD selection between using LIC and not using LIC is shown. When the test LIC flag value is equal to true, this includes the following.

[0151] o In the case of the five first embodiments, the same determination of the LIC parameters for the current CU is performed as on the decoder side. This corresponds to Fig. 20 The process described in .

[0152] o In the case of the sixth and seventh embodiments, the rate-distortion optimization loop may be dedicated to selecting the best LIC parameters for the current CU among the LIC parameters currently stored in the dedicated memory buffer. Thus, the index of the best LIC parameter candidate is selected to be used as the LIC for the current CU, i.e., resulting in the minimum rate-distortion cost. Then, if the encoder RD search process determines that LIC should be used for the current CU, a LIC flag equal to true and the found best LIC parameter index are transmitted in the bitstream.

[0153] -Finally, according to any embodiment, each time a CU is encoded, the LIC parameters associated with that CU are calculated in exactly the same way as at the decoder side. This means that in the case of the five first embodiments, part of the current reconstructed CU (the lower part of the right part) is used. In the case of the sixth embodiment, the entire reconstructed CU can be used.

[0154] In one embodiment of the present application, a modified LIC parameter calculation, storage and use is proposed, which allows propagation in merge mode in the case of inter-frame decoding units decoded in AMVP mode. This mainly means the AMVP decoding / decoding described previously. In fact, the merge decoding mode consists in deriving the motion information of the CU from the neighboring decoded / decoded CUs (called merge candidates). The best merge candidate is usually selected from the RD cost view on the encoder side, and the merge index identifying the selected merge candidate is transmitted in the bitstream.

[0155] According to the present application, the IC flag associated with the merge candidate CU is propagated to the current CU predicted in the merge mode. This means that the CU in the merge mode is also assigned an IC flag equal to the IC flag of the merge candidate CU selected for predicting the motion information of the current CU.

[0156] Regarding LIC parameters, the current CU in merge mode also inherits the LIC parameters associated with the selected merge candidate.

[0157] Therefore, the motion compensated temporal prediction of this merged CU includes the application of illumination variation compensation in exactly the same way as for inter-AMVP coding units.

[0158] A particular aspect of the present application relates to the case where a CU level index is used to identify which stored LIC parameters are used for temporal prediction of the current CU. This relates to the sixth and seventh embodiments explained above. If the current CU is in merge mode, it inherits the LIC parameter index of the selected merge candidate.

[0159] Another specific aspect relates to CUs that are inter-coded in merge mode and employ bidirectional temporal prediction. These CUs derive motion information that may come from different candidates that use unidirectional LIC. In this case, LIC is traditionally not allowed in bidirectionally predicted CUs. Here, in the case of the fifth embodiment involving predetermined LIC parameters, LIC can be activated. As already stated, we also call this embodiment unidirectional generalized prediction. In the case where two selected candidates employ generalized unidirectional prediction with the same predetermined parameters (a, b), then the linear model using these (a, b) parameters is also applied to the bidirectional temporal prediction of the current CU. Otherwise, if neither merge candidate employs generalized unidirectional prediction or both of them use generalized unidirectional prediction but with different parameters, then the linear model is not used in the bidirectional prediction of the current CU.

[0160] Fig.28 A flow chart of LIC parameter determination according to at least one embodiment of the present application is described. In a first step, the LIC parameters are determined according to one of the above-described embodiments. In a second step, the LIC parameters are stored in a buffer according to one of the above-described embodiments. In a third step, the use of the LIC parameters and the index of the LIC parameters in the buffer are signaled. In at least one embodiment, when the LIC parameters are determined before providing the content (e.g., at the encoder), this is done, for example, using some dedicated syntax embedded in the signal. In at least another embodiment, when the LIC parameters are determined after content access (e.g., at the decoder), this is done, for example, using some internal variables.

[0161] Fig.29 Flowchart describing the use of LIC parameters according to at least one embodiment of the present application. In a first step, it is checked whether the LIC parameters should be used. An example of the check is to detect the presence of appropriate signaling within the video content, for example in the form of flags and indices. If the LIC parameters are to be used, these parameters are accessed (e.g. obtained from a buffer storing them) and used to reconstruct the video (e.g. by applying the LIC parameters to reconstruct a decoding unit of a coding tree unit to compensate for changes in illumination on the decoding unit).

[0162] Various implementations involve decoding. "Decoding" as used in this application may include, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also or alternatively includes various embodiments described herein (e.g., Fig.29 The process performed by the decoder of the embodiment presented in .

[0163] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0164] Various implementations involve encoding. In a manner similar to the discussion above regarding "decoding", "encoding" as used in this application may include, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as segmentation, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such processes also or alternatively include various implementations described in this application (e.g., Fig.29 The process performed by the encoder of the embodiment of the present invention.

[0165] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will become clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0166] Note that the syntax elements used here are descriptive terms. Therefore, they do not exclude the use of other syntax element names.

[0167] This application describes multiple aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described as being specific and, at least to illustrate individual characteristics, are often described in a manner that may sound restrictive. However, this is for the purpose of clarity of description and does not limit the application or scope of those aspects. In fact, all the different aspects can be combined and interchanged to provide additional aspects. In addition, these aspects can also be combined and interchanged with aspects described in earlier applications. The aspects described and contemplated in this application can be implemented in many different forms. The above Figure 1 , Figure 2 and Figure 3 Some embodiments are provided, but other embodiments are contemplated, and the discussion of the figures does not limit the breadth of implementations.

[0168] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "memory" and "buffer" are used interchangeably, the term "image", the term "pixel" and "sample" are used interchangeably, the terms "image", "picture" and "frame" are used interchangeably, and the terms "index" and "idx" are used interchangeably. Usually, but not necessarily, the term "reconstruction" is used on the encoder side, and "decoding" is used on the decoder side.

[0169] Various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0170] Various numerical values ​​are used in this application, such as regarding block size. The specific values ​​are for example purposes, and the described aspects are not limited to these specific values.

[0171] References to "one embodiment" or "an embodiment" or "one implementation" or "implementation" and other variations mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations in various places throughout this document are not necessarily all referring to the same embodiment.

[0172] Additionally, the application or its claims may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0173] Furthermore, this application or its claims may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, predicting information, or estimating information.

[0174] Additionally, the application or its claims may refer to "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory or optical media storage device). Furthermore, during operations such as storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is generally referred to in one way or another.

[0175] It should be understood that, for example, in the case of "A / B", "A and / or B", and "at least one of A and B", the use of any of the following " / ", "and / or", and "at least one of..." is intended to cover the selection of only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to include the selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to many of the listed items, as will be apparent to one of ordinary skill in this and related arts.

[0176] As will be appreciated by those skilled in the art, implementations may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

Claims

1. A method for video decoding, the method comprising: Get local illumination compensation (LIC) information; selecting a first LIC parameter set for a first decoding block based on the LIC information; as well as The first coded block is decoded based on the determined LIC parameter set.

2. The method of claim 1, wherein the LIC information comprises a LIC parameter candidate list and a LIC parameter index associated with the LIC parameter candidate list, wherein the first LIC parameter set is selected from the LIC parameter candidate list based on the LIC parameter index.

3. The method according to claim 1, wherein the LIC information comprises a LIC parameter candidate list, and the method further comprises: For the first coding block, a LIC parameter index associated with the LIC parameter candidate list is obtained, wherein the first LIC parameter set is further selected based on the LIC parameter index and the LIC parameter candidate list.

4. The method according to claim 1, wherein the LIC information comprises a LIC parameter candidate list, and the method further comprises: For the second decoding block, selecting a third LIC parameter set from the LIC parameter candidate list based on the LIC parameter candidate list; The second coded block is decoded based on the selected third LIC parameter set.

5. The method of claim 4, wherein the third LIC parameter set is the second LIC parameter set.

6. The method of claim 1, wherein the LIC information comprises a LIC parameter candidate list, and the method further comprises: Obtaining a reconstructed sub-block of the first decoding block; determining a second LIC parameter set based on the reconstructed sub-block of the first decoding block; as well as The second LIC parameter set is added to the LIC parameter candidate list.

7. The method of claim 6, wherein the LIC information further comprises a LIC parameter index associated with the LIC candidate list, wherein the reconstructed subblock of the first coded block is a right vertical subblock of the first coded block, and wherein the second LIC parameter set is indexed in the LIC parameter candidate list as a function of a vertical position associated with the subblock of the first coded block.

8. The method of claim 6, wherein the LIC information further comprises a LIC parameter index associated with the LIC candidate list, wherein the reconstructed subblock is a lower horizontal subblock of the first coded block, and wherein the second LIC parameter set is indexed in the LIC parameter candidate list as a function of a horizontal position associated with the subblock of the first coded block.

9. The method of claim 6, wherein determining the second LIC parameter set further comprises: Determine a vertical LIC parameter set based on a right vertical sub-block of the first decoding block; as well as A horizontal LIC parameter set is determined based on a lower horizontal subblock of the first coding block, wherein the vertical LIC parameter set is indexed in the LIC parameter candidate list as a function of a vertical position of a subblock associated with a right vertical subblock of the first coding block, and wherein the horizontal LIC parameter set is indexed in the LIC parameter candidate list as a function of a horizontal position associated with a lower horizontal subblock of the first coding block.

10. The method according to claim 1, wherein the method further comprises: An instruction to perform LIC on the first decoding block is received.

11. A method for video encoding, the method comprising: Get local illumination compensation (LIC) information; selecting a LIC parameter set for a first decoding block based on the LIC information; as well as encoding the first coded block based on the selected LIC parameter set; as well as An indication of an index to the first LIC parameter set for the first coding block is included in video data.

12. The method of claim 11, wherein the LIC information comprises a LIC parameter candidate list and a LIC parameter index associated with the LIC parameter candidate list, wherein the first LIC parameter set is selected from the LIC parameter candidate list based on the LIC parameter index.

13. The method according to claim 11, wherein the LIC information comprises a LIC parameter candidate list, and the method further comprises: For the first coding block, a LIC parameter index associated with the LIC parameter candidate list is obtained, wherein the first LIC parameter set is further selected based on the LIC parameter index and the LIC parameter candidate list.

14. The method of claim 11, wherein the LIC information comprises a LIC parameter candidate list, and wherein the method further comprises: For the second decoding block, selecting a third LIC parameter set from the LIC parameter candidate list based on the LIC parameter candidate list; encoding a second decoding block based on the third LIC parameter set; as well as An index to the third LIC parameter set for the second coding block is included in video data.

15. The method of claim 14, wherein the third LIC parameter set is the second LIC parameter set.

16. The method of claim 11, wherein the LIC information comprises a LIC parameter candidate list, and the method further comprises: Obtaining a reconstructed sub-block of the first block; determining a second LIC parameter set based on the reconstructed sub-block of the first block; adding the second LIC parameter set to the LIC parameter candidate list; as well as An indication of an index to the second LIC parameter set is included in the video data.

17. The method of claim 16, wherein the LIC information comprises a LIC parameter candidate list, wherein the reconstructed subblock of the first block is a right vertical subblock of the first coded block, and wherein the second LIC parameter set is indexed in the LIC parameter candidate list as a function of a vertical position associated with the subblock of the first coded block.

18. The method of claim 16, wherein the LIC information comprises a LIC parameter candidate list, wherein the reconstructed subblock of the first block is a lower horizontal subblock of the first coded block, and wherein the second LIC parameter set is indexed in the LIC parameter candidate list as a function of a horizontal position associated with the subblock of the first coded block.

19. The method of claim 16, wherein determining the second LIC parameter set further comprises: determining a vertical LIC parameter set based on a right vertical sub-block of the first block; as well as A horizontal LIC parameter set is determined based on a lower horizontal subblock of the first coding block, wherein the vertical LIC parameter set is indexed in the LIC parameter candidate list as a function of a vertical position of a subblock associated with a right vertical subblock of the first coding block, and wherein the horizontal LIC parameter set is indexed in the LIC parameter candidate list as a function of a horizontal position associated with a lower horizontal subblock of the first coding block.

20. An apparatus for video decoding, the apparatus comprising a processor, the processor being configured to: Obtaining local illumination compensation (LIC) information; selecting a first LIC parameter set for a first coding block based on the LIC information; and The first coded block is decoded based on the determined LIC parameter set.

21. The apparatus of claim 20, wherein the LIC information comprises a LIC parameter candidate list and a LIC parameter index associated with the LIC parameter candidate list, wherein the first LIC parameter set is selected from the LIC parameter candidate list based on the LIC parameter index.

22. The apparatus of claim 20, wherein the LIC information comprises a LIC parameter candidate list, and the processor is further configured to: For the first coding block, a LIC parameter index associated with the LIC parameter candidate list is obtained, wherein the first LIC parameter set is further selected based on the LIC parameter index and the LIC parameter candidate list.

23. The method of claim 20, wherein the LIC information comprises a LIC parameter candidate list, and the processor is further configured to: For the second decoding block, selecting a third LIC parameter set from the LIC parameter candidate list based on the LIC parameter candidate list; The second coded block is decoded based on the selected third LIC parameter set.

24. The method of claim 23, wherein the third LIC parameter set is the second LIC parameter set.

25. The method of claim 20, wherein the LIC information comprises a LIC parameter candidate list, and the processor is further configured to: Obtaining a reconstructed sub-block of the first decoding block; determining a second LIC parameter set based on the reconstructed sub-block of the first decoding block; and The second LIC parameter set is added to the LIC parameter candidate list.

26. The method of claim 25, wherein the LIC information further comprises a LIC parameter index associated with the LIC candidate list, wherein the reconstructed subblock of the first coded block is a right vertical subblock of the first coded block, and wherein the second LIC parameter set is indexed in the LIC parameter candidate list as a vertical position associated with the subblock of the first coded block.

27. The method of claim 25, wherein the LIC information further comprises a LIC parameter index associated with the LIC candidate list, wherein the reconstructed subblock is a lower horizontal subblock of the first coding block, and wherein the second LIC parameter set is indexed in the LIC parameter candidate list as a horizontal position associated with the subblock of the first coding block.

28. The method of claim 25, wherein determining the second LIC parameter set further comprises: Determine a vertical LIC parameter set based on a right vertical sub-block of the first decoding block; as well as A horizontal LIC parameter set is determined based on a lower horizontal subblock of the first coding block, wherein the vertical LIC parameter set is indexed in the LIC parameter candidate list as a function of a vertical position of a subblock associated with a right vertical subblock of the first coding block, and wherein the horizontal LIC parameter set is indexed in the LIC parameter candidate list as a function of a horizontal position associated with a lower horizontal subblock of the first coding block.

29. The method of claim 20, wherein the processor is further configured to: An instruction to perform LIC on the first decoding block is received.

30. An apparatus for video encoding, the apparatus comprising a processor, the processor being configured to: Obtaining local illumination compensation (LIC) information; selecting a LIC parameter set for a first coding block based on the LIC information; and encoding the first coded block based on the selected LIC parameter set; and An indication of an index to the first LIC parameter set for the first coding block is included in video data.

31. The method of claim 30, wherein the LIC information comprises a LIC parameter candidate list and a LIC parameter index associated with the LIC parameter candidate list, wherein the first LIC parameter set is selected from the LIC parameter candidate list based on the LIC parameter index.

32. The method of claim 30, wherein the LIC information comprises a LIC parameter candidate list, and the processor is further configured to: For the first coding block, a LIC parameter index associated with the LIC parameter candidate list is obtained, wherein the first LIC parameter set is further selected based on the LIC parameter index and the LIC parameter candidate list.

33. The method of claim 30, wherein the LIC information comprises a LIC parameter candidate list, and wherein the processor is further configured to: For the second decoding block, selecting a third LIC parameter set from the LIC parameter candidate list based on the LIC parameter candidate list; Encoding a second decoding block based on the third LIC parameter set; and An index to the third LIC parameter set for the second coding block is included in video data.

34. The method of claim 33, wherein the third LIC parameter set is the second LIC parameter set.

35. The method of claim 30, wherein the LIC information comprises a LIC parameter candidate list, and the processor is further configured to: Obtaining a reconstructed sub-block of the first block; determining a second LIC parameter set based on the reconstructed sub-block of the first block; adding the second LIC parameter set to the LIC parameter candidate list; as well as An indication of an index to the second LIC parameter set is included in the video data.

36. The method of claim 35 , wherein the LIC information comprises a LIC parameter candidate list, wherein the reconstructed subblock of the first block is a right vertical subblock of the first coded block, and wherein the second LIC parameter set is indexed in the LIC parameter candidate list as a function of a vertical position associated with the subblock of the first coded block.

37. The method of claim 35, wherein the LIC information comprises a LIC parameter candidate list, wherein the reconstructed subblock of the first block is a lower horizontal subblock of the first coded block, and wherein the second LIC parameter set is indexed in the LIC parameter candidate list as a function of a horizontal position associated with the subblock of the first coded block.

38. The method of claim 35, wherein determining the second LIC parameter set further comprises: determining a vertical LIC parameter set based on a right vertical sub-block of the first block; as well as A horizontal LIC parameter set is determined based on a lower horizontal subblock of the first coding block, wherein the vertical LIC parameter set is indexed in the LIC parameter candidate list as a function of a vertical position of a subblock associated with a right vertical subblock of the first coding block, and wherein the horizontal LIC parameter set is indexed in the LIC parameter candidate list as a function of a horizontal position associated with a lower horizontal subblock of the first coding block.

39. A method comprising: determining a first local illumination compensation (LIC) parameter associated with the reconstructed decoding unit; obtaining a LIC parameter set, wherein the LIC parameter set includes at least a second LIC parameter; determining whether to store a first LIC parameter based on the LIC parameter set; determining whether to discard at least one LIC parameter from the LIC parameter set based on determining whether to store the first LIC parameter and based on the buffer size; as well as At least one decoding function or encoding function is performed based on the first LIC parameters.

40. The method of claim 39, wherein the method further comprises: and determining a number of LIC parameters associated with the LIC parameter set, wherein determining to discard at least one LIC parameter from the LIC parameter set comprises determining to store the first LIC parameter and determining that the number of LIC parameters associated with the LIC parameter set exceeds the buffer size.

41. The method of claim 39, wherein determining to store the first LIC parameter comprises determining that the first LIC parameter is different from the second LIC parameter.

42. The method of claim 39, wherein the method further comprises: The LIC parameter set is indicated in video data, wherein the LIC parameter set includes at least one of the first LIC parameters or the second LIC parameters.

43. The method of claim 39, wherein the method further comprises: Based on determining to discard at least one LIC parameter from the LIC parameter set, a third LIC parameter from the LIC parameter set is determined to be discarded.

44. The method of claim 43, wherein the third LIC parameter is a second LIC parameter.

Images