Method and apparatus for encoding and decoding blocks based on illumination compensation

Abstract

Methods (800, 1600, 1700) and apparatus (1800) for video encoding and decoding are provided. The method (800, 1600, 1700) includes deriving one or more illumination compensation parameters based on neighboring samples of a subblock of a block and neighboring samples of a reference subblock for motion compensation; deriving a predicted subblock by applying illumination compensation on the reference subblock for motion compensation using the derived one or more illumination compensation parameters. A computer-readable storage medium and a computer program product are also described.

Description

Technical Field

[0001] The present embodiments generally relate to video encoding and decoding using illumination compensation. Background Art

[0002] This section is intended to introduce the reader to various aspects of the art, which may be related to various aspects of at least one of the present embodiments described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of various aspects of at least one embodiment. Therefore, it should be understood that these statements should be understood from this perspective.

[0003] In order to achieve high compression efficiency, video coding schemes usually use prediction and transformation to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlation. Then, the difference between the original picture and the predicted picture of the video (usually expressed as prediction error or prediction residual) is transformed, quantized and entropy encoded and decoded. In order to reconstruct the picture, the compressed data is decoded by the inverse process corresponding to prediction, transformation, quantization and entropy encoding and decoding. Summary of the invention

[0004] This section provides a simplified overview of at least one of the present embodiments in order to provide a basic understanding of some aspects of the present disclosure. This overview is not an extensive overview of the embodiments. It is not intended to identify the key or important elements of the embodiments. The following overview merely presents some aspects of at least one of the present embodiments in a simplified form as a preface to a more detailed description provided elsewhere in the document.

[0005] According to a general aspect of at least one of the present embodiments, there is provided a method for encoding a block of a video, comprising: deriving one or more illumination compensation parameters based on neighboring samples of a subblock of the block and neighboring samples of a reference subblock for motion compensation; deriving a predicted subblock by applying illumination compensation on a reference subblock for motion compensation using the derived one or more illumination compensation parameters; and encoding the subblock using the predicted subblock.

[0006] According to another general aspect of at least one of the present embodiments, there is provided a method for decoding a block of a video, comprising: deriving one or more illumination compensation parameters based on neighboring samples of a subblock of the block and neighboring samples of a reference subblock for motion compensation; deriving a predicted subblock by applying illumination compensation on a reference subblock for motion compensation using the derived one or more illumination compensation parameters; and decoding the subblock using the predicted subblock.

[0007] According to another general aspect of at least one of the present embodiments, there is provided an apparatus for encoding a block of a video, comprising components for performing the following operations: deriving one or more illumination compensation parameters based on neighboring samples of a subblock of the block and neighboring samples of a reference subblock for motion compensation; deriving a predicted subblock by applying illumination compensation on a reference subblock for motion compensation using the derived one or more illumination compensation parameters; and encoding the subblock using the predicted subblock.

[0008] According to another general aspect of at least one of the present embodiments, there is provided an apparatus for decoding a block of a video, comprising components for performing the following operations: deriving one or more illumination compensation parameters based on neighboring samples of a subblock of the block and neighboring samples of a reference subblock for motion compensation; deriving a predicted subblock by applying illumination compensation on a reference subblock for motion compensation using the derived one or more illumination compensation parameters; and decoding the subblock using the predicted subblock.

[0009] According to an embodiment, multiple subblocks of the same block are encoded using predicted subblocks, and encoding or decoding the subblocks includes applying a separate transform to each residual derived from the subblock and the associated predicted subblock.

[0010] According to an embodiment, the neighboring samples of the current sub-block are samples inside the block.

[0011] According to an embodiment, encoding or decoding a sub-block comprises applying a single transform to a prediction residual derived from the sub-block and a prediction residual derived from a predicted sub-block.

[0012] According to an embodiment, one or more neighboring samples of the current sub-block are samples outside the block.

[0013] According to an embodiment, neighboring samples of a subblock and the predicted subblock are formed only from samples of at least one further predicted subblock.

[0014] According to an embodiment, neighboring samples of a subblock and the predicted subblock are formed from samples outside the block and samples of at least one further predicted subblock inside the block.

[0015] According to an embodiment, one or more illumination compensation parameters are derived from neighboring samples within a subset of sub-blocks of the block.

[0016] According to other general aspects of at least one of the present embodiments, a non-transitory computer-readable storage medium and a computer program product are provided.

[0017] Specific properties of at least one of the present embodiments as well as other objects, advantages, features and uses of at least one of the present embodiments will become apparent from the following description of examples taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In the accompanying drawings, examples of several embodiments are shown. The accompanying drawings show:

[0019] Figure 1 shows an L-shaped set of reconstructed neighboring samples for a current block in a current picture, for a reference 0 block in a reference 0 picture, and for a reference 1 block in a reference 1 picture according to the prior art;

[0020] Figure 2 and 3 Some inconveniences of the illumination compensation method according to the prior art are shown;

[0021] Figure 4 shows a simplified block diagram of an exemplary encoder according to an embodiment;

[0022] Figure 5 A simplified block diagram 500 showing some modules of an exemplary encoder according to at least one embodiment;

[0023] Figure 6 shows a simplified block diagram of an exemplary decoder 600 according to at least one embodiment;

[0024] Figure 7 A simplified block diagram 700 showing some modules of an exemplary decoder according to at least one embodiment;

[0025] Figure 8 A flowchart 800 showing a method of encoding / decoding a block based on illumination compensation according to at least one embodiment;

[0026] Fig. 9 An example of subdividing a block according to at least one embodiment is shown;

[0027] Fig.10 A flow chart illustrating a method 1000 of deriving reference sub-blocks for motion compensation of a block according to at least one embodiment;

[0028] Fig.11 shows an example of motion vectors for sub-blocks based on an affine motion model according to at least one embodiment;

[0029] Fig.12 A flowchart of a method 1200 for deriving one or more IC parameters and adjusting a reference sub-block for motion compensation according to the present embodiment is shown;

[0030] Fig.13a -d shows an L-shaped set definition according to different sub-partitions of a block according to at least one embodiment;

[0031] Fig.14A flow chart showing a method 1400 of encoding a sub-block of a block from a predicted sub-block according to at least one embodiment;

[0032] Fig.15 It shows the case where 4 VPDUs are created to group sub-blocks of a block;

[0033] Fig.16 A flowchart of an exemplary method of video encoding according to this embodiment is shown;

[0034] Fig.17 A flowchart showing an exemplary method of video decoding according to this embodiment; and

[0035] Fig.18 A block diagram of a computing environment is shown in which aspects of the present disclosure may be implemented and executed. DETAILED DESCRIPTION

[0036] This detailed description illustrates the principles of the present embodiment. It will therefore be appreciated that those skilled in the art will be able to design various arrangements that, although not explicitly described or shown herein, embody the principles of the present embodiment and are included within its scope.

[0037] All examples and conditional language listed herein are for educational purposes to help readers understand the principles of the present embodiments and the concepts contributed by the inventors to promote the art, and should be interpreted as not being limited to these specifically listed examples and conditions.

[0038] In addition, all statements and specific examples of the principles, aspects and embodiments of the present disclosure are intended to include structural and functional equivalents thereof. In addition, such equivalents are intended to include currently known equivalents and equivalents developed in the future, i.e., any element of the developed performance of the same function regardless of structure.

[0039] Thus, for example, those skilled in the art will appreciate that the block diagrams presented herein represent conceptual diagrams of illustrative circuits embodying the principles of the present disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo-codes, etc. represent various processes that can be substantially represented in a computer-readable medium and executed by a computer or processor, whether or not such a computer or processor is explicitly shown.

[0040] Hereinafter, the present embodiment will be described more fully with reference to the accompanying drawings, in which examples of the present embodiment are shown. However, the embodiment can be implemented in a variety of alternative forms and should not be construed as being limited to the examples set forth herein. Therefore, it should be understood that the embodiment is not intended to be limited to the specific form disclosed. On the contrary, the present embodiment is intended to cover all modifications, equivalents, and substitutes falling within the spirit and scope of the present application.

[0041] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a figure is presented in the form of a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0042] The functions of the various elements shown in the drawings may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, a single shared processor, or multiple separate processors, some of which may be shared. In addition, the explicit use of the term "processor" or "controller" should not be interpreted as referring exclusively to hardware capable of executing software, but may implicitly include, but is not limited to, digital signal processor (DSP) hardware, read-only memory (ROM), random access memory (RAM), and non-volatile memory for storing software.

[0043] Other hardware, conventional and / or custom, may also be included. Similarly, any switches shown in the figure are only conceptual. Their functions may be implemented through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, and the specific technique may be selected by the implementer as more specifically understood from the context.

[0044] Similar or identical elements in the drawings are denoted by the same reference numerals.

[0045] Some figures may represent syntax tables widely used in video compression standard specifications for defining bitstream structures that conform to the video compression standards. In those syntax tables, the term "..." represents an unchanged portion of the syntax relative to the well-known definition given in the specification of the video compression standard, and is removed in the figure for ease of reading. The bold terms in the syntax table indicate that the value of the term is obtained by parsing the bitstream. The right column of the syntax table indicates the number of bits used to encode the data of the syntax element. For example, u(4) indicates that 4 bits are used to encode the data, u(8) indicates 8 bits, and ae(v) indicates a syntax element for context-adaptive arithmetic entropy coding and decoding.

[0046] In the claims herein, any element represented as a means for performing a particular function is intended to encompass any means of performing that function, including, for example, a) a combination of circuit elements that perform that function, or b) any form of software, thus including firmware, microcode, etc., combined with appropriate circuitry for executing the software to perform that function. The present embodiments defined by such claims reside in the fact that the functions provided by the various enumerated components are combined and brought together in the manner required by the claims. Therefore, any components that can provide those functions are considered equivalent to those shown herein.

[0047] It should be understood that the drawings and descriptions have been simplified to show elements relevant to a clear understanding of the present embodiments, while many other elements found in typical encoding and / or decoding devices have been eliminated for clarity.

[0048] It should be understood that although the terms first and second can be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish an element from another element. Various methods are described above, and each method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined.

[0049] In the following sections, the words "reconstruction" and "decoding" are used interchangeably. Usually, but not necessarily, "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side. In addition, the words "codec" and "encoding" are used interchangeably. In addition, the words "image", "picture" and "frame" are used interchangeably. In addition, the words "codec", "source codec" and "compression" are used interchangeably.

[0050] It should be understood that a picture (also denoted as an image or frame) can be a luma sampling array in a monochrome format, or a luma sampling array and two corresponding chroma sampling arrays in a 4:2:0, 4:2:2 or 4:4:4 color format, or three arrays of three color components (e.g., RGB).

[0051] In video compression standards, a picture is divided into blocks that may be of different sizes and / or different shapes. It should be understood that a block is a two-dimensional array or matrix. The horizontal or x direction (or axis) represents the width, and the vertical or y direction (or axis) represents the height. Indexing starts at 0. The x direction represents the columns, and the y direction represents the rows. The maximum x index is the width - 1. The maximum y index is the height - 1.

[0052] The present embodiment is directed to block-based illumination compensation (IC) used in inter-prediction mode to adjust block prediction samples obtained via motion compensation (MC) by taking into account any spatial or temporal local illumination variations.

[0053] In the prior art, IC parameters are estimated by comparing an L-shaped set of reconstructed neighboring samples of a current block in a current picture with a corresponding L-shaped set of reconstructed neighboring samples of a reference block in a reference picture. In the case of bi-prediction (i.e., there are two reference blocks for each current block), the corresponding L-shaped sets of reconstructed neighboring samples of two reference i blocks (i=0 or 1) are compared with the L-shaped sets of reconstructed neighboring samples of the current block, respectively.

[0054] Figure 1 An L-shaped set 170 of neighboring samples of a current block 160 in a current picture 150 according to the prior art, a corresponding L-shaped set (L-shape-ref-0) 130 of a motion compensated reference 0 block (ref-0, also referred to herein as MC-0 or motion compensated 0 block) 120 in a reference 0 picture 110, and a corresponding L-shaped set (L-shape-ref-1) 135 of a motion compensated reference 1 block (ref-1, also referred to herein as MC-1 or motion compensated 1 block) 125 in a reference 1 picture 115 are shown. Therefore, the term reference i block or ref-i (i=0,1) below corresponds to a motion compensated reference block, i.e., a reference block located at position (x+MV) in reference picture “i”. ix ,y+MV iy ), where (x, y) is the position of the current block in the current picture, and MV represents the motion vector. Figure 1 , the motion vector between the current block 160 and the reference 0 block 120 is identified as MV0 140 , and the motion vector between the current block 160 and the reference 1 block 125 is identified as MV1 145 .

[0055] Block-based illumination compensation (IC) can use an illumination compensation (IC) model, which is usually linear and defined by the following equation:

[0056] IC(x)=a*x+b (1)

[0057] where a and b are the IC parameters, usually called the slope and intercept of the linear function, respectively.

[0058] In particular, IC parameters may be estimated, for example, by minimizing the difference between samples in the L-shaped set 170 of the current block and samples in the L-shaped set (130 or 135) of the reference block (120 or 125) adjusted (corrected) with illumination compensation using the IC model. Under the least squares method, the difference between samples may be minimized as follows:

[0059]

[0060] Among them, (a i ,b i ) are the optimal IC parameters associated with the reference module, argmin(.) are the independent variables for the minimum, and the summation is performed over each pairing of a sample x in the L-shape set L-shape-cur sampled in the current picture with its corresponding sample y in the L-shape set L-shape-ref sampled in the reference picture.

[0061] According to the prior art, the idea of ​​using an L-shaped set of neighboring samples relies on the assumption that the IC parameters optimized for the neighboring samples of the current block are still suitable for the current block. This assumption is usually correct because the neighboring samples of the L-shaped set are the available samples closest to the current block samples. However, this approach introduces pipeline dependencies for inter-frame prediction because it is necessary to wait for neighboring blocks to be reconstructed before constructing the current prediction block, such as Figure 2 As shown in FIG. 1 , the decoding of the current block CU depends on the adjacent reconstructed samples of the neighboring blocks CU1 and CU2. In addition, the IC parameters are estimated using samples that may be relatively far away from the samples to be adjusted, such as Figure 3 shown.

[0062] The present embodiment solves some shortcomings existing in the prior art. Specifically, the present embodiment derives one or more illumination compensation parameters based on neighboring samples of a sub-block of a current block and neighboring samples of a reference sub-block for motion compensation, derives a predicted sub-block by applying illumination compensation to a reference sub-block for motion compensation using the derived one or more illumination compensation parameters, and encodes the sub-block using the predicted sub-block, which will be further described in the following paragraphs and drawings.

[0063] coding

[0064] Figure 4 A simplified block diagram of an exemplary encoder 400 is shown in accordance with at least one embodiment.

[0065] The encoder 400 may be included in a transmitter or a headend of a communication system.

[0066] In order to encode a video sequence with one or more pictures, the pictures are partitioned into blocks of possibly different sizes and / or shapes (module 410). For example, in HEVC ("ITU-T H.265 ITU Telecommunication Standardization Sector (10 / 2014), H Series: Audiovisual and Multimedia Systems, Infrastructure for Audiovisual Services - Motion Video Codec, High Efficiency Video Codec, Recommendation ITU-T H.265"), a picture may be partitioned into square CTUs (Codec Tree Units) of configurable size. A set of consecutive CTUs may be grouped into slices. A CTU is the root of a quadtree partitioned into blocks, usually denoted CU (Codec Unit).

[0067] In the exemplary encoder 400, pictures are encoded by a block-based video encoder module as described below.

[0068] Each block is encoded using either an intra prediction mode or an inter prediction mode.

[0069] When encoding a block in intra prediction mode (module 460), the encoder 400 performs intra prediction (also called spatial prediction) based on at least one block in the same picture. As an example, the predicted block is obtained by intra prediction of a block from reconstructed neighboring samples.

[0070] When encoding a block in the inter prediction mode, the encoder 400 performs inter prediction (also called temporal prediction) based on at least one reference block in at least one reference picture.

[0071] Inter-frame prediction coding is performed by performing motion estimation (module 475) and motion compensation (module 470) on reference blocks stored in the reference picture buffer 480.

[0072] In uni-directional inter prediction modes, the predicted block is typically (but not necessarily) based on an earlier reference picture.

[0073] In bi-directional inter prediction mode, the predicted block is usually (but not necessarily) based on earlier and later picture frames.

[0074] The encoder decides (block 405) which of intra prediction mode or inter prediction mode to use for encoding the block and indicates the intra / inter decision via a prediction mode syntax element.

[0075] The prediction residual block is calculated by subtracting (module 420) the prediction block (also called predictor) from the block.

[0076] The prediction residual block is transformed (module 425) and quantized (module 430). The transform module 425 can transform the block from the pixel (spatial) domain to the transform (frequency) domain. The transform can be, for example, a cosine transform, a sine transform, a wavelet transform, etc. Quantization (module 430) can be performed according to, for example, a rate-distortion criterion.

[0077] The quantized transform coefficients and motion vectors and other syntax elements are entropy coded (module 445) to output a bitstream. The entropy codec can be, for example, context adaptive binary arithmetic coding (CABAC), context adaptive variable length coding (CAVLC), Huffman, arithmetic, exponential Golomb (exp-Golomb), etc.

[0078] The encoder may also skip the transform and apply quantization directly to the untransformed prediction residual block. The encoder may also bypass both the transform and quantization, i.e., directly encode and decode the prediction residual block without applying the transform or quantization process.

[0079] In direct PCM coding, no prediction is applied and the block samples are encoded directly into the bitstream.

[0080] The encoder 400 includes a decoding loop and thus decodes the encoded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (module 440) and inverse transformed (module 450) to decode the prediction residual block. The block is reconstructed by combining (module 455) the decoded prediction residual block and the prediction block. An in-loop filter (465) can be applied to the reconstructed picture, for example, to perform deblocking / sampling adaptive offset (SAO) filtering to reduce encoding and decoding artifacts. The filtered picture is stored in a reference picture buffer 480.

[0081] The modules of the encoder 400 may be implemented in software and executed by a processor, or may be implemented using circuit components known to those skilled in the art of compression. In particular, the video encoder 400 may be implemented as an integrated circuit (IC).

[0082] The modules of encoder 400 are present in a conventional block-based video encoder (e.g., a HEVC encoder), except for the differences described in the present embodiment, particularly the differences in the modules motion compensation 470 and / or motion estimation 475 based on illumination compensation, as will be described in more detail in the following paragraphs and figures.

[0083] With respect to functionality other than illumination compensation (eg, modules other than 470 and 475 ), encoder 400 may be similar to any conventional block-based video encoder, and these functions are not described in detail herein.

[0084] The motion estimation module 475 may include motion compensation, since its purpose is to determine the best motion vector, which may use an iterative search that typically terminates when the rate-distortion cost (RD cost) is sufficiently low or has reached a minimum. As a result, IC may also be applied in the motion estimation module 475.

[0085] Figure 5 A simplified block diagram 500 is shown illustrating some modules of an exemplary encoder in accordance with at least one embodiment.

[0086] In accordance with at least one embodiment, modules Motion Estimation 475 and Motion Compensation 470 specify IC-related functions within modules of an encoder.

[0087] According to the present embodiment, the module motion estimation 475 and motion compensation 470 may include a sub-division module 571 for dividing a block into sub-blocks.

[0088] Compared to conventional methods, sub-dividing the block makes the IC parameters spatially adapt to local features and reduces the maximum distance of the samples used to estimate the IC parameters to any adjusted samples, which will be further described in the following paragraphs and figures.

[0089] According to the present embodiment, the modules motion estimation 475 and motion compensation 470 may further include an internal motion compensation module 572 which performs motion compensation of reference sub-blocks.

[0090] According to the present embodiment, the modules motion estimation 475 and motion compensation 470 may further include an IC parameter derivation module 573 which derives IC parameters of a reference sub-block for motion compensation.

[0091] Finally, according to the present embodiment, the modules motion estimation 475 and motion compensation 470 may include an IC application module 574 that derives a predicted sub-block from the motion compensated reference sub-block and derived IC parameters.

[0092] The motion estimation module 475 may include motion compensation, since its purpose is to determine the best motion vector, which may use an iterative search that typically terminates when the rate distortion cost (RD cost) is sufficiently low or has reached a minimum. Different motion vector candidates are tested iteratively. The module motion estimation 475 may then include a RD cost calculation module 575 that determines the RD cost and determines whether a suitable value or minimum value of RD has been achieved in order to provide the best motion vector MV to the motion compensation module 570.

[0093] In one embodiment, the motion estimation module 575 may also output IC parameters associated with the best motion vector MV. In this case, the IC parameters do not need to be recalculated in module 570, and module 573 may be skipped from the motion compensation module 570.

[0094] Based on lighting compensation Figure 5 The modules in will be described in more detail in the following paragraphs and figures.

[0095] In one embodiment, IC can be adaptively enabled or disabled for each inter-frame prediction codec block. In the bitstream, when IC is enabled for a block, picture, slide, or sequence, an IC flag can be encoded for each block to indicate whether IC is enabled for the block. The IC flag can then be retrieved at the decoder. In another embodiment, the IC flag can be inferred from other codec parameters (e.g., merge index, block size).

[0096] In one embodiment, when the derived IC parameters do not result in a predicted improvement, then the IC may be locally or globally deactivated (eg, the IC flag is set to false).

[0097] In one embodiment, when the IC flag enables IC for a block (eg, the IC flag is set to true), IC parameters for the block may be included and optionally encoded in the bitstream to be retrieved at a decoder.

[0098] decoding

[0099] Figure 6 A simplified block diagram of an exemplary decoder 600 is shown in accordance with at least one embodiment.

[0100] Decoder 600 may be included in a receiver of a communication system.

[0101] Decoder 600 generally performs a decoding process that is the inverse of the encoding process performed by encoder 400, such as Figure 4 and 5 As described in , not all operations in the decoder are inverse operations of the encoding process (such as intra-frame and inter-frame prediction).

[0102] In particular, the input of the decoder 600 comprises a bitstream which may be generated by the encoder 400 .

[0103] The bitstream is first entropy decoded (module 630) to obtain transform coefficients, motion vectors MV, picture segmentation information, possible prediction mode information, other syntax elements and / or coding information.

[0104] For example, in HEVC, the picture partitioning information indicates the size of the CTU and the way in which the CTU is divided into CUs. Therefore, the decoder can divide (635) the picture into CTUs according to the picture partitioning information, and divide each CTU into CUs.

[0105] The transform coefficients are dequantized (module 640) and inverse transformed (module 650) to decode the prediction residual block. The decoded prediction residual block is then combined (module 655) with the prediction block (also called predictor) to obtain a decoded / reconstructed block.

[0106] Possibly depending on the prediction mode information, the prediction block may be obtained (module 605) from intra prediction (module 660) or motion compensated prediction (i.e., inter prediction) (module 670). An in-loop filter (module 665) may be applied to the reconstructed picture. The in-loop filter may include a deblocking filter and / or a SAO filter. The filtered picture is stored in a reference picture buffer 680.

[0107] The modules of decoder 600 can be implemented in software and executed by a processor, or can be implemented using circuit components known to those skilled in the art of compression. In particular, decoder 600 can be implemented as an integrated circuit (IC), alone or in combination with encoder 400 as a codec.

[0108] The modules of decoder 600 are also present in a conventional block-based video decoder (e.g., a HEVC decoder), except for the differences described in the present embodiment, in particular the differences in the motion compensation module 670 based on illumination compensation, as will be described in more detail in the following paragraphs and figures.

[0109] With respect to functionality other than illumination compensation (eg, modules other than 670 ), decoder 600 may be similar to any conventional block-based video decoder, and these functions are not described in detail herein.

[0110] Figure 7 A simplified block diagram 700 is shown illustrating some modules of an exemplary decoder in accordance with at least one embodiment.

[0111] Module 570 illustrates IC-related functions within the modules of the decoder according to the present embodiment.

[0112] The motion compensation 570 may include a sub-segmentation module 571 , an intra-motion compensation module 572 , an IC parameter derivation module 573 , and an IC application module 574 .

[0113] Based on lighting compensation Figure 7 The modules in will be described in more detail in the following paragraphs and figures.

[0114] In one embodiment, IC may be adaptively enabled or disabled for each inter-frame prediction codec block.

[0115] From the bitstream, when IC is enabled for the current slide / picture or sequence, an IC flag may be decoded for each block to indicate whether IC is enabled for the block.

[0116] Alternatively, the IC flag may be inferred, for example from a previously decoded block using merge mode. The IC flag is then retrieved at the video decoder.

[0117] In case the derived IC parameters do not result in an improvement in prediction, the IC may be locally or globally deactivated (e.g., the IC flag is set to false). When the IC flag enables IC for a block (e.g., the IC flag is set to true), the IC parameters for the block may be included and optionally decoded from the bitstream to be retrieved at the decoder.

[0118] It should be understood that when the IC flag and IC parameters are included and optionally decoded from the bitstream, module 570 retrieves the IC flag and IC parameters from the bitstream rather than calculating or deriving the IC parameters.

[0119] In the following, embodiments for deriving or calculating IC parameters according to embodiments are described.

[0120] Figure 8 A flow chart 800 is shown of a method of encoding / decoding a block based on illumination compensation in accordance with at least one embodiment.

[0121] In step 810, if Fig. 9 As shown, the block is sub-partitioned into sub-blocks. At the decoder, the sub-partitioning can be triggered by the picture partitioning information decoded from the bitstream.

[0122] Next, if Fig.10 and 11 As explained in detail in , a reference sub-block for motion compensation is obtained from the motion compensation (step 820); Fig.12 and 13a As explained in detail in step 8, IC parameters are derived from the neighboring samples of the motion compensated reference sub-block and the neighboring samples of the sub-block (step 830), and the predicted sub-block is derived by applying illumination compensation on the motion compensated reference sub-block using the IC parameters. Fig.14 As explained in detail in , the sub-block is encoded / decoded using the predicted sub-block (step 840). The method iterates for each sub-block of the block.

[0123] Steps 810 - 840 may be performed, for example, by encoder 400 or 500 or decoder 600 or 700 .

[0124] In particular, steps 810 - 830 may be performed by, for example, modules 470 , 475 and 471 of the encoder or module 670 of the decoder.

[0125] Fig. 9 An example of subdividing a block is shown in accordance with at least one embodiment (step 810).

[0126] In one embodiment, the block may be divided horizontally or vertically into at least two sub-blocks.

[0127] For example, when a block is divided horizontally (or vertically), the encoding and decoding of the sub-blocks are repeatedly applied downward (horizontal division) or rightward (vertical division), such as Fig. 9 As indicated by the arrow in .

[0128] The sub-block sizes (height and / or width) may be equal or unequal.

[0129] For example, the sub-block width (or height) can be equal to half of the block width (or height) or 1 / 4 of the block width (or height), resulting in a 2:2 partition or a 1:3 / 3:1 partition. The 2:2 partition is similar to a binary tree (BT) partition, while the 1:3 / 3:1 partition is similar to an asymmetric binary tree (ABT) partition. For example, the minimum block size is 4x8 (or 8x4). If the block size is larger than 4x8 (or 8x4), the corresponding block is divided into 4 sub-blocks.

[0130] In one embodiment, the number of sub-blocks may depend on the block size. For example, if one side of the block is 8, then 1:3 / 3:1 partitioning along that side is not allowed. For example, the expected minimum block size is 4x8 (or 8x4). If the block size is larger than 4x8 (or 8x4), the block is partitioned into 4 sub-blocks.

[0131] In one embodiment, the sub-partitioning may be signaled in the bitstream via syntax elements.Step 571, at the decoder, the syntax elements are then obtained and the block is partitioned according to the signaled sub-partitioning.

[0132] Fig.10 A flow chart of a method 1000 of deriving reference sub-blocks for motion compensation of a block is shown in accordance with at least one embodiment.

[0133] For example, a motion vector is estimated for each sub-block using any known block-based motion estimation method.

[0134] In a variant, the motion vector of each sub-block is derived from an affine motion model that captures an estimate of the translational motion field of the block. This variant improves temporal prediction.

[0135] like Fig.11 As shown in (left), the affine motion model can be defined by two control point motion vectors (4 parameters) or three control point motion vectors (6 parameters) or any number of control point motion vectors.

[0136] For a 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block can be derived as:

[0137]

[0138] For the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the block can be derived as:

[0139]

[0140] Where (mv 0x ,mv 0y ) is the motion vector of the upper left control point, (mv 1x ,mv 1y ) is the motion vector of the upper right control point, and (mv 2x ,mv 2y ) is the motion vector of the lower left control point.

[0141] Any known motion estimation method may be used to estimate the motion vectors at the control points.

[0142] For example, the motion vector of the center sample of each sub-block calculated according to the above equation and possibly rounded to 1 / 16 fractional accuracy is the motion vector of the sub-block, such as Fig.11 (right) shown.

[0143] Fig.12 A flowchart of a method 1200 for deriving one or more IC parameters and adjusting a reference sub-block for motion compensation according to the present embodiment is shown.

[0144] The following illumination compensation (IC) model may be used to perform illumination compensation on a reference sub-block for motion compensation.

[0145] The IC model is defined as follows:

[0146] IC(x sb )=a sb *x sb +b sb (3)

[0147] where a sb and b sb are the IC parameters of the reference sub-block sb for motion compensation, usually referred to as the slope or scale and the intercept or offset of the linear function, respectively.

[0148] In one embodiment, the neighboring samples of a sub-block may be as follows: Fig.13a -d shows an L-shaped set of neighboring samples. In a variant, not all neighboring samples of the L-shaped set are used, but only a part of them (eg, samples not reconstructed using intra mode).

[0149] The L-shaped set is defined as the set of causal neighboring samples of a sub-block, i.e., the neighboring samples that can be used to reconstruct the current sub-block. Fig.13a (horizontal division) and Fig.13b (vertical partitioning) as shown, some samples of the L-shape may be inside or outside the block.

[0150] Fig.13c and 13d Other L-shaped set definitions according to different sub-divisions of a block are shown. The arrows indicate the scanning order and neighboring sub-blocks to encode a sub-block of a block.

[0151] exist Fig.12 In step 1210, a set of neighboring samples of the current sub-block is obtained.

[0152] At step 1220, a reference subblock in a reference picture is pointed to by a motion vector of the subblock, and neighboring samples of the reference subblock are neighboring samples of a motion compensated reference subblock.

[0153] Once the IC parameter a sb and b sb By adjusting the neighboring samples of the current sub-block and the neighboring samples of the reference sub-block for motion compensation, the IC parameters can be estimated by minimizing the difference between these neighboring samples.

[0154] At step 1230, the differences between adjacent samples may be minimized under the least squares method as follows:

[0155]

[0156] where i is the reference index, 0 or 1, (a sb ,b sb ) is the optimal IC parameter, argmin(.) is the independent variable for the minimum, and the summation is performed on each neighboring sample x (here in the L-shape set L-shape-cur) paired with its corresponding neighboring sample y (here in the L-shape set L-shape-ref).

[0157] In step 1240, by using the IC parameter (a sb ,b sb ) applies the illumination compensation model on the motion compensated reference sub-block to derive the adjusted motion compensated reference sub-block, that is, each sample (x, y) of the motion compensated reference sub-block is adjusted according to Equation 3.

[0158] Fig.14 A flow chart of a method 1400 of encoding a sub-block of a block from a predicted sub-block is shown in accordance with at least one embodiment.

[0159] At step 1410, a prediction residual subblock for a subblock of a block is calculated as the difference between the subblock and the predicted subblock.

[0160] At step 1420, the prediction residual sub-block is transformed, quantized, and entropy encoded.

[0161] This method iterates over each sub-block of a block.

[0162] In one embodiment, the prediction residual sub-block is transformed using one single transform.

[0163] In a variation of this embodiment, the neighboring samples of the subblock and the predicted subblock are reconstructed samples, ie samples of the coded and decoded block. The IC parameters are then derived only from the reconstructed samples.

[0164] In one embodiment, neighboring samples of a subblock and the predicted subblock are formed only from internal samples. This allows independence of encoding / decoding of blocks and thus parallel processing.

[0165] For example, in Fig.13c In the case of , sub-block (a) may use only samples on the left, sub-block (b) may use only samples on the top, and sub-block (c) may use samples on both the left and top.

[0166] In one embodiment, the prediction residual block is formed by all prediction residual sub-blocks, and the prediction residual block is transformed, quantized, and entropy encoded using a single transform.

[0167] In a variation of this embodiment, the neighboring samples of a subblock and the predicted subblock are formed only from external samples (e.g. Fig.13a -d).

[0168] In another variant of this embodiment, neighboring samples of a subblock and the predicted subblock are formed only from samples of at least one further predicted subblock.

[0169] In an embodiment, neighboring samples of a subblock and the predicted subblock are formed from outer samples and samples of at least one further predicted subblock within the block.

[0170] For example, in Fig.13a In the case of , the samples on the left side of the current sub-block and the samples of the top adjusted predicted sub-block are used.

[0171] In another variation, when traversing the set of sub-blocks in scanning order, a first sub-block is identified as a first sub-block to be encoded. The IC parameters of this first sub-block are default IC parameters, or the first predicted sub-block is not illumination compensated.

[0172] In one embodiment, IC parameters are derived based on a subset of sub-blocks in order to reduce the amount of computation and thus reduce complexity.

[0173] For example, in HEVC, a Virtual Pipeline Data Unit (VPDU) defines a subset of sub-blocks inside a block. IC parameters are then derived based only on the samples inside the VPDU.

[0174] Fig.15 It shows the case where 4 VPDUs are created to group sub-blocks of a block.

[0175] In a variation of this embodiment, the first sub-block of the VPDU may not use IC parameters, or may use default IC parameters. In this way, there is no data dependency between VPDUs, and they can be processed in parallel.

[0176] This can also be applied in case of horizontal or vertical partitioning.

[0177] In one embodiment, the IC function may include equations 1 and 3 or other linear or nonlinear functions of IC parameters. For example, the IC function may only consider the slope parameter without considering the intercept parameter, i.e., IC(x)=a*x. In one embodiment, the IC parameters may be more than two parameters, depending on the function (e.g., depending on the degree of the polynomial function).

[0178] In one embodiment, equations 2, 4 may be based on absolute differences, rather than squared differences.

[0179] The IC parameters may include at least one parameter, for example, at least one amplitude scale or slope parameter, an amplitude offset or intercept parameter, a position scale parameter, a position offset parameter, a time scale parameter, a time offset parameter, or the like.

[0180] Fig.16 A flowchart 1600 of an exemplary method of video encoding according to this embodiment is shown.

[0181] The method 1600 includes obtaining a block in a picture of a video and encoding the block according to the method 800. The method 1600 iterates to encode all blocks of the picture.

[0182] Fig.17 A flowchart 1700 of an exemplary method of video decoding according to this embodiment is shown.

[0183] The method 1700 includes decoding information from a bitstream, such as transform coefficients, motion vectors, picture segmentation information, possible prediction mode information, other syntax elements, and / or other codec information.

[0184] In one embodiment, the IC prediction flag is decoded to determine whether IC is enabled for a block, picture, slice, or sequence.

[0185] In one embodiment, when the IC flag enables IC for a block (eg, the IC flag is set to true), IC parameters for the block may be included and optionally decoded from the bitstream.

[0186] Next, the method includes dequantizing (module 640) and inverse transforming (module 650) the transform coefficients to decode the prediction residual block. Next, the method includes subdividing the block and deriving the prediction block. Finally, the method includes combining the decoded prediction residual block with the prediction block to obtain a decoded / reconstructed block.

[0187] The prediction block may be obtained (module 605) from intra prediction (module 660) or motion compensated prediction (ie, inter prediction) (module 670), possibly depending on the prediction mode information. The prediction block is derived from steps 810-830 of method 800 when IC is enabled.

[0188] The IC parameters may include at least one amplitude scale or slope parameter, an amplitude offset or intercept parameter, a position scale parameter, a position offset parameter, a time scale parameter, a time offset parameter, etc. The block may be a reconstructed block.

[0189] According to an embodiment of the method, the function may be one of a sum of squared differences and a sum of absolute differences.

[0190] Fig.18 A block diagram is shown of an example of a system in which various aspects and embodiments are implemented.

[0191] System 1800 can be embodied as a device including various components described below, and is configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances and servers. The elements of system 1800 can be embodied in a single integrated circuit (IC), multiple ICs and / or discrete components, either alone or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1800 are distributed over multiple ICs and / or discrete components. In various embodiments, system 1800 is coupled to one or more other systems or other electronic devices via, for example, a communication bus 1940 or by dedicated input and / or output ports. In various embodiments, system 1800 is configured to implement one or more aspects described in this document.

[0192] The system 1800 includes at least one processor 1810 configured to execute instructions loaded therein for implementing, for example, various aspects described in this document. The processor 1810 may include embedded memory, input-output interfaces, and various other circuits known in the art. The system 1800 includes at least one memory 1820 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1800 includes a storage device 1840, which may include a non-volatile memory and / or a volatile memory, including but not limited to an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a magnetic disk drive, and / or an optical disk drive. As a non-limiting example, the storage device 1840 may include an internal storage device, an attached storage device (including a removable and non-removable storage device), and / or a network accessible storage device.

[0193] The system 1800 includes an encoder / decoder module 1830, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 1830 may include its own processor and memory. The encoder / decoder module 1830 represents a (multiple) module that can be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 1830 may be implemented as a separate element of the system 1800, or may be incorporated into the processor 1810 as a combination of hardware and software known to those skilled in the art. Program code to be loaded onto the processor 1810 or the encoder / decoder 1830 to perform various aspects described in this document may be stored in the storage device 1840 and then loaded onto the memory 1820 for execution by the processor 1810. According to various embodiments, one or more of the processor 1810, the memory 1820, the storage device 1840, and the encoder / decoder module 1830 may store one or more of the various items during the execution of the process described in this document. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing of equations, formulas, operations, and operational logic.

[0194] In some embodiments, memory internal to the processor 1810 and / or encoder / decoder module 1830 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1810 or the encoder / decoder module 1830) is used for one or more of these functions. The external memory may be memory 1820 and / or storage device 1840, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, an operating system for the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video codec and decoding operations, such as working memory for MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Codec, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Codec, a new standard developed by JVET (Joint Video Experts Group)).

[0195] Input to the elements of system 1800 may be provided through various input devices, as shown in block 1930. Such input devices include, but are not limited to: (i) an RF section that receives a radio frequency (RF) signal, such as sent over the air by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Fig.18 Other examples not shown include composite video.

[0196] In various embodiments, the input device of block 1930 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for the following functions: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select a signal frequency band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements to perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal sent by wired (for example, cable) medium, and filter to the frequency band of expectation again by filtering, down conversion and perform frequency selection.Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements of similar or different functions.Adding element can include inserting element between existing element, such as, for example inserting amplifier and analog-to-digital converter.In various embodiments, the RF part comprises antenna.

[0197] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1800 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, can be implemented as needed, for example, in a separate input processing IC or processor 1810. Similarly, various aspects of USB or HDMI interface processing can be implemented as needed in a separate interface IC or in the processor 1810. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements including, for example, a processor 1810 and an encoder / decoder 1830 operating in combination with a memory and storage element to process the data stream as needed for presentation on an output device.

[0198] The various elements of system 1800 may be disposed within an integrated housing in which the various elements may be interconnected and transmit data therebetween using a suitable connection arrangement, such as an internal bus 1940 (including an inter-IC (I2C) bus), wiring, and printed circuit boards as are known in the art.

[0199] The system 1800 includes a communication interface 1850 capable of communicating with other devices via a communication channel 1060. The communication interface 1850 may include, but is not limited to, a transceiver configured to send and receive data through the communication channel 1860. The communication interface 1850 may include, but is not limited to, a modem or a network card, and the communication channel 1860 may be implemented, for example, within a wired and / or wireless medium.

[0200] In various embodiments, data is streamed or otherwise provided to the system 1800 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signals of these embodiments are received via a communication channel 1860 and a communication interface 1850 suitable for Wi-Fi communications. The communication channel 1860 of these embodiments is typically connected to an access point or router that provides access to an external network including the Internet to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that transmits data via an HDMI connection of an input block 1830 to provide streaming data to the system 1800. Still other embodiments use an RF connection of an input block 1830 to provide streaming data to the system 1800. As described above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use a wireless network other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0201] The system 1800 can provide output signals to various output devices, including a display 1900, a speaker 1910, and other peripheral devices 1920. The display 1900 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1900 can be used for a television, a tablet computer, a laptop computer, a cellular phone (mobile phone), or other devices. The display 1900 can also be integrated with other components (e.g., as in a smart phone), or separately (e.g., an external monitor for a laptop computer). In various examples of embodiments, other peripheral devices 1920 include one or more of a stand-alone digital video disk (or digital versatile disk) (for these two items, it can be referred to as a DVR), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1920 that provide functions based on the output of the system 1800. For example, a disk player performs the function of playing the output of the system 1800.

[0202] In various embodiments, control signals are communicated between the system 1800 and the display 1900, speaker 1910, or other peripheral device 1920 using signaling such as AV.link, consumer electronics control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices may be communicatively coupled to the system 1800 via dedicated connections through respective interfaces 1870, 1880, and 1890. Alternatively, output devices may be connected to the system 1800 via communication interface 1850 using communication channel 1860. The display 1900 and speaker 1910 may be integrated into a single unit in an electronic device (such as, for example, a television) along with other components of the system 1800. In various embodiments, the display interface 1870 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0203] For example, if the RF portion of input 1930 is part of a separate set-top box, the display 1900 and speaker 1910 may alternatively be separate from one or more of the other components. In various embodiments where the display 1900 and speaker 1910 are external components, the output signals may be provided via dedicated output connections, including, for example, an HDMI port, a USB port, or a COMP output.

[0204] The implementations described herein may be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if discussed only in the context of a single implementation form (e.g., discussed only as a method), the implementation of the features discussed may also be implemented in other forms (e.g., a device or program). The device may be implemented with, for example, appropriate hardware, software, and firmware. The method may be implemented in, for example, a device such as, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as, for example, a computer, a cellular phone, a portable / personal digital assistant (PDA), and other devices that facilitate information communication between end users.

[0205] According to one aspect of this embodiment, a device 1800 for video encoding is provided, which includes a processor 1810 and at least one memory 1820, 1840 coupled to the processor, and the processor 1810 is configured to execute any embodiment of the above-mentioned method 800, 1600 and / or 1700.

[0206] According to an aspect of the present disclosure, there is provided an apparatus for video encoding, comprising a component for deriving one or more illumination compensation parameters based on neighboring samples of a subblock of a block and neighboring samples of a reference subblock for motion compensation; a component for deriving a predicted subblock by applying illumination compensation on a reference subblock for motion compensation using the derived one or more illumination compensation parameters; and a component for encoding the subblock using the predicted subblock. Figure 4 , 5 The video encoder of may include structures or components of the apparatus, in particular blocks 470, 475, 571-575. The apparatus for video encoding may perform any embodiment of any of the methods 800 and 1600.

[0207] According to an aspect of the present disclosure, there is provided an apparatus for video decoding, comprising a component for deriving one or more illumination compensation parameters based on neighboring samples of a subblock of a block and neighboring samples of a reference subblock for motion compensation; a component for deriving a predicted subblock by applying illumination compensation on a reference subblock for motion compensation using the derived one or more illumination compensation parameters; and a component for encoding the subblock using the predicted subblock. Figure 6 , 7 The video decoder of may include structures or components of the apparatus, in particular blocks 670, 571-574. The apparatus for video decoding may perform any embodiment of any of the methods 800 and 1700.

[0208] It will be apparent to those skilled in the art that implementations may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be sent via a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0209] In addition, any of the methods 800, 1600, and / or 1700 may be implemented as a computer program product (independently or jointly) comprising computer executable instructions that may be executed by a processor. The computer program product having computer executable instructions may be stored in a corresponding transient or non-transitory computer readable storage medium of the system 1800, encoder 400 (or 500), and / or decoder 600 (or 700).

[0210] It is important to note that in some embodiments, one or more elements of processes 800, 1600, and / or 1700 may be combined, performed in a different order, or eliminated while still implementing aspects of the present disclosure. Other steps may be performed in parallel, where the processor does not wait for one step to be fully completed before starting another step.

[0211] In addition, aspects of the present embodiment may take the form of a computer-readable storage medium. Any combination of one or more computer-readable storage media may be utilized. A computer-readable storage medium may take the form of a computer-readable program product embodied in one or more computer-readable media and having a computer-readable program code embodied thereon that can be executed by a computer. The computer-readable storage medium used herein is considered to be a non-transitory storage medium having an inherent ability to store information therein and an inherent ability to provide information retrieval therefrom. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or apparatus, or any suitable combination of the foregoing.

[0212] It should be understood that the following list, while providing more specific examples of computer-readable storage media to which the present disclosure may be applied, is merely an illustrative and non-exhaustive list, as will be readily appreciated by one of ordinary skill in the art. The list of examples includes a portable computer disk, a hard disk, a ROM, an EPROM, a flash memory, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0213] According to one aspect of the present disclosure, a computer-readable storage medium carrying a software program is provided, the software program including program code instructions for executing any embodiment of any method of the present disclosure (including method 800, 1600 and / or 1700).

[0214] It should be understood that references to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" of the present disclosure and other variations thereof mean that the specific features, structures, characteristics, etc. described in conjunction with the embodiment are included in at least one embodiment of the present disclosure. Therefore, the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations that appear in various places throughout the specification do not necessarily refer to the same embodiment.

[0215] Additionally, the disclosure or claims thereof may refer to "determining" various information. Determining, deriving information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0216] Furthermore, this application or its claims may refer to "providing" various information. Providing information may include, for example, one or more of outputting information, storing information, sending information, transmitting information, displaying information, showing information, or moving information.

[0217] Furthermore, this application or its claims may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, processing information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0218] Furthermore, this application or its claims may refer to "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" generally involves, in one way or another, during an operation such as, for example, storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0219] It should be understood that the various features shown and described are interchangeable. Unless otherwise specified, the features shown in one embodiment can be combined in another embodiment. In addition, the features described in the various embodiments can be combined or separated, unless otherwise specified as inseparable or non-combinable.

[0220] As previously mentioned, the functions of the various elements shown in the drawings can be provided by using dedicated hardware as well as hardware capable of executing software associated with appropriate software. In addition, when provided by a processor, the functions can be provided by a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which can be shared.

[0221] It should also be understood that because some of the constituent system components and methods depicted in the accompanying drawings are preferably implemented in software, the actual connections between the system components or process function blocks may vary depending on the manner in which the processes of the present disclosure are programmed. Given the teachings herein, one of ordinary skill in the relevant art will be able to conceive of these and similar implementations or configurations of the present disclosure.

[0222] Although illustrative embodiments have been described herein with reference to the accompanying drawings, it should be understood that the present disclosure is not limited to those precise embodiments, and that persons of ordinary skill in the relevant art may implement various changes and modifications therein without departing from the scope of the present disclosure. In addition, the various embodiments may be combined without departing from the scope of the present disclosure. All such changes and modifications are intended to be included within the scope of the present disclosure as set forth in the appended claims.

[0223] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to encompass selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to many of the listed items, as will be apparent to one of ordinary skill in this and related arts.

[0224] It will be apparent to those skilled in the art that implementations may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing the method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be sent via a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

Claims

1. A method for encoding a current block of a video, comprising: Splitting the current block into a plurality of sub-blocks; Applying a process for deriving a set of illumination compensation parameters to the plurality of sub-blocks, the process for deriving a set of illumination compensation parameters for a sub-block being based on samples of the current block neighboring the sub-block and samples neighboring a reference sub-block for motion compensation of the sub-block; For each of the plurality of subblocks, deriving a predicted subblock, wherein, for a subblock for which the illumination compensation parameter set is derived, the predicted subblock is obtained by applying illumination compensation to a reference subblock for motion compensation by using the illumination compensation parameter set derived for the subblock; Based on the predicted sub-block obtained for the one sub-block, a residual sub-block for each sub-block is obtained, forming a single residual block by combining the residual sub-blocks, and A single transform is applied to the single residual block to encode the current block.

2. The method according to claim 1, in, For the first sub-block of the current block, no illumination compensation parameter set is derived.

3. The method according to claim 2, wherein: Deriving a predicted subblock for the first subblock of the plurality of subblocks includes applying illumination compensation to a motion compensated reference subblock of the first subblock using a default illumination compensation parameter set, or applying illumination compensation to a motion compensated reference subblock of the first subblock without applying illumination compensation.

4. A method for decoding a block of a video, comprising: Split the current block into multiple sub-blocks; Applying a process for deriving a set of illumination compensation parameters to the plurality of sub-blocks, the process for deriving a set of illumination compensation parameters for a sub-block being based on samples of the current block neighboring the sub-block and samples neighboring a reference sub-block for motion compensation of the sub-block; For each of the plurality of subblocks, deriving a predicted subblock, wherein, for a subblock for which the illumination compensation parameter set is derived, the predicted subblock is obtained by applying illumination compensation to a reference subblock for motion compensation by using the illumination compensation parameter set derived for the subblock; Get the residual sub-block for each sub-block, forming a single residual block by combining the residual sub-blocks, and A single inverse transform is applied to the single residual block, and the single residual block is combined with the predicted sub-block to reconstruct the current block.

5. The method according to claim 4, wherein: For the first sub-block of the current block, no illumination compensation parameter set is derived.

6. The method according to claim 5, wherein: Deriving a predicted subblock for the first subblock of the plurality of subblocks includes applying illumination compensation to a motion compensated reference subblock of the first subblock using a default illumination compensation parameter set, or applying illumination compensation to a motion compensated reference subblock of the first subblock without applying illumination compensation.

7. An apparatus for encoding a current block of a video, comprising an electronic circuit configured to: Splitting the current block into a plurality of sub-blocks; Applying a process for deriving a set of illumination compensation parameters to the plurality of sub-blocks, the process for deriving a set of illumination compensation parameters for a sub-block being based on samples of the current block neighboring the sub-block and samples neighboring a reference sub-block for motion compensation of the sub-block; For each sub-block in the plurality of sub-blocks, a predicted sub-block is derived, wherein: For a sub-block for which the illumination compensation parameter set is derived, the prediction sub-block is obtained by applying illumination compensation to the reference sub-block for motion compensation by using the illumination compensation parameter set derived for the sub-block; Based on the predicted sub-block obtained for the one sub-block, a residual sub-block for each sub-block is obtained, forming a single residual block by combining the residual sub-blocks, and A single transform is applied to the single residual block to encode the current block.

8. The device according to claim 7, wherein: For the first sub-block of the current block, no illumination compensation parameter set is derived.

9. The device according to claim 8, wherein: Deriving a predicted subblock for the first subblock of the plurality of subblocks includes applying illumination compensation to a motion compensated reference subblock of the first subblock using a default illumination compensation parameter set, or applying illumination compensation to a motion compensated reference subblock of the first subblock without applying illumination compensation.

10. An apparatus for decoding a block of a video, comprising electronic circuitry configured to: Split the current block into multiple sub-blocks; Applying a process for deriving a set of illumination compensation parameters to the plurality of sub-blocks, the process for deriving a set of illumination compensation parameters for a sub-block being based on samples of the current block neighboring the sub-block and samples neighboring a reference sub-block for motion compensation of the sub-block; For each sub-block in the plurality of sub-blocks, a predicted sub-block is derived, wherein: For a sub-block for which the illumination compensation parameter set is derived, the prediction sub-block is obtained by applying illumination compensation to the reference sub-block for motion compensation by using the illumination compensation parameter set derived for the sub-block; Get the residual sub-block for each sub-block, forming a single residual block by combining the residual sub-blocks, and A single inverse transform is applied to the single residual block, and the single residual block is combined with the predicted sub-block to reconstruct the current block.

11. The device according to claim 10, wherein: For the first sub-block of the current block, no illumination compensation parameter set is derived.

12. The device according to claim 11, wherein Deriving a predicted subblock for the first subblock of the plurality of subblocks includes applying illumination compensation to a motion compensated reference subblock of the first subblock using a default illumination compensation parameter set, or applying illumination compensation to a motion compensated reference subblock of the first subblock without applying illumination compensation.

13. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by one or more processors, implement the method of claim 1.

14. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by one or more processors, implement the method of claim 4.

15. A computer program product comprising instructions for performing the method of claim 1 when executed by one or more processors.

16. A computer program product comprising instructions for performing the method of claim 4 when executed by one or more processors.

Images