Method and apparatus for encoding and decoding blocks based on illumination compensation

Abstract

To provide a method, an apparatus, a computer-readable storage medium, and a computer program for encoding and decoding video on the basis of illumination compensation.SOLUTION: A method (800) includes deriving (830) one or more illumination compensation parameters on the basis of adjacent samples of a subblock of a block and adjacent samples of a motion-compensated reference subblock, and deriving a prediction subblock by applying illumination compensation to the motion-compensated reference subblock using the one or more derived illumination compensation parameters.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] Technical Field FIELD OF THE DISCLOSURE Embodiments of the present invention generally relate to video encoding and decoding using illumination compensation. [Background technology]

[0002] background This section is intended to introduce the reader to various aspects of technology that may be related to various aspects of at least one embodiment of the present invention, which are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of at least one embodiment, and thus, it should be understood that these descriptions are to be read in this light.

[0003] To achieve high compression efficiency, video coding schemes typically exploit spatial and temporal redundancy in video content using prediction and transformation. In general, intra- or inter-prediction is used to exploit correlation within or between frames. The difference between the original video picture and the predicted picture, often denoted as prediction error or prediction residual, is then transformed, quantized, and entropy coded. To reconstruct the picture, the compressed data is decoded by the inverse process corresponding to prediction, transformation, quantization, and entropy coding. Summary of the Invention

[0004] overview This section presents a simplified summary of at least one embodiment of the invention in order to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the embodiments. It is not intended to identify key or critical elements of the embodiments. The following summary merely presents some aspects of at least one embodiment of the invention in a simplified form as a prelude to the more detailed description presented elsewhere in this specification.

[0005] According to at least one general aspect of an embodiment of the present invention, there is provided a method for encoding a block of a video, the method including deriving one or more illumination compensation parameters based on neighboring samples of a sub-block of the block and neighboring samples of a motion-compensated reference sub-block, deriving a predictive sub-block by applying illumination compensation to the motion-compensated reference sub-block using the one or more derived illumination compensation parameters, and encoding the sub-block using the predictive sub-block.

[0006] According to at least one other general aspect of an embodiment of the present invention, there is provided a method for decoding a block of a video, the method including deriving one or more illumination compensation parameters based on neighboring samples of a sub-block of the block and neighboring samples of a motion compensated reference sub-block, deriving a predictive sub-block by applying illumination compensation to the motion compensated reference sub-block using the one or more derived illumination compensation parameters, and decoding the sub-block using the predictive sub-block.

[0007] According to at least one other general aspect of an embodiment of the present invention, there is provided an apparatus for encoding a block of a video, the apparatus including: means for deriving one or more illumination compensation parameters based on neighboring samples of a sub-block of the block and neighboring samples of a motion compensated reference sub-block; means for deriving a predictive sub-block by applying illumination compensation to the motion compensated reference sub-block using the one or more derived illumination compensation parameters; and means for encoding the sub-block using the predictive sub-block.

[0008] According to at least one other general aspect of an embodiment of the present invention, an apparatus for decoding a block of a video is provided, the apparatus including: means for deriving one or more illumination compensation parameters based on neighboring samples of a sub-block of the block and neighboring samples of a motion compensated reference sub-block; means for deriving a predictive sub-block by applying illumination compensation to the motion compensated reference sub-block using the one or more derived illumination compensation parameters; and means for decoding a sub-block using the predictive sub-block.

[0009] According to one embodiment, multiple sub-blocks of the same block are coded using a predictive sub-block, and coding or decoding a sub-block involves applying a separate transform to each residual derived from the sub-block and the associated predictive sub-block.

[0010] According to one embodiment, the neighboring samples of the current sub-block are the samples within the block.

[0011] According to one embodiment, encoding or decoding a sub-block comprises applying a single transform to a prediction residual derived from the sub-block and a prediction residual derived from the predictive sub-block.

[0012] According to one embodiment, one or more neighboring samples of the current sub-block are samples outside the block.

[0013] According to one embodiment, the neighbouring samples of a subblock and a predictive subblock are formed exclusively using samples of at least one further predictive subblock.

[0014] According to one embodiment, the neighbouring samples of a sub-block and a predictive sub-block are formed using samples outside the block and samples of at least one further predictive sub-block within the block.

[0015] According to one embodiment, one or more illumination compensation parameters are derived from neighboring samples within a subset of sub-blocks of a block.

[0016] According to at least one other general aspect of embodiments of the invention, a non-transitory computer-readable storage medium and a computer program product are provided.

[0017] At least one particular property of an embodiment of the present invention, as well as at least one other of the above objects, advantages, features, and uses of an embodiment of the present invention, will become apparent from the following description of examples taken in conjunction with the accompanying drawings.

[0018] Some example embodiments are illustrated in the drawings. [Brief description of the drawings]

[0019] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] 1 illustrates an L-shaped set of reconstructed neighboring samples for a current block in a current picture, for a Reference 0 block in a Reference 0 picture, and for a Reference 1 block in a Reference 1 picture according to the prior art. [Diagram 2] 1 illustrates some disadvantages of prior art illumination compensation techniques. [Diagram 3] 1 illustrates some disadvantages of prior art illumination compensation techniques. [Figure 4] 1 shows a simplified block diagram of an exemplary encoder according to one embodiment. [Diagram 5] 5 shows a simplified block diagram 500 of some modules of an exemplary encoder according to at least one embodiment. [Figure 6] 6 shows a simplified block diagram of an example decoder 600 according to at least one embodiment. [Figure 7] 7 shows a simplified block diagram 700 of some modules of an exemplary decoder in accordance with at least one embodiment. [Figure 8]8 illustrates a flow diagram 800 of a method for encoding / decoding a block based on illumination compensation according to at least one embodiment. [Figure 9] 1 illustrates an example of subdividing a block, according to at least one embodiment. [Figure 10] 10 illustrates a flow diagram of a method 1000 for deriving a motion compensated reference sub-block for a block according to at least one embodiment. [Figure 11] 4 illustrates an example of motion vectors for sub-blocks based on an affine motion model, in accordance with at least one embodiment. [Figure 12] 12 shows a flow diagram of a method 1200 for deriving one or more IC parameters and adjusting a motion compensated reference sub-block according to an embodiment of the present invention. [Figure 13a] 1 illustrates the definition of L-shaped sets according to different subdivisions of a block, in accordance with at least one embodiment. [Figure 13b] 1 illustrates the definition of L-shaped sets according to different subdivisions of a block, in accordance with at least one embodiment. [Figure 13c] 1 illustrates the definition of L-shaped sets according to different subdivisions of a block, in accordance with at least one embodiment. [Figure 13d] 1 illustrates the definition of L-shaped sets according to different subdivisions of a block, in accordance with at least one embodiment. [Figure 14] 14 illustrates a flow diagram of a method 1400 for encoding a sub-block of a block from a predictive sub-block, according to at least one embodiment. [Figure 15] An example is shown in which four VPDUs are created to group sub-blocks of a block. [Figure 16] 1 shows a flow diagram of an exemplary method for encoding video, in accordance with an embodiment of the present invention. [Figure 17] 4 illustrates a flow diagram of an exemplary method for decoding video, in accordance with an embodiment of the present invention. [Figure 18] FIG. 1 illustrates a block diagram of a computing environment in which aspects of the present disclosure can be implemented and executed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Detailed Description This detailed description illustrates the principles of embodiments of the present invention, and it will thus be understood that those skilled in the art will be able to devise various arrangements which embody the principles of embodiments of the present invention and are within the scope of such embodiments, even if not explicitly described or shown herein.

[0021] All examples and conditional language provided herein are for instructional purposes to aid the reader in understanding the principles and concepts of the embodiments of the present invention brought by the inventors to advance the art, and should not be construed as being limited to such specifically provided examples and conditions.

[0022] Moreover, all statements herein reciting principles, aspects, and embodiments of the present disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. In addition, such equivalents are intended to include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

[0023] Thus, for example, those skilled in the art will appreciate that the block diagrams presented herein represent conceptual views of illustrative circuitry embodying the principles of embodiments of the invention. Similarly, it will be appreciated that any flow charts, process diagrams, state transition diagrams, pseudocode, or the like, may be substantially represented in a computer-readable medium and thus represent various processes performed by a computer or processor, whether or not such a computer or processor is explicitly depicted.

[0024] The embodiments of the present invention will be described more fully below with reference to the accompanying drawings, which show examples of the aforementioned embodiments of the present invention. However, the embodiments may be embodied in many alternative forms and should not be construed as being limited to the examples set forth herein. It should therefore be understood that there is no intention to limit the embodiments to the particular forms disclosed. On the contrary, the embodiments of the present invention are intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present application.

[0025] Where a drawing is shown as a flow diagram, it should be understood that the drawing also provides a block diagram of the corresponding apparatus. Similarly, where a drawing is shown as a block diagram, it should be understood that the drawing also provides a flow diagram of the corresponding method / process.

[0026] The functions of the various elements shown may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors, some of which may be shared. Furthermore, explicit use of the terms "processor" or "controller" should not be construed as referring exclusively to hardware capable of executing software, but may implicitly include, without limitation, digital signal processor (DSP) hardware, read only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage.

[0027] Other conventional and / or custom hardware may also be included. Similarly, any switches shown are conceptual only. Their functions may be performed by operation of program logic, by dedicated logic, by an interaction of program control and dedicated logic, or even manually, with the particular technique being selectable by the implementer as more clearly understood from the context.

[0028] Similar or identical elements in the drawings are referred to using the same reference numbers.

[0029] Some figures may represent syntax tables widely used within the aforementioned video compression standard specifications to define the structure of a bitstream that complies with the video compression standard. Within those syntax tables, the term "..." indicates an invariant portion of the syntax with respect to the well-known definition given within the video compression standard specifications and has been removed in the figures for ease of reading. A bolded term within a syntax table indicates that the value of that term is obtained by parsing the bitstream. The right-hand row of the syntax table indicates the number of bits to code the data of the syntax element. For example, u(4) indicates the use of 4 bits to code the data, u(8) indicates 8 bits, and ae(v) indicates a context-adaptive arithmetic entropy coded syntax element.

[0030] In the claims herein, any element expressed as a means for performing a specified function is intended to encompass any way of performing that function, including, for example, a) a combination of circuit elements that perform that function, or b) software in any form combined with appropriate circuitry for executing software to perform that function, thus including firmware, microcode, etc. The embodiment of the invention defined by such claims resides in the fact that the functionality provided by the various recited means are combined and brought together in the manner required by the claims. Thus, any means capable of providing those functionality are regarded as equivalent to the means shown herein.

[0031] It should be understood that the drawings and description have been simplified to illustrate elements relevant to a clear understanding of the embodiments of the present invention, while many other elements found in a typical encoding and / or decoding device have been omitted for clarity.

[0032] It will be understood that although the terms first and second may be used herein to describe various elements, the elements should not be limited by these terms. These terms are merely used to distinguish one element from another. Various methods are described above, each of which includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for the method to operate properly, the order and / or use of specific steps and / or actions can be modified or combined.

[0033] In the following sections, the terms "reconstruct" and "decode" may be used interchangeably. Usually, but not always, "reconstruct" is used on the encoder side, while "decode" is used on the decoder side. Additionally, the terms "encode" and "encode" may be used interchangeably. Additionally, the terms "image", "picture", and "frame" may be used interchangeably. Additionally, the terms "encode", "source coding", and "compression" may be used interchangeably.

[0034] It should be understood that a picture (also denoted as image or frame) can be an array of luma samples in monochrome format, or an array of luma samples and two corresponding arrays of chroma samples in 4:2:0, 4:2:2, or 4:4:4 color format, or three arrays of three color components (e.g., RGB).

[0035] In video compression standards, a picture is sometimes divided into blocks of different sizes and / or shapes. It should be understood that a block is a two-dimensional array or matrix. The horizontal or x direction (or axis) represents the width and the vertical or y direction (or axis) represents the height. Indexing starts at 0. The x direction represents the columns and the y direction represents the rows. The maximum x index is width - 1. The maximum y index is height - 1.

[0036] An embodiment of the present invention is directed to block-based illumination compensation (IC) used within inter-prediction modes to adjust block prediction samples obtained by motion compensation (MC) by taking into account any local illumination variations, either spatially or temporally.

[0037] In the prior art, IC parameters are estimated by comparing an L-shaped set of reconstructed neighboring samples for a current block in a current picture with a corresponding L-shaped set of reconstructed neighboring samples for a reference block in a reference picture. In the case of bi-prediction (i.e., there are two reference blocks per current block), the corresponding L-shaped sets of reconstructed neighboring samples for two reference i blocks (i=0 or 1) are compared separately with the L-shaped set of reconstructed neighboring samples for the current block.

[0038] 1 illustrates an L-shaped set of neighboring samples 170 for a current block 160 in a current picture 150, a corresponding L-shaped set (L-shape-ref-0) 130 for a motion compensated reference 0 block (ref-0, also referred to herein as MC-0 or motion compensated 0 block) 120 in a reference 0 picture 110, and a corresponding L-shaped set (L-shape-ref-1) 135 for a motion compensated reference 1 block (ref-1, also referred to herein as MC-1 or motion compensated 1 block) 125 in a reference 1 picture 115, according to the prior art. Thus, hereafter, the term reference i block or ref-i (i=0,1) refers to a motion compensated reference block, i.e., a block at position (x+MV ix ,y+MV iy ), where (x,y) is the position of the current block in the current picture, and MV stands for motion vector. In FIG. 1, the motion vector between current block 160 and Reference 0 block 120 is regarded as MV0 140, and the motion vector between current block 160 and Reference 1 block 125 is regarded as MV1 145.

[0039] Block-based illumination compensation (IC) can use an illumination compensation (IC) model that is typically linear and defined by the following equation: IC(x)=a*x+b (1) where a and b are IC parameters commonly called the slope and intercept of a linear function, respectively.

[0040] Specifically, the IC parameters can be estimated by, for example, minimizing the difference between samples in the L-shaped set 170 for the current block and samples in the L-shaped set (130 or 135) for the reference block (120 or 125) that are adjusted (corrected) with illumination compensation using the IC model. The difference between the samples can be minimized under the least squares method as follows:

number

[0041] According to the prior art, the concept of using an L-shaped set of neighboring samples relies on the assumption that IC parameters optimized for the neighboring samples of a current block remain suitable for the current block. This assumption is generally true because the neighboring samples of the L-shaped set are the closest samples available to the samples of the current block. However, this approach creates a pipeline dependency for inter-prediction, because it is necessary to wait for the neighboring blocks to be reconstructed before constructing the current prediction block, as shown in Figure 2, where the decoding of the current block CU depends on the neighboring reconstructed samples of the neighboring blocks CU1 and CU2. Furthermore, as shown in Figure 3, the IC parameters are estimated using samples that may be relatively far from the sample to be adjusted.

[0042] The embodiments of the present invention address some of the disadvantages found in the prior art. Specifically, as further described in the following paragraphs and figures, the embodiments of the present invention derive one or more illumination compensation parameters based on adjacent samples of a sub-block of a current block and adjacent samples of a motion-compensated reference sub-block, derive a prediction sub-block by applying illumination compensation to the motion-compensated reference sub-block using the one or more derived illumination compensation parameters, and use the prediction sub-block to encode the sub-block,

[0043] encoding FIG. 4 illustrates a simplified block diagram of an example encoder 500 according to at least one embodiment.

[0044] The encoder 400 may be included in a transmitter or headend in a communication system.

[0045] To code a video sequence having one or more pictures, a picture is possibly divided into blocks of different sizes and / or shapes (module 410). For example, in HEVC ("ITU-T H.265 TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (10 / 2014), SERIES H: AUDIOVISUAL AND MULTIMEDIA SYSTEMS, Infrastructure of audiovisual services - Coding of moving video, High efficiency video coding, Recommendation ITU-T H.265"), a picture can be divided into square CTUs (coding tree units) with configurable size. A set of consecutive CTUs can be grouped into a slice. A CTU is the root of a quadtree division into blocks, usually denoted CU (coding unit).

[0046] In the exemplary encoder 500, pictures are encoded by a block-based video encoder as described below.

[0047] Each block is coded using an intra-prediction mode or an inter-prediction mode.

[0048] When a block is coded by an intra prediction mode (module 460), the encoder 400 performs intra prediction (also denoted spatial prediction) based on at least one block in the same picture. As an example, the predicted block is obtained by intra predicting the block from reconstructed neighboring samples.

[0049] When encoding a block in an inter prediction mode, the encoder 400 performs inter prediction (also denoted as temporal prediction) based on at least one reference block of at least one reference picture.

[0050] Inter-predictive coding is performed by performing motion estimation (module 475 ) and motion compensation (in module 470 ) of reference blocks stored in a reference picture buffer 480 .

[0051] In unity inter prediction mode, prediction blocks may generally (but not necessarily) be based on previous reference pictures.

[0052] In bi-inter prediction mode, the prediction block may generally (but not necessarily) be based on previous and subsequent pictures.

[0053] The encoder decides whether to use an intra- or inter-prediction mode to code the block (module 405) and indicates the intra / inter decision via a prediction mode syntax element.

[0054] A prediction residual block is calculated by subtracting a prediction block (also known as a predictor) from the block (module 420).

[0055] The prediction residual block is transformed (module 425) and quantized (module 430). The transformation module 425 can transform the block from the pixel (spatial) domain to the transform (frequency) domain. The transformation can be, for example, a cosine transform, a sine transform, a wavelet transform, etc. The quantization can be performed (module 430), for example according to a rate-distortion criterion.

[0056] The quantized transform coefficients as well as the motion vectors and other syntax elements are entropy coded (module 445) to output a bitstream. The entropy coding can be, for example, Context-Adaptive Binary Arithmetic Coding (CABAC), Context-Adaptive Variable Length Coding (CAVLC), Huffman, Arithmetic, Exponential-Golomb, etc.

[0057] The encoder can also skip the transform and directly apply quantization to the non-transformed predicted residual block. The encoder can also bypass both the transform and quantization, i.e., directly code the predicted residual block without applying the transform or quantization processes.

[0058] In direct PCM coding, no prediction is applied and block samples are coded directly into the bitstream.

[0059] The encoder 400 includes a decoding loop, thus decoding the coding block and providing a reference for further prediction. The quantized transform coefficients are dequantized (module 440) and inverse transformed (module 450) to decode the prediction residual block. A block is reconstructed by combining the decoded prediction residual block with the prediction block (module 455). An in-loop filter (465) can be applied to the reconstructed picture, for example to perform deblocking / sample adaptive offset (SAO) filtering to reduce coding artifacts. The filtered picture is stored in a reference picture buffer 480.

[0060] The modules of the encoder 400 may be implemented in software and executed by a processor, or may be implemented using circuitry well known to those skilled in the art of compression. In particular, the video encoder 400 may be implemented as an integrated circuit (IC).

[0061] Except for the differences described in the embodiments of the present invention, in particular the motion compensation 470 and / or motion estimation 475 modules based on illumination compensation, which are described in more detail in the following paragraphs and figures, the modules of the encoder 400 are found in a traditional block-based video encoder (e.g. an HEVC encoder).

[0062] With respect to its functionality other than illumination compensation (eg, modules other than 470 and 475), encoder 400 can be similar to any conventional block-based video encoder, and its functionality will not be described in detail here.

[0063] The motion estimation module 475 may include motion compensation since the purpose of the motion estimation module 475 is to determine the best motion vector, which may typically use an iterative search that terminates when the rate-distortion cost (RD cost) is low enough or reaches a minimum. As a result, IC may also be applied within the motion estimation module 475.

[0064] FIG. 5 illustrates a simplified block diagram 500 of some modules of an exemplary encoder in accordance with at least one embodiment.

[0065] The motion estimation 475 and motion compensation 470 modules specify IC-related functions within a module for an encoder according to at least one embodiment.

[0066] The motion estimation 475 and motion compensation 470 modules may include a subdivision module 571 for dividing blocks into sub-blocks according to an embodiment of the present invention.

[0067] As further described in the following paragraphs and figures, subdividing blocks spatially adapts the IC parameters to local features and reduces the maximum distance of the samples used to estimate the IC parameters to any of the adjusted samples compared to conventional methods.

[0068] The motion estimation 475 and motion compensation 470 modules may also include an intra motion compensation module 572 that performs motion compensated reference sub-blocks in accordance with an embodiment of the present invention.

[0069] The motion estimation 475 and motion compensation 470 modules may also include an IC parameter derivation module 573 that derives IC parameters for motion compensated reference sub-blocks in accordance with an embodiment of the present invention.

[0070] Finally, the motion estimation 475 and motion compensation 470 modules may include an IC application module 574 that derives a prediction sub-block from a motion compensated reference sub-block and derived IC parameters according to an embodiment of the present invention.

[0071] The motion estimation module 475 may include motion compensation since the objective of the motion estimation module 475 is to determine the best motion vector, which may typically use an iterative search that terminates when the rate-distortion cost (RD cost) is low enough or reaches a minimum. The iteration tests various motion vector candidates. The motion estimation 475 module may include an RD cost calculation module 575 that determines the RD cost to provide the motion compensation module 570 with the best motion vector MV, and identifies whether an appropriate or minimum value of RD has been achieved.

[0072] In one embodiment, the motion estimation module 575 can also output the IC parameter associated with the best motion vector MV, in which case there is no need to recalculate the IC parameter within module 570 and module 573 can be skipped from the motion compensation module 570.

[0073] The module of FIG. 5 based on illumination compensation is explained in more detail in the following paragraphs and figures.

[0074] In one embodiment, IC can be adaptively enabled or disabled for each inter-predictive coded block. In the bitstream, if IC is enabled for a block, picture, slide, or sequence, an IC flag can be coded for each block to indicate whether IC is enabled for the block. The IC flag can then be obtained at the decoder. In another embodiment, the IC flag can be inferred from other coding parameters (e.g., merge index, block size).

[0075] In one embodiment, if the derived IC parameters do not result in improved prediction, IC may be locally or globally deactivated (eg, the IC flag is set to false).

[0076] In one embodiment, if the IC flag enables IC for a block (e.g., the IC flag is set to true), the IC parameters for the block may be included and optionally encoded in the bitstream to be obtained at the decoder.

[0077] Decryption FIG. 6 illustrates a simplified block diagram of an example decoder 600 according to at least one embodiment.

[0078] The decoder 600 may be included in a receiver in a communication system.

[0079] The decoder 600 generally performs a decoding path that is the inverse of the encoding path performed by the encoder 400 described in Figures 4 and 5, although not all operations in the decoder are the inverse of the encoding process (e.g., intra-prediction and inter-prediction).

[0080] In particular, the input of the decoder 600 includes a bitstream that may be generated by the encoder 400 .

[0081] The bitstream is first entropy decoded (module 630) to obtain transform coefficients, motion vectors MV, picture partition information, possibly prediction mode information, other syntax elements and / or coded information.

[0082] For example, in HEVC, the picture partition information indicates the size of the CTU and how to partition the CTU into CUs, so that the decoder can divide the picture into CTUs and each CTU into CUs according to the picture partition information (635).

[0083] To decode the prediction residual block, the transform coefficients are inverse quantized (module 640) and inverse transformed (module 650). The decoded prediction residual block is combined (module 655) with a prediction block (also known as a predictor) to obtain a decoded / reconstructed block.

[0084] The prediction block may possibly result from intra prediction (module 660) or motion compensated prediction (i.e. inter prediction) (module 670) depending on the prediction mode information (module 605). An in-loop filter (module 665) may be applied to the reconstructed picture. The in-loop filter may include a deblocking filter and / or a SAO filter. The filtered picture is stored in a reference picture buffer 680.

[0085] The modules of the decoder 600 may be implemented in software and executed by a processor, or may be implemented using circuitry well known to those skilled in the art of compression. In particular, the decoder 600 may be implemented as an integrated circuit (IC) alone or in combination with the encoder 400 as a codec.

[0086] Except for the differences described in the embodiments of the present invention, in particular the motion compensation module 670 based on illumination compensation, which will be explained in more detail in the following paragraphs and figures, the modules of the decoder 600 are found in a traditional block-based video decoder (e.g., an HEVC decoder).

[0087] With respect to functionality other than illumination compensation (eg, modules other than 670), the decoder 600 may be similar to any conventional block-based video decoder, and its functionality will not be described in detail herein.

[0088] FIG. 7 illustrates a simplified block diagram 700 of some modules of an exemplary decoder in accordance with at least one embodiment.

[0089] Module 570 illustrates IC related functions within a module for a decoder according to an embodiment of the present invention.

[0090] The motion compensation 570 may include a subdivision module 571, an intra motion compensation module 572, an IC parameter derivation module 573, and an IC application module 574.

[0091] The module of FIG. 7 based on illumination compensation is explained in more detail in the following paragraphs and figures.

[0092] In one embodiment, IC may be adaptively enabled or disabled for each inter-predictive coded block.

[0093] If IC is enabled for the current slice / picture or sequence, an IC flag can be decoded from the bitstream for each block to indicate whether IC is enabled for the block.

[0094] Alternatively, the IC flag can be inferred, e.g., the IC flag can be derived from previously decoded blocks using a merge mode, and then the IC flag is obtained in the video decoder.

[0095] If the derived IC parameters do not result in improved prediction, IC can be locally or globally deactivated (e.g., the IC flag is set to false). If the IC flag enables IC for a block (e.g., the IC flag is set to true), the IC parameters for the block can be included and optionally decoded from the bitstream to be obtained at the decoder.

[0096] It should be understood that if the IC flag and IC parameters are included and, optionally, decoded from the bitstream, module 570 obtains the IC flag and IC parameters from the bitstream instead of calculating or deriving the IC parameters.

[0097] Below, embodiments relating to deriving or calculating IC parameters according to the embodiments are described.

[0098] FIG. 8 illustrates a flow diagram 800 of a method for encoding / decoding a block based on illumination compensation, according to at least one embodiment.

[0099] In step 810, the block is subdivided into sub-blocks as shown in Figure 9. At the decoder, the sub-division may be triggered by picture partition information decoded from the bitstream.

[0100] Next, a motion compensated reference sub-block is obtained from motion compensation (step 820), as will be described in detail in Figures 10 and 11, IC parameters are derived from neighboring samples of the motion compensated reference sub-block and neighboring samples of the sub-block (step 830), and a prediction sub-block is derived by applying illumination compensation to the motion compensated reference sub-block using the IC parameters, as will be described in detail in Figures 12 and 13a-d. Finally, the prediction sub-block is used to encode / decode the sub-block (step 840), as will be described in detail in Figure 14. This method is repeated for each sub-block of the block.

[0101] Steps 810 to 840 may be performed by, for example, the encoder 400 or 500 or the decoder 600 or 700 .

[0102] In particular, steps 810 to 830 may be performed by the encoder modules 470, 475 and 471 or the decoder module 670, for example.

[0103] FIG. 9 illustrates an example of subdividing a block (step 810) according to at least one embodiment.

[0104] In one embodiment, a block may be divided horizontally or vertically into at least two sub-blocks.

[0105] For example, if a block is divided horizontally (or vertically), the coding and decoding of sub-blocks is applied repeatedly downward (horizontal division) or rightward (vertical division) as indicated by the arrows in FIG.

[0106] The size (height and / or width) of the sub-blocks may or may not be equal.

[0107] For example, the width (or height) of a subblock can be equal to half or a quarter of the block width (or height), resulting in a 2:2 division or a 1:3 / 3:1 division. A 2:2 division is like a binary tree (BT) division, while a 1:3 / 3:1 division is like an asymmetric binary tree (ABT) division. For example, the minimum block size is 4x8 (or 8x4). If the block size is more than 4x8 (or 8x4), the corresponding block is divided by 4 subblocks.

[0108] In one embodiment, the number of sub-blocks may depend on the size of the block. For example, if one side of the block is 8, then 1:3 / 3:1 division along that side is not allowed. For example, the minimum expected block size is 4x8 (or 8x4). If the block size is greater than 4x8 (or 8x4), then the block is divided by 4 sub-blocks.

[0109] In one embodiment, the sub-division may be signaled in the bitstream by a syntax element. Step 571 in the decoder then obtains the syntax element and divides the block according to the signaled sub-division.

[0110] FIG. 10 illustrates a flow diagram of a method 1000 for deriving a motion compensated reference sub-block for a block, according to at least one embodiment.

[0111] For example, any well-known block-based motion estimation method may be used to estimate a motion vector for each sub-block.

[0112] In a modification, the motion vector for each sub-block is derived from an estimated affine motion model that captures the translational motion field of the block. This modification improves the temporal prediction.

[0113] As shown in FIG. 11 (left side), the affine motion model can be defined by two control point motion vectors (four parameters), or three control point motion vectors (six parameters), or any number of control point motion vectors.

[0114] In a four-parameter affine motion model, the motion vector at a sample location (x,y) within a block can be derived as follows:

number

[0115] With a 6-parameter affine motion model, the motion vector at a sample location (x,y) within a block can be derived as follows:

number

[0116] The motion vectors at the control points can be estimated using any well-known motion estimation method.

[0117] For example, as shown in FIG. 11 (right side), the motion vector of the center sample of each sub-block calculated according to the above equations and possibly rounded to 1 / 16 fractional precision is the motion vector of the sub-block.

[0118] FIG. 12 shows a flow diagram of a method 1200 for deriving one or more IC parameters and adjusting a motion compensated reference sub-block according to an embodiment of the present invention.

[0119] The following illumination compensation (IC) model can be used to illumination compensate the motion compensated reference sub-blocks.

[0120] The IC model is: IC(x sb )=a sb *x sb +b sb (3) is determined by a sb and b sb are the IC parameters of the motion compensated reference sub-block sb, which are commonly called the slope or scale and intercept or offset of a linear function, respectively.

[0121] As shown in Figures 13a-d, in one embodiment, the neighboring samples of a sub-block may be an L-shaped set of neighboring samples, in a variant, not all neighboring samples of the L-shaped set are used, but only a part of them (e.g. no intra-mode reconstructed samples are used).

[0122] The L-set is defined as a set of causal neighboring samples of a subblock, i.e., neighboring samples that can be used to reconstruct the current subblock. Thus, some samples of the L-set may be inside or outside the block, as shown in Fig. 13a (horizontal partition) and Fig. 13b (vertical partition).

[0123] Figures 13c and 13d show alternative definitions of L-shaped sets according to different subdivisions of a block, where the arrows indicate the scanning order and adjacent sub-blocks for coding the sub-blocks of a block.

[0124] In step 1210 of FIG. 12, a set of adjacent samples of the current sub-block is obtained.

[0125] In step 1220, a reference sub-block in a reference picture is pointed to by the motion vector of the sub-block, and the neighboring samples of the reference sub-block are the neighboring samples of the motion compensated reference sub-block.

[0126] The difference between the adjacent samples of the current sub-block and the adjacent samples of the motion-compensated reference sub-block is defined as the IC parameter a sb and b sb The IC parameters can be estimated by minimizing the neighboring samples with

[0127] In step 1230, the difference between adjacent samples can be minimized under the least squares method as follows:

number

[0128] In step 1240, the IC parameters (a sb ,b sb ) to derive an adjusted motion compensated reference sub-block by applying the illumination compensation model to the motion compensated reference sub-block, i.e., each sample (x,y) of the motion compensated reference sub-block is adjusted according to Equation 3.

[0129] FIG. 14 illustrates a flow diagram of a method 1400 for encoding a sub-block of a block from a predictive sub-block, according to at least one embodiment.

[0130] Step 1410 calculates a prediction residual sub-block for a sub-block of the block as the difference between the sub-block and the prediction sub-block.

[0131] At step 1420, the prediction residual sub-block is transformed, quantized, and entropy coded.

[0132] This method is repeated for each sub-block of the block.

[0133] In one embodiment, the prediction residual sub-blocks are transformed by using one separate transform.

[0134] In a variation of this embodiment, the neighboring samples of the subblock and the predicted subblock are the reconstructed samples and the samples of the decoded block to be coded, and the IC parameters are derived only from the reconstructed samples.

[0135] In one embodiment, the neighboring samples of a subblock and a predicted subblock are formed only with intra samples, which allows independence of the coding / decoding of the blocks and therefore parallel processing.

[0136] For example, in the case of FIG. 13c, the sub-block can (a) use only the left sample, (b) use only the top sample, or (c) use both the left and top samples.

[0137] In one embodiment, a prediction residual block is formed from all the prediction residual sub-blocks, and the prediction residual block is transformed using a single transform, quantized, and entropy coded.

[0138] In a variation of this embodiment, the neighbouring samples of a sub-block and a predictive sub-block are formed only with extrinsic samples (eg as shown in Figures 13a to 13d).

[0139] In another variant of this embodiment, the neighbouring samples of the subblock and the predictive subblock are formed exclusively from samples of at least one further predictive subblock.

[0140] In one embodiment, the neighbouring samples of a subblock and a predictive subblock are formed with the exterior samples and samples of at least one further predictive subblock within the block.

[0141] For example, in the case of FIG. 13a, the left sample of the current sub-block and the sample of the upper adjusted prediction sub-block are used.

[0142] In another variation, a first subblock is identified as being the first subblock to be coded when traversing the set of subblocks in scan order, and the IC parameters of this first subblock are default or the first predicted subblock is not illumination compensated.

[0143] In one embodiment, the IC parameters are derived based on a subset of the sub-blocks to reduce the amount of computation and therefore the complexity.

[0144] For example, in HEVC, Virtual Pipeline Data Units (VPDUs) define a subset of subblocks within a block, and IC parameters are derived based only on samples within the VPDU.

[0145] FIG. 15 shows the case where four VPDUs are created to group the sub-blocks of a block.

[0146] In a variation of this embodiment, the first sub-block of a VPDU may not use IC parameters or may use default IC parameters, in this way there are no data dependencies between VPDUs and they can be processed in parallel.

[0147] This can be applied to the horizontal or vertical split cases as well.

[0148] In one embodiment, the IC function may include Equation 1 and Equation 3, or other linear or nonlinear functions of the IC parameters. For example, the IC function may consider only the slope parameter without the intercept parameter (i.e., IC(x)=a*x). In one embodiment, the IC parameters may be more than two parameters depending on the function (e.g., depending on the degree of a polynomial function).

[0149] In one embodiment, Equation 2 and Equation 4 may be based on the absolute difference rather than the squared difference.

[0150] The IC parameters may include at least one parameter, such as at least one of an amplitude scale or slope parameter, an amplitude shift or intercept parameter, a position scale parameter, a position shift parameter, a time scale parameter, a time shift parameter, and the like.

[0151] FIG. 16 illustrates a flow diagram 1600 of an exemplary method for encoding video in accordance with an embodiment of the present invention.

[0152] The method 1600 includes obtaining a block in a picture of a video and encoding the block according to the method 800. The method 1600 is repeated to encode all blocks of the picture.

[0153] FIG. 17 illustrates a flow diagram 1700 of an exemplary method for decoding video in accordance with an embodiment of the present invention.

[0154] The method 1700 includes decoding information from a bitstream, such as transform coefficients, motion vectors, picture partition information, possibly prediction mode information, other syntax elements, and / or other coded information.

[0155] In one embodiment, the IC prediction flag is decoded to determine whether IC is enabled for a block, picture, slice, or sequence.

[0156] In one embodiment, if the IC flag enables IC for a block (eg, the IC flag is set to true), the IC parameters for the block may be included and optionally decoded from the bitstream.

[0157] The method then includes dequantizing the transform coefficients (module 640) and inverse transforming (module 650) to decode the prediction residual block. The method then includes subdividing the block and deriving a prediction block. Finally, the method includes combining the decoded prediction residual block with the prediction block to obtain a decoded / reconstructed block.

[0158] The prediction block may possibly be obtained from intra prediction (module 660) or motion compensated prediction (i.e. inter prediction) (module 670) depending on the prediction mode information (module 605). If IC is enabled, the prediction block is derived from steps 810-830 of the method 800.

[0159] The IC parameters may include at least one of an amplitude scale or slope parameter, an amplitude shift or intercept parameter, a position scale parameter, a position shift parameter, a time scale parameter, a time shift parameter, etc. The block may be a reconstructed block.

[0160] According to one embodiment of this method, the function may be one of a sum of squared differences and a sum of absolute differences.

[0161] FIG. 18 illustrates a block diagram of an example of a system in which various aspects and embodiments may be implemented.

[0162] System 1800 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include various electronic devices, such as, but not limited to, personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected consumer electronics, and servers. The elements of system 1800 may be embodied singly or in combination within a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1800 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1800 is communicatively coupled to one or more other systems or other electronic devices, for example, via communication bus 1940 or by dedicated input and / or output ports. In various embodiments, system 1800 is configured to implement one or more of the aspects described herein.

[0163] The system 1800 includes at least one processor 1810 configured to execute instructions loaded therein to implement various aspects described herein, for example. The processor 1810 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 1800 includes at least one memory 1820, such as a volatile memory device and / or a non-volatile memory device. The system 1800 includes a storage device 1840, which may include non-volatile and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read Only Memory (EEPROM), Read Only Memory (ROM), Programmable Read Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Flash, magnetic disk drives, and / or optical disk drives. The storage device 1840 may include, by way of non-limiting example, an internal storage device, an additional storage device (including removable and non-removable storage devices), and / or a network-accessible storage device.

[0164] The system 1800 includes an encoder / decoder module 1830 configured to process data to provide, for example, encoded or decoded video, and the encoder / decoder module 1830 may include its own processor and memory. The encoder / decoder module 1830 represents a module that may be included within a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Additionally, the encoder / decoder module 1830 may be implemented as a separate element of the system 1800 or may be incorporated within the processor 1810 as a combination of hardware and software, as is known to those skilled in the art. Program code that is loaded onto the processor 1810 or the encoder / decoder 1830 to perform various aspects described herein may be stored in the storage device 1840 and subsequently loaded onto the memory 1820 for execution by the processor 1810. According to various embodiments, one or more of the processor 1810, memory 1820, storage device 1840, and encoder / decoder module 1830 may store one or more of various items while performing the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results resulting from the processing of equations, formulas, operations, and computational logic.

[0165] In some embodiments, memory internal to the processor 1810 and / or the encoder / decoder module 1830 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing unit (e.g., the processing unit can be the processor 1810 or the encoder / decoder module 1830) is used for one or more of these functions. The external memory can be the memory 1820 and / or the storage 1840, for example dynamic volatile memory and / or non-volatile flash memory. In some embodiments, an external non-volatile flash memory is used, for example, to store the television's operating system. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for video coding and decoding operations such as MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET, the Joint Video Experts Team).

[0166] Inputs to the elements of system 1800 may be provided by various input devices, as shown in block 1930. Such input devices include, but are not limited to, (i) an RF section that receives an RF (radio frequency) signal transmitted wirelessly, for example by a broadcaster, (ii) a component (COMP) input (or a set of COMP inputs), (iii) a universal serial bus (USB) input, and / or (iv) a high-definition multimedia interface (HDMI) input. Other examples not shown in FIG. 18 include composite video.

[0167] In various embodiments, the input devices of block 1930 have associated individual input processing elements known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band, which may be referred to as a channel in a particular embodiment (for example), (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a band-limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner to perform various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency close to the baseband band) or to the baseband band. In some set-top box embodiments, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, remove some of the elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0168] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting the system 1800 to other electronic devices across the USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented as desired, e.g., in a separate input processing IC or in the processor 1810. Similarly, aspects of the USB or HDMI interface processing may be implemented as desired, e.g., in a separate interface IC or in the processor 1810. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, e.g., the processor 1810 and an encoder / decoder 1830 operating in conjunction with memory and storage elements, to process the data stream as desired for presentation on an output device.

[0169] The various elements of the system 1800 may be provided within a unitary housing in which the various elements are interconnected and may transmit data therebetween using an internal bus 1940 known in the art, including any suitable connection configuration, such as an Inter-IC (I2C) bus, hardwired, and printed circuit boards.

[0170] System 1800 includes a communication interface 1850 that enables communication with other devices over a communication channel 1860. Communication interface 1850 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 1860. Communication interface 1850 may include, but is not limited to, a modem or a network card, and communication channel 1860 may be implemented in a wired and / or wireless medium, for example.

[0171] In various embodiments, data is streamed or provided to the system 1800 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received on a communication channel 1860 and communication interface 1850 adapted for Wi-Fi communication. The communication channel 1860 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. Other embodiments provide stream data to the system 1800 using a set-top box that delivers data on the HDMI connection of the input block 1830. Still other embodiments provide stream data to the system 1800 using the RF connection of the input block 1830. As indicated above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use wireless networks other than Wi-Fi, e.g., a cellular network or a Bluetooth network.

[0172] The system 1800 can provide output signals to various output devices including a display 1900, speakers 1910, and other peripherals 1920. The display 1900 in various embodiments includes one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display, for example. The display 1900 can be for a television, a tablet, a laptop, a cell phone, or other device. The display 1900 can be integrated with other components (e.g., as in a smartphone) or can be separate (e.g., an external monitor for a laptop). In various example embodiments, the other peripherals 1920 include one or more of a stand-alone digital video disc (or digital versatile disc) (DVR in both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripherals 1920 that provide functionality based on the output of the system 1800. For example, a disc player performs the function of playing the output of the system 1800.

[0173] In various embodiments, control signals are communicated between the system 1800 and the display 1900, speaker 1910, or other peripheral device 1920 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that allow inter-device control with or without user intervention. Output devices may be communicatively coupled to the system 1800 via dedicated connections through individual interfaces 1870, 1880, and 1890. Alternatively, output devices may be connected to the system 1800 using a communication channel 1860 via the communication interface 1850. The display 1900 and speaker 1910 may be integrated into a single unit with other components of the system 1800 in an electronic device such as a television. In various embodiments, the display interface 1870 includes a display driver, such as a timing controller (T Con) chip.

[0174] For example, if the RF portion of input 1930 is part of a separate set-top box, then display 1900 and speakers 1910 may alternatively be separate from one or more of the other components. In various embodiments in which display 1900 and speakers 1910 are external components, the output signal may be provided by a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0175] The implementations described herein may be implemented, for example, by a method or process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single type of implementation (e.g., only discussed as a method), the implementations of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented, for example, by appropriate hardware, software, and firmware. A method may be implemented by an apparatus, such as, for example, a processor, which refers generally to processing devices, including, for example, computers, microprocessors, integrated circuits, or programmable logic devices. Processors also include communication devices, such as, for example, computers, cell phones, portable / personal digital assistants (PDAs), and other devices that facilitate communication of information between end users.

[0176] According to one aspect of an embodiment of the present invention, there is provided an apparatus 1800 for encoding video, the apparatus including a processor 1810 and at least one memory 1820, 1840 coupled to the processor, the processor 1810 configured to perform any of the embodiments of the methods 800, 1600, and / or 1700 described above.

[0177] According to one aspect of the present disclosure, an apparatus for encoding a video is provided, including: means for deriving one or more illumination compensation parameters based on neighboring samples of a subblock of a block and neighboring samples of a motion-compensated reference subblock; means for deriving a prediction subblock by applying illumination compensation to the motion-compensated reference subblock using the one or more derived illumination compensation parameters; and means for encoding a subblock using the prediction subblock. The video encoder of FIG. 4 and FIG. 5 may include the structure or means of the apparatus, specifically, blocks 470, 475, 571 to 575. The apparatus for encoding a video may perform any of the embodiments of the methods 800 and 1600.

[0178] According to one aspect of the present disclosure, there is provided an apparatus for decoding video, including: means for deriving one or more illumination compensation parameters based on neighboring samples of a subblock of a block and neighboring samples of a motion-compensated reference subblock; means for deriving a prediction subblock by applying illumination compensation to the motion-compensated reference subblock using the one or more derived illumination compensation parameters; and means for encoding a subblock using the prediction subblock. The video decoder of FIG. 6 and FIG. 7 may include an apparatus structure or means, specifically blocks 670, 571-574. The apparatus for decoding video may perform any of the embodiments of the methods 800 and 1700.

[0179] As will be apparent to one of ordinary skill in the art, implementations can result in a wide variety of signals formatted to carry information that can be, for example, stored or transmitted. Information can include, for example, instructions for performing a method or data produced by one of the described implementations. For example, a signal can be formatted to carry a bit stream of the described embodiments. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal can be, for example, analog information or digital information. The signal can be transmitted over a wide variety of different wired or wireless links as is known. The signal can be stored on a processor-readable medium.

[0180] Additionally, any of the methods 800, 1600, and / or 1700 may be implemented as a computer program product including (independently or together) computer-executable instructions that may be executed by a processor. The computer program product having computer-executable instructions may be stored in a transitory or non-transitory computer-readable storage medium of the system 1800, the encoder 400 (or 500), and / or the decoder 600 (or 700), respectively.

[0181] It is important to note that in some embodiments, one or more of the elements in processes 800, 1600, and / or 1700 may be combined, performed in a different order, or eliminated while still implementing aspects of the present disclosure. Other steps may be performed in parallel, where the processor does not wait for one step to be fully completed before starting another step.

[0182] Furthermore, aspects of embodiments of the present invention may take the form of a computer readable storage medium. Any combination of one or more computer readable storage media may be utilized. The computer readable storage medium may take the form of a computer readable program product embodied in one or more computer readable media and having computer readable program code embodied thereon that is executable by a computer. As used herein, a computer readable storage medium is considered to be a non-transitory storage medium given the inherent ability to store information therein as well as the inherent ability to allow information to be retrieved therefrom. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above.

[0183] As will be readily appreciated by those skilled in the art, the following list provides more specific examples of computer-readable storage media to which the present disclosure may be applied, but should be understood to be merely illustrative and not exhaustive: The list of examples includes portable computer diskettes, hard disks, ROMs, EPROMs, flash memories, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0184] According to one aspect of an embodiment of the present invention, there is provided a computer readable storage medium carrying a software program including program code instructions for performing any of the embodiments of the methods of the present invention, including methods 800, 1600, and / or 1700.

[0185] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" of the present disclosure, as well as other variants thereof, should be understood to mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment of the present disclosure. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" in various places throughout this specification, as well as any other variants, do not necessarily all refer to the same embodiment.

[0186] Additionally, this disclosure or claims thereof may refer to "determining" various pieces of information. Determining or deriving information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0187] Additionally, the application or claims may refer to "providing" various pieces of information. Providing information may include, for example, one or more of outputting information, storing information, transmitting information, sending information, displaying information, showing information, or moving information.

[0188] Additionally, the application or claims thereof may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, processing information, moving information, replicating information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0189] Additionally, the application or claims may refer to "receiving" various pieces of information. Receiving is intended to be broad, similar to "accessing." Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Additionally, "receiving" typically involves some form of operation, such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0190] It should be understood that the various features shown and described are interchangeable. Features shown in one embodiment can be combined in another embodiment unless otherwise specified. Furthermore, features described in the various embodiments can be combined or separated unless otherwise indicated as inseparable or incombinable.

[0191] As noted above, the functions of the various elements shown may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. Further, when provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a number of individual processors, some of which may be shared.

[0192] It should be further understood that some of the constituent system components and methods illustrated in the accompanying figures are preferably implemented by software, and therefore the actual connections between the system components or process function blocks may vary depending on the manner in which the processes of the present disclosure are programmed. Given the teachings herein, one of ordinary skill in the art will be able to contemplate these and similar implementations or configurations of the present disclosure.

[0193] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the disclosure is not limited to those exact embodiments, and various changes and modifications may be made by those skilled in the art without departing from the scope of the disclosure. In addition, individual embodiments may be combined without departing from the scope of the disclosure. It is intended that all such changes and modifications are included within the scope of the disclosure as defined in the appended claims.

[0194] For example, the use of " / ", "and / or" and "at least one of" in the cases of "A / B", "A and / or B" and "at least one of A and B" is intended to encompass selecting only the first listed option (A), or selecting only the second listed option (B), or selecting both options (A and B). As a further example, in the cases of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass selecting only the first listed option (A), or selecting only the second listed option (B), or selecting only the third listed option (C), or selecting only the first and second listed options (A and B), or selecting only the first and third listed options (A and C), or selecting only the second and third listed options (B and C), or selecting all three options (A and B and C). As one skilled in the art would readily appreciate, this representation can be expanded to include as many items as are listed.

[0195] As will be apparent to one of ordinary skill in the art, implementations can result in a wide variety of signals formatted to carry information that can be, for example, stored or transmitted. Information can include, for example, instructions for performing a method or data produced by one of the described implementations. For example, a signal can be formatted to carry a bit stream of the described embodiments. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal can be, for example, analog information or digital information. The signal can be transmitted over a wide variety of different wired or wireless links as is known. The signal can be stored on a processor-readable medium.

Claims

1. 1. A method of encoding a block of video, comprising the steps of: deriving one or more illumination compensation parameters based on samples neighboring a sub-block of the block that is a current sub-block and samples neighboring a motion compensated reference sub-block of the current sub-block; deriving a prediction sub-block by applying illumination compensation to the motion compensated reference sub-block using the one or more derived illumination compensation parameters; encoding the current sub-block using the predicted sub-block; A method comprising:

2. 1. A method of decoding a block of video, comprising the steps of: deriving one or more illumination compensation parameters based on samples neighboring a sub-block of the block that is a current sub-block and samples neighboring a motion compensated reference sub-block of the current sub-block; deriving a prediction sub-block by applying illumination compensation to the motion compensated reference sub-block using the one or more derived illumination compensation parameters; decoding the current sub-block using the predicted sub-block; A method comprising:

3. 1. An apparatus for encoding blocks of video, comprising: means for deriving one or more illumination compensation parameters based on samples adjacent to a sub-block of the block being a current sub-block and samples adjacent to a motion compensated reference sub-block of the current sub-block; means for deriving a prediction sub-block by applying illumination compensation to the motion compensated reference sub-block using the one or more derived illumination compensation parameters; means for encoding the current sub-block using the predicted sub-block; Including, equipment.

4. 1. An apparatus for decoding blocks of video, comprising: means for deriving one or more illumination compensation parameters based on samples adjacent to a sub-block of the block being a current sub-block and samples adjacent to a motion compensated reference sub-block of the current sub-block; means for deriving a prediction sub-block by applying illumination compensation to the motion compensated reference sub-block using the one or more derived illumination compensation parameters; means for decoding the current sub-block using the predicted sub-block; Including, equipment.

5. 5. The method of claim 1 or 2 or the apparatus of claim 3 or 4, wherein multiple sub-blocks of a same block are coded using a predictive sub-block, and coding or decoding the sub-blocks comprises applying a separate transform to each residual derived from a sub-block and the associated predictive sub-block.

6. The method according to claim 1 or 2, 5 or the apparatus according to claim 3 to 5, wherein the neighboring samples of the current sub-block are samples within the block.

7. A method according to claim 1 or 2 or an apparatus according to claim 3 or 4, wherein encoding or decoding the sub-block comprises applying a single transform to a prediction residual derived from the sub-block and the prediction residual derived from the predictive sub-block.

8. The method or apparatus of claim 7 , wherein one or more neighboring samples of the current sub-block are samples outside the block.

9. 9. The method or device according to claim 7 or 8, wherein the neighbouring samples of the sub-block and the prediction sub-block are formed exclusively using samples of at least one further prediction sub-block.

10. The method according to claim 1, 2, 5 to 9 or the device according to claim 3 to 9, wherein the adjacent samples of a subblock and a predictive subblock are formed using samples outside the block and samples of at least one further predictive subblock within the block.

11. The method according to claim 1 , 2 , 5 to 10 or the apparatus according to claim 3 to 10, wherein one or more illumination compensation parameters are derived from adjacent samples within a subset of sub-blocks of the block.

12. 12. A non-transitory computer readable storage medium storing instructions for encoding or decoding blocks of video according to the method of any one of claims 1, 2, and 5 to 11.

13. A non-transitory computer readable medium comprising picture data content generated according to the method or apparatus of any one of claims 1 to 12.

14. A computer program product comprising instructions for carrying out the method of any one of claims 1, 2 and 5 to 11 when executed by one of a plurality of processors.

Images