Video decoding apparatus and video decoding method

Abstract

The invention relates to a video decoding apparatus and a video decoding method. The video decoding apparatus includes: a processor configured to obtain an affine motion model associated with a block, the affine motion model including a plurality of control point motion vectors; obtaining local illumination compensation information associated with the block based on a plurality of control point motion vectors of an affine motion model associated with the block; and decoding the block based on the affine motion model and the local illumination compensation information.

Description

[0001] This application is a divisional application of the Chinese invention patent application with application date of May 7, 2019 and application number 201980043676.X. Technical Field

[0002] The present disclosure relates to video encoding and decoding. Background Art

[0003] In order to achieve high compression efficiency, image and video coding schemes (such as those defined by the HEVC (High Efficiency Video Coding) standard) typically employ prediction and transform coding to exploit spatial and temporal redundancy in video content. Generally, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlation, and then the difference between the original block and the predicted block (often expressed as a prediction error or prediction residual) is transformed, quantized, and entropy encoded. In order to reconstruct the video, the compressed data is decoded by an inverse process corresponding to prediction, transformation, quantization, and entropy coding. Recent additions to video compression technology include reference software and / or documentation of various versions of the Joint Exploration Model (JEM) developed by the Joint Video Exploration Team (JVET). Efforts such as JEM aim to further improve existing standards such as HEVC. Summary of the invention

[0004] In general, an example of an embodiment of a method or an apparatus including, for example, one or more processors may include: processing video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination variation in the video information based on a plurality of motion vectors contained in the motion compensation information; and the video information includes a coding unit having a plurality of sub-blocks, the plurality of sub-blocks including a first row of sub-blocks and a first column of sub-blocks; and the plurality of motion vectors include a motion vector group associated with a corresponding sub-block in each sub-block included in the first row of sub-blocks and each sub-block included in the first column of sub-blocks; and determining the at least one model parameter includes obtaining a set of reconstructed samples forming an L-like shape based on the motion vector group, and evaluating local distortion based on the set of reconstructed samples.

[0005] Another example of an embodiment of a method or an apparatus including, for example, one or more processors may include: processing video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model; wherein the video information includes a coding unit having a plurality of sub-blocks, the plurality of sub-blocks including a first row of sub-blocks and a first column of sub-blocks; and the motion compensation information includes a first set of motion vectors associated with a corresponding sub-block in each sub-block included in the first row of sub-blocks and a second set of motion vectors associated with a corresponding sub-block in each sub-block included in the first column of sub-blocks; and obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination variation in the video information based on one or more of a first subset of the first set of motion vectors or a second subset of the second set of motion vectors; and determining the at least one model parameter includes obtaining a set of reconstructed samples forming an L-like shape based on one or more of the first subset of the first set of motion vectors or a second subset of the second set of motion vectors, and evaluating local distortion based on the set of reconstructed samples.

[0006] Another example of an embodiment of a method or an apparatus including, for example, one or more processors may include: processing video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector contained in the motion compensation information; and the video information includes a coding unit having a plurality of sub-blocks; and the first motion vector is associated with a first sub-block of the plurality of sub-blocks located in the upper left corner of the coding unit; and determining the at least one model parameter includes obtaining a set of reconstructed samples forming an L shape based on the first motion vector, and evaluating local distortion based on the set of reconstructed samples.

[0007] Another example of an embodiment of a method or an apparatus comprising, for example, one or more processors may include: processing video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model comprises determining at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector contained in the motion compensation information; and the first motion vector is associated with a center of a coding unit; and determining the at least one model parameter comprises obtaining a set of reconstructed samples forming an L shape based on the first motion vector, and evaluating local distortion based on the set of reconstructed samples.

[0008] Another example of an embodiment of a method or an apparatus including, for example, one or more processors may include: processing video information based on a coding unit including multiple sub-blocks, thereby generating motion compensation information including multiple motion vectors associated with corresponding sub-blocks in the multiple sub-blocks; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination variation in the video information based on at least one of the multiple motion vectors; and the multiple sub-blocks included in the coding unit include a first row of sub-blocks and a first column of sub-blocks; and the multiple motion vectors include a motion vector group associated with corresponding sub-blocks in each sub-block included in the first row of sub-blocks and each sub-block included in the first column of sub-blocks; and determining the at least one model parameter includes obtaining a group of reconstructed samples forming an L-like shape based on the motion vector group, and evaluating local distortion based on the group of reconstructed samples.

[0009] Another example of an embodiment of a method or an apparatus including, for example, one or more processors may include: processing video information based on a coding unit including a plurality of sub-blocks to generate motion compensation information; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model; wherein the plurality of sub-blocks included in the coding unit include a first row of sub-blocks and a first column of sub-blocks; and the motion compensation information includes a first set of motion vectors associated with a corresponding sub-block in each sub-block included in the first row of sub-blocks and a second set of motion vectors associated with a corresponding sub-block in each sub-block included in the first column of sub-blocks; and obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination variation in the video information based on one or more of a first subset of the first set of motion vectors or a second subset of the second set of motion vectors; and determining the at least one model parameter includes obtaining a set of reconstructed samples forming an L-like shape based on one or more of the first subset of the first set of motion vectors or a second subset of the second set of motion vectors, and evaluating local distortion based on the set of reconstructed samples.

[0010] Another example of an embodiment of a method or an apparatus comprising, for example, one or more processors may include: processing video information based on a coding unit comprising multiple sub-blocks to generate motion compensation information comprising multiple motion vectors associated with corresponding sub-blocks of the multiple sub-blocks; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model comprises determining at least one model parameter of a linear model of illumination variation in the video information based on a first motion vector contained in the motion compensation information; and the first motion vector is associated with a first sub-block of the multiple sub-blocks located in the upper left corner of the coding unit; and determining the at least one model parameter comprises obtaining a set of reconstructed samples forming an L shape based on the first motion vector, and evaluating local distortion based on the set of reconstructed samples.

[0011] Another example of an embodiment of a method or an apparatus comprising, for example, one or more processors may include: processing video information based on a coding unit comprising multiple sub-blocks to generate motion compensation information comprising multiple motion vectors associated with corresponding sub-blocks of the multiple sub-blocks; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model comprises determining at least one model parameter of a linear model of illumination variation in the video information based on a first motion vector contained in the motion compensation information; and the first motion vector is associated with a center of the coding unit; and determining the at least one model parameter comprises obtaining a set of reconstructed samples forming an L shape based on the first motion vector, and evaluating local distortion based on the set of reconstructed samples.

[0012] Generally speaking, an embodiment may relate to a method for encoding video information, comprising: processing the video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model.

[0013] In general, another embodiment may relate to a device for encoding video information, comprising one or more processors, wherein the one or more processors are configured to: process the video information based on an affine motion model to generate motion compensation information; obtain a local illumination compensation model; and encode the video information based on the motion compensation information and the local illumination compensation model.

[0014] Generally speaking, another embodiment may relate to a method of decoding video information, comprising: processing the video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and decoding the video information based on the motion compensation information and the local illumination compensation model.

[0015] In general, another embodiment may relate to a device for decoding video information, comprising one or more processors, wherein the one or more processors are configured to: process the video information based on an affine motion model to generate motion compensation information; obtain a local illumination compensation model; and encode the video information based on the motion compensation information and the local illumination compensation model.

[0016] In general, another embodiment may relate to a bitstream formatted to include encoded video information, wherein the encoded video data is encoded by: processing the video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model.

[0017] Generally speaking, another embodiment may relate to a video decoding device, comprising: a processor, configured to: obtain an affine motion model associated with a block, the affine motion model comprising a plurality of control point motion vectors; obtain local illumination compensation information associated with the block based on the plurality of control point motion vectors of the affine motion model associated with the block; and decode the block based on the affine motion model and the local illumination compensation information.

[0018] Generally speaking, another embodiment may relate to a method of video decoding, comprising: obtaining an affine motion model associated with a block, the affine motion model comprising a plurality of control point motion vectors; obtaining local illumination compensation information associated with the block based on the plurality of control point motion vectors of the affine motion model associated with the block; and decoding the block based on the affine motion model and the local illumination compensation information.

[0019] In general, another embodiment may relate to a method of generating a local illumination compensation LIC flag, comprising: grouping a plurality of sub-blocks into a sub-block group; deriving a pair of local illumination compensation LIC parameters; associating the pair of LIC parameters with the sub-block group; and correcting the illumination of a sub-block in the sub-block group based on the LIC parameters associated with the sub-block group.

[0020] In general, one or more embodiments may also provide a computer-readable medium, such as a non-volatile computer-readable storage medium, having instructions stored thereon for encoding or decoding video data according to the methods or apparatus described herein. One or more of the presented embodiments may also provide a computer-readable storage medium having a bitstream stored thereon, the bitstream being generated according to the methods or apparatus described herein. One or more of the presented embodiments may also provide a method and apparatus for sending or receiving a bitstream generated according to the methods or apparatus described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present disclosure may be better understood by considering the following detailed description in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A block diagram illustrating an example of an embodiment of a video encoder;

[0023] Figure 2 A block diagram illustrating an example of an embodiment of a video decoder;

[0024] Figure 3 A block diagram illustrating an example of an embodiment of a system providing video encoding and / or decoding;

[0025] Figure 4 An example of dividing a coding tree unit (CTU) into coding units (CU), prediction units (PU) and transform units (TU) in, for example, HEVC is shown;

[0026] Figure 5 An example of an affine motion model, such as that used in the Joint Exploration Model (JEM), is shown;

[0027] Figure 6 An example of an affine motion vector field based on a 4×4 sub-CU is shown;

[0028] Figure 7 An example of a motion vector (MV) prediction process for an affine AMVP CU is shown;

[0029] Figure 8 An example of motion vector prediction candidates in affine merge mode is shown;

[0030] Fig. 9 The spatial derivation of the affine motion field control points in the affine merge mode is shown;

[0031] Fig.10 An example of the use of neighboring samples for deriving parameters associated with a local illumination compensation (LIC) model is shown;

[0032] Fig.11 An example of the derivation of parameters for the LIC model in bidirectional prediction mode is shown;

[0033] Fig.12 shows examples of neighboring samples located in different L shapes surrounding a block of a reference picture, the block of the reference picture corresponding to a subblock of a current coding unit based on a motion vector associated with a subblock of the current coding unit;

[0034] Fig.13 An example of an embodiment of a portion of a video encoder is shown;

[0035] Fig.14 An example of another embodiment of a portion of a video encoder is shown;

[0036] Fig.15 shows an example of neighboring samples located in an L-like shape surrounding a block of a reference picture, the block of the reference picture corresponding to a subblock of a current coding unit based on a motion vector associated with a subblock of a current coding unit;

[0037] Fig.16 An example of another embodiment of a portion of a video encoder is shown;

[0038] Fig.17 shows examples of neighboring samples located in different L shapes surrounding a block of a reference picture, the block of the reference picture corresponding to a subblock group of a current coding unit based on a motion vector associated with the subblock group of the current coding unit;

[0039] Fig.18 Another example of an embodiment of a portion of a video encoder is shown;

[0040] Fig.19 An example of an embodiment of a portion of a video decoder is shown;

[0041] Fig. 20 Another example of an embodiment of a portion of a video decoder is shown;

[0042] Fig.21 Another example of an embodiment of a portion of a video decoder is shown;

[0043] Fig. 22 Another example of an embodiment of a portion of a video decoder is shown;

[0044] Fig.23 Another example of an embodiment of a portion of a video encoder is shown;

[0045] Fig.24 Another example of an embodiment of a portion of a video encoder is shown;

[0046] Fig.25 An example of an embodiment of a motion vector at the center of a coding unit is shown;

[0047] Fig.26 Another example of an embodiment of a motion vector at the center of a coding unit is shown;

[0048] Fig. 27 Another example of an embodiment of a motion vector at the center of a coding unit is shown; and

[0049] Fig.28 An example of an embodiment to determine parameters of a local illumination compensation model is shown.

[0050] Like reference numbers in the various drawings refer to the same or similar features. DETAILED DESCRIPTION

[0051] Recently, efforts to improve video compression technology, such as the Joint Exploration Model (JEM) developed by the Joint Video Exploration Team (JVET) have provided advanced features and tools. For example, such development efforts include providing support for additional motion models to improve temporal prediction. One such motion model is the affine motion model, described in more detail below. Support for tools such as block-based local illumination compensation (LIC) is also provided. The LIC tool involves applying the LIC model to predict illumination changes that may occur between a prediction block and a corresponding reference block for motion compensated prediction. Various aspects and embodiments described herein relate to motion models and tools, such as, for example, affine motion models and LIC tools.

[0052] This document describes various aspects including tools, features, embodiments, models, schemes, etc. Many of these aspects are described in conjunction with specificity, at least individual characteristics will be shown, and are often described in a manner that sounds restrictive. However, this is for the purpose of clarity of description and does not limit the application or scope of these aspects. In fact, all of the different aspects can be combined and interchanged to provide other aspects. In addition, these aspects can also be combined and interchanged with aspects described in earlier submissions.

[0053] The aspects described and contemplated in this document may be implemented in many different forms. Figure 1 , Figure 2 and Figure 3 and other figures throughout this document provide some embodiments, but other embodiments are contemplated and are not intended to be construed in detail. Figure 1 , Figure 2 and Figure 3The discussion does not limit the breadth of the scope of the embodiments. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting the generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, or computer-readable storage media. For example, the computer-readable storage medium can be a non-transitory computer-readable medium. The computer-readable storage medium can have stored thereon instructions for encoding or decoding video data according to any of the described methods and / or a bitstream generated according to any of the described methods.

[0054] In this application, the terms "reconstruction" and "decoding" can be used interchangeably, the terms "pixel" and "sample" can be used interchangeably, and the terms "image", "picture" and "frame" can be used interchangeably. The term "reconstruction" is usually used on the encoder side and "decoding" is used on the decoder side, but this is not necessarily the case.

[0055] Various methods are described above, and each of these methods includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for the correct operation of the method, the order and / or use of specific steps and / or actions can be modified or combined.

[0056] Various methods and other aspects described in this document may be used to modify one or more modules of a video encoder and / or decoder, such as, for example, Figure 1 and Figure 2 The motion estimation module 170, motion compensation module 175 and / or motion compensation module 275 of the JVET or HEVC encoder 100 and decoder 200 shown. In addition, the various aspects described are not limited to JVET or HEVC, and can be applicable to, for example, other standards and recommendations (whether existing or developed in the future) and extensions of any such standards and recommendations (including JVET and HEVC). Unless otherwise specified or technically excluded, the various aspects described in this document can be used alone or in combination.

[0057] Various numerical values ​​may be used in this document. Any specific numerical values ​​are examples, and the various aspects described are not limited to these specific values.

[0058] Figure 1 , Figure 2 and Figure 3 Block diagrams of examples of embodiments of encoder 100, decoder 200, and system 1000 are shown, respectively. Variations of encoder 100, decoder 200, and system 1000 are contemplated, but the examples described below are provided and described for purposes of clarity rather than describing all possible or contemplated variations.

[0059] exist Figure 1 In the embodiment of the present invention, a video sequence may be subjected to a pre-encoding process (101) before being encoded, for example, applying a color transform to an input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping on input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0060] In encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is partitioned (102) and processed in units such as CUs. Each unit is encoded using, for example, intra or inter mode. When the unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of intra mode or inter mode to use to encode the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. For example, the prediction residual is calculated by subtracting (110) the predicted block from the original image block.

[0061] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy encoded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., encode the residual directly without applying the transform and quantization process.

[0062] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual is combined with the predicted block (155) to reconstruct the image block. An in-loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0063] Figure 2 2 shows a block diagram of an example of a video decoder 200. In the decoder 200, the bitstream is decoded by the decoder elements as described below. The video encoder 200 generally performs the same Figure 1 The decoding channel relative to the encoding channel described in . As mentioned above, Figure 1 The encoder 100 in FIG. 1 also typically performs video decoding as part of encoding the video data, eg, to provide a reference for further prediction.

[0064] Specifically, the input to the decoder includes a video bitstream, which may be generated by a video encoder such as Figure 1 The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors and other encoding information. The picture partition information indicates how the picture is partitioned. Therefore, the decoder can divide (235) the picture according to the decoded picture partition information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined (255) with the predicted block to reconstruct the image block. The predicted block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0065] The decoded picture may further undergo post-decoding processing (285), such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4), or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0066] Figure 3 A block diagram of a system in which various aspects and embodiments may be implemented is shown. System 1000 may be embodied as a device including various components described below and configured to perform one or more of the various aspects described in this document. Examples of such devices include, but are not limited to, personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Figure 3 As shown in and as known to those skilled in the art, system 1000 can be communicatively coupled to other similar systems and displays via communication channels to implement one or more of the various aspects described in this document.

[0067] The system 1000 may include at least one processor 1010 configured to execute instructions loaded therein for implementing one or more of the various aspects described in this document. The processor 1010 may include embedded memory, input and output interfaces, and various other circuits known in the art. The system 1000 may include at least one memory 1020 (e.g., a volatile memory device, a non-volatile memory device). The system 1000 may include a storage device 1040, which may include a non-volatile memory, including but not limited to an EEPROM, a ROM, a PROM, a RAM, a DRAM, an SRAM, a flash memory, a disk drive, and / or an optical drive. The storage device 1040 may include an internal storage device, an attached storage device, and / or a network-accessible storage device, as a non-limiting example. The system 1000 may include an encoder / decoder module 1030, which is configured to process data to provide encoded video or decoded video.

[0068] The encoder / decoder module 1030 represents a module that can be included in a device to perform encoding and / or decoding functions. As is known, a device can include one or both of an encoding module and a decoding module. In addition, the encoder / decoder module 1030 can be implemented as a separate element of the system 1000, or can be incorporated into the processor 1010 as a combination of hardware and software, which is known to those skilled in the art.

[0069] Program code to be loaded onto the processor 1010 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. Depending on the embodiment, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during the execution of the processes described in this document, including but not limited to input video, decoded video, bitstreams, equations, formulas, matrices, variables, operations, and operation logic.

[0070] The system 1000 may include a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data from the communication channel 1060. The communication interface may include, but is not limited to, a modem or a network card, and the communication channel may be implemented in a wired and / or wireless medium. The various components of the system 1000 may be connected or communicatively coupled together using various suitable connections including, but not limited to, internal buses, wiring, and printed circuit boards.

[0071] As will be explained in detail below, various aspects and embodiments according to the present disclosure may involve Figure 1 , Figure 2 and Figure 3 , such as motion estimation functions (e.g., Figure 1 module 175) and motion compensation functions (such as Figure 1 Module 170 and / or Figure 2 Module 275).

[0072] For clarity of description, the following detailed description will describe various aspects with reference to embodiments that relate to video compression techniques such as, for example, HEVC, JEM and / or H.266. However, the various aspects described are applicable to other video processing techniques and standards.

[0073] In the HEVC video compression standard, a picture is divided into so-called coding tree units (CTUs), and each CTU is represented by a coding unit (CU) in the compressed domain. Each CU is then given some intra or inter prediction parameters (prediction information). To do this, it is spatially partitioned into one or more prediction units (PUs), each of which is assigned some prediction information. Intra or inter coding modes are assigned at the CU level. Figure 4 An example of dividing a coding tree unit into coding units, prediction units, and transform units is shown.

[0074] In inter-coding mode, motion-compensated temporal prediction is used to exploit the redundancy that exists between successive pictures of a video. To do this, in HEVC each PU is assigned exactly one motion vector (MV). Thus, in HEVC, the motion model that relates a PU to its reference block simply consists of translation.

[0075] In at least one version of the Joint Exploration Model (JEM) developed by the JVET (Joint Video Exploration Team) group, the CU is no longer divided into PUs or TUs, and some motion information (prediction information in inter-frame mode) is directly assigned to each CU. In JEM, a CU can be divided into sub-CUs, and motion vectors can be calculated for each sub-CU. In addition, some richer motion models are supported to improve temporal prediction. One of the new motion models introduced in JEM is the affine model, which basically consists of representing the motion vectors in the CU using an affine model. Figure 5 The motion model used is shown. An affine model can be used to generate a motion field within a CU for motion prediction. For example, the affine motion field includes motion vector component values ​​for each position (x, y) within the considered block, as defined by Equation 1:

[0076]

[0077] Equation 1: Affine model for generating the motion field within the CU to be predicted

[0078] Among them, (v 0x ,v 0y ) and (v 1x ,v 1y ) are the so-called control point motion vectors, which are used to generate the affine motion field. (v 0x ,v 0y ) is the upper left corner control point of the motion vector; (v 1x ,v 1y ) is the upper right control point of the motion vector.

[0079] In practice, to keep the complexity reasonable, the motion vectors may be computed for each 4×4 sub-block (sub-CU) of the considered CU, e.g., Figure 6 As shown in . The affine motion vector is calculated by the control point motion vector at the center of each sub-block. Therefore, the temporal prediction of a CU in affine mode involves motion-compensated prediction of each sub-block using its own motion vector.

[0080] In e.g. JEM, affine motion compensation can be used in at least two ways: affine AMVP (Advanced Motion Vector Prediction) or AF_AMVP and affine merging:

[0081] -Affine AMVP (AF_AMVP).

[0082] A CU in AMVP mode (whose size is greater than 8×8) can be predicted in affine AMVP mode. This is signaled in the bitstream by a flag. The affine motion field generation for this inter CU includes determining the control point motion vector (CPMV), which is obtained by the decoder by adding the motion vector difference and the control point motion vector prediction (CPMVP). The CPMVP is a pair of motion vector candidates, which are taken from the lists (A, B, C) and (D, E), respectively, as shown in Figure 1. An example of the motion vector prediction process for an affine AMVP CU is shown. Figure 7 shown.

[0083] -Affine merging.

[0084] In affine merge mode, a CU-level flag indicates whether the merged CU uses affine motion compensation. If so, Figure 8 The ordered set of candidate positions shown (A, B, C, D,

[0085] E) selecting the first available neighboring CU that has been coded in affine mode, where Figure 8 Motion vector prediction candidates in affine merge mode are shown.

[0086] Once the first neighboring CU in affine mode is obtained, the three motion vectors from the top left, top right and bottom left corners of the neighboring CU are retrieved like Fig. 9 Based on these three vectors, the two CPMVs of the upper left and upper right corners of the current CU are derived as shown in Equation 2:

[0087]

[0088] Equation 2: Derivation of the CPMV of the current CU based on the three corner motion vectors of the neighboring CUs

[0089] After obtaining the control point motion vector of the current CU and When , the motion field in the current CU is calculated based on the 4×4 sub-CU through the model of Equation 1.

[0090] Block-based local illumination compensation (LIC) can also be applied in JEM. The goal of the LIC tool is basically to predict the changes in illumination that may occur between a prediction block and its reference block, which is the reference block adopted by motion compensated prediction. For each inter-mode coded CU, a LIC flag is signaled or implicitly derived to indicate the use of LIC. The LIC tool is based on a linear model of illumination changes using a scaling factor a and an offset b called LIC parameters. In codecs such as JEM's, the LIC tool is disabled for inter-mode coded CUs that employ the mentioned affine motion compensation.

[0091] In inter mode, block-based local illumination compensation (LIC) allows correcting block prediction samples obtained via motion compensation by taking into account spatial or temporal local illumination variations. It is based on a model of illumination variations, such as a first-order linear model of illumination variations using a scaling factor a and an offset b. Fig.10 As shown in , LIC parameters (a and b) can be estimated by comparing a set of reconstructed samples (located in a neighborhood Vcur) surrounding a current block ("current block") with a set of reconstructed samples (which may have various sizes depending on the application) in a neighborhood Vref(MV) of a reference block ("reference block") in a reference picture. MV represents a motion vector between the current block and the reference block. Typically, Vcur and Vref(MV) include samples located in an L shape (at the top, left, and upper left) around the current block and the reference block, respectively.

[0092] The LIC parameters may be selected based on any of a variety of schemes. For example, the LIC parameters may be selected based on minimizing local distortion. One scheme for minimizing local distortion may involve minimizing the mean square error (MSE) difference between samples in Vcur and corrected samples in Vref (MV). As an example, the LIC model may be a linear model, i.e., LIC(x)=a*x+b, where parameters a and b may be determined based on minimizing the MSE difference, as shown in Equation 3:

[0093]

[0094] Equation 3: Local distortion used to derive LIC parameters

[0095] Where r and s correspond to the pixel positions in Vcur and Vref (MV), respectively. Another approach may involve using a Min-Max approach. For example, Fig.28 As shown in FIG. , two reference samples (XA, XB) with a minimum value (Min) and a maximum value (Max) and associated reconstructed samples (YA, YB) (e.g., at Fig.28 ), to derive the slope "a", as shown in Equation 4.

[0096]

[0097] Equation 4: Derivation of LIC parameter “a” using the Min-Max method

[0098] A variation may involve also using a second minimum point and a second maximum point (e.g. Fig.28 Points A' and B' in the figure are used to improve robustness.

[0099] In the case of bidirectional prediction, the LIC parameters (a0, b0) and (a1, b1) may be derived independently for Vref0 (MV0) and Vref1 (MV1), respectively. In other embodiments, the derivations may be related or independent. For example, Fig.11 As shown in , in bidirectional prediction, the L shapes in reference 0 and reference 1 are compared with the current block L shape to derive the LIC parameters.

[0100] When encoding a CU in merge mode, the LIC flag may be copied from a neighboring block in a manner similar to motion information copying in merge mode; otherwise, the LIC flag for the CU is signaled to indicate whether LIC is applied. However, in schemes such as JEM's scheme, when a CU uses an affine model (affine AMVP, affine merge), its LIC flag is always set to false.

[0101] At least one embodiment is directed to enabling a block-based local illumination compensation (LIC) tool when employing affine motion prediction for inter-mode coded coding units (CUs).

[0102] At least one embodiment is directed to activating the LIC tool for an inter-mode coded CU that employs an affine model to represent motion vectors.

[0103] At least one embodiment is directed to activating a LIC tool for an inter-mode coded CU, for example, using an affine model, and may include LIC flag decisions for affine AMVP and affine merge and corresponding LIC parameter derivation rules.

[0104] At least one embodiment is directed to how to activate the LIC tool and formulate related rules for inter-coded CUs using affine motion prediction in a way that provides good compression efficiency (rate-distortion performance) with minimal complexity increase in the coding design.

[0105] At least one embodiment includes:

[0106] Determine the LIC flag for inter-mode coded CUs using affine motion prediction. For affine AMVP, an iterative loop for the LIC tool may be applied to determine the LIC flag and signal the LIC flag to the bitstream. Otherwise, for affine merge, the LIC flag may be obtained based on neighboring blocks, e.g., derived from affine control points associated with neighboring blocks, in a manner similar to motion information replication in merge mode. [Encoder / Decoder]

[0107] At least one embodiment includes:

[0108] Based on determining that the LIC flag is true, seek to derive corresponding LIC parameters. One or more features may be involved. For example, using a single motion vector for the first sub-block or any other sub-block (e.g., a center sub-block), or considering multiple motion vectors of sub-blocks in the first row / column, for example, all motion vectors associated with sub-blocks in the first row / column or motion vectors associated with a subset of all motion vectors in the first row and / or first column. As another example, generating a unique pair of LIC parameters for the entire CU. As another example, deriving multiple pairs of LIC parameters. [Encoder / Decoder]

[0109] At least one embodiment improves the correction of block prediction samples based on accounting for illumination changes, for example, by enabling the LIC tool for inter-mode coded CUs using affine motion prediction.

[0110] Various embodiments described herein achieve improvements in coding efficiency.

[0111] At least one embodiment is based on the inventors' recognition that schemes involving disabling LIC in affine models cannot fully exploit the potential performance of block prediction samples obtained through affine motion compensation because illumination variations are not corrected by taking them into account.

[0112] In general, at least one embodiment may be directed to deriving LIC parameters at the decoder side according to one or more embodiments for deriving these parameters at the encoder side without encoding additional bits into the bitstream, thereby not introducing additional burden on the bit rate. However, embodiments may include one or more syntax elements inserted into the signaling.

[0113] In at least one embodiment, an affine MV may be calculated for each 4×4 sub-block of a CU using an affine model. This leads to a set of reconstructed samples in Vref(MV), which may be different for each sub-block with its own motion vector. Fig.12 An example for a 16×16 CU using an affine model is presented in .

[0114] exist Fig.12 An example of neighboring samples located in different L-shapes for a 16×16 CU using an affine model is shown in FIG.

[0115] The current upper left sub-block (“current sub-blk0”) is associated with MV0, and its corresponding reconstructed samples are located in the neighborhood Vref(MV0) of its reference block (“reference blk0”) in the reference picture; at the same time, for the sub-block with MV i The current lower left sub-block ("current sub-blk i ”), the reconstructed sample is located in its reference block in the reference picture (“reference blk i ”) of the neighborhood Vref(MV i ). Vref(MV0) and Vref(MV i ) can generate different L shapes around the relevant reference block. As described in detail below, the LIC parameter derivation can be adapted to the affine model based on various embodiments.

[0116] Fig.13 An example of an embodiment of a method for determining a LIC flag for a CU in an inter slice is shown (e.g., in an encoder). It can be seen that additional inter modes that will employ the LIC tool include affine merge mode and affine AMVP mode. For affine merge mode, the LIC flag is inferred from neighboring blocks. For affine AMVP mode, a performance or quality metric, such as rate-distortion search, may be evaluated for the current CU. Fig.13In the example of , such evaluation includes deriving LIC parameters at step 1340, performing motion compensation at step 1350, applying LIC at step 1360, and calculating a cost (e.g., a rate-distortion cost) at step 1370. Fig.13 As shown in , these steps are repeated in a loop for the LIC flag (i.e., the LIC flag is turned on and off). Thus, each possible LIC flag value is evaluated for the current CU from a rate-distortion perspective in both the affine AMVP mode and the AMVP mode. Once the LIC flag is determined, the flag is signaled in the bitstream, for example, using an existing syntax for signaling LIC usage, thereby avoiding the additional overhead of adding syntax, or by adding syntax if appropriate for a particular environment, situation, or application.

[0117] According to at least one embodiment, for affine AMVP mode, the LIC flag of the current CU may be derived from its candidates and / or from its neighboring MVPs.

[0118] At least one embodiment may use only one motion vector to derive the LIC parameters for the entire CU. For example, the motion vector may be the motion vector MV0 (v 0x ,v 0y ). At least one embodiment may include obtaining a set of reconstructed samples located in the same L-shape around a reference block; and after calculating unique LIC parameters, applying these unique LIC parameters to all sub-blocks in the current block.

[0119] Fig.14 An embodiment of a method for deriving LIC parameters for an affine model using one motion vector is shown. At step 300, the LIC flag is evaluated. If the LIC flag is false, then motion compensation processing occurs at step 303A, followed by step 305, where a performance metric, such as rate-distortion (RD) analysis, is performed. A motion vector (MV) is provided based on the metric analysis results at step 305. If the LIC flag is true at step 300, then processing continues at step 301, where the motion vector MV0 (v) of the first sub-block is calculated by Equation 1. 0x ,v 0y). Step 301 is followed by step 302, in which LIC parameters are derived by minimizing local distortion according to Equation 3 using a set of correction samples in Vref(MV0). Step 302 is followed by block 306, which includes a motion compensation process at 303B (the same motion compensation process as described above with respect to 303A), and then applying LIC at 304 based on the LIC parameters derived at 302. A loop of 306 is performed for each sub-block, wherein the LIC parameters obtained at 302 are employed at 304 to correct illumination variations for each sub-block. After completing the execution of the loop of 306, i.e., after passing steps 303B and 304, processing continues at step 305 as described above for all sub-blocks.

[0120] As described, Fig.14 The example of the embodiment shown in may involve a single motion vector corresponding to the first sub-block. However, in at least one other embodiment, the single motion vector may be calculated based on any sub-block other than the first sub-block (e.g., the center or for a neighboring block, etc.). For the example of a single MV at the "center" of the coding unit, a single MV may be obtained or calculated based on one or more sub-blocks depending on the selected embodiment. Fig.25 , Fig.26 and Fig. 27 A specific example of an embodiment for obtaining a single MV at the center of a CU is illustrated in .

[0121] For example, in Fig.25 , the MV associated with the subblock designated by the dashed circle within the subblock can be the "center" or central MV, which is based on selecting the subblock that includes the point at the center of the coding unit, that is, the point (W / 2, H / 2), where W and H are the width and height of the coding unit, respectively. As another example, the MV at the "center" can be obtained by combining multiple MVs near the center, that is, the point (W / 2, H / 2). As an example, Fig.26 The embodiment shown in combines the MVs of four sub-blocks around the point (W / 2, H / 2) which are Fig.26 These four MVs are labeled MV1, MV2, MV3, and MV4 in the image, and are averaged to provide a single motion vector that can be considered as the center or central MV:

[0122]

[0123] Fig. 27 Another example is shown, where CPMV (Control Point Motion Vector) MV (v 0x ,v 0y) is calculated and applied by the affine model at the point (W / 2, H / 2), thereby obtaining the MV at the "center" (such as indicated by the thick arrow), where

[0124]

[0125] The embodiments described above may address different L-shapes around the reference block, which may be generated because each sub-block has its own affine motion vector. However, the LIC parameters calculated by only one motion vector may not be optimal for all sub-blocks because the illumination change between each sub-block and its reference block may be different. In general, at least one embodiment derives the LIC parameters for the entire CU by considering multiple motion vectors as a reference. More specifically, instead of using a complete L-shape around the reference block, a "quasi-L-shape" generated by several possibly unconnected tiles is used. The quasi-L-shape can be generated because multiple motion vectors may refer to corresponding reconstructed samples that do not form a continuous L-shaped data configuration.

[0126] For example, Fig.15 As shown in , for the current upper left sub-block ("current sub-blk0"), the corresponding reconstructed samples in Vref(MV0) above the reference block MV0 are used as the upper left corner slice of the "L-like shape". The sub-blocks located in the first row of the CU (corresponding to the reference blocks MV0, MV1, MV2 and MV3) thus generate the "L-like shape" top slices (Vref(MV0), Vref(MV1), Vref(MV2), Vref(MV3)). In addition, the "L-like shape" left slice is formed by the sub-blocks in the first column (corresponding to the reference blocks MV0, MV4 and MV5) by using the reconstructed samples on the left side of the reference blocks of these sub-blocks. An additional "Vref(MV0)" is formed on the left side of MV0 in the reference. Note that Vref(MV5) is shown as a double block because multiple sub-blocks in the picture have the same motion vector MV5. Using this "L-like shape", the LIC parameters can be derived and then applied to the entire CU, for example.

[0127] For example, a scheme to minimize local distortion using multiple motion vectors to select LIC parameters can be based on minimizing the MSE difference in a manner similar to that discussed above with respect to Equation 3, which is modified for the multiple motion vector case in Equation 5:

[0128]

[0129] Equation 5: Proposed local distortion for deriving LIC parameters using multiple MVs

[0130] Among them, r still corresponds to the L-shaped pixel position in Vcur; s0 corresponds to the pixel position in Vref(MV0), and s1 corresponds to the pixel position in Vref(MV1). j Always corresponds to Vref(MV i ) until all patches forming the “L-like shape” are traversed. As discussed above with respect to Equation 4 for a single motion vector, another approach to obtaining LIC model parameters for multiple motion vectors may involve using a Min-Max method.

[0131] Fig.16 At least one embodiment of a method for deriving LIC parameters using multiple motion vectors is shown. At step 401, motion vectors for sub-blocks in the first row and first column are generated via Equation 1 by looping over the sub-blocks in the first row and first column. At step 402, when the LIC flag is true, LIC parameters may be derived by minimizing local distortion using Equation 4 using an "L-like shape" around a reference block. The processing at steps 403 and 404 is performed as described above with respect to Fig.14 The process continues in a similar manner as described in steps 303 and 304 .

[0132] In at least one embodiment, to reduce complexity, the "L-like shape" can also be generated using only two motion vectors (e.g., top and left, respectively). The first motion vector is from the top row, e.g., the sub-block in the middle of the first row, and the second motion vector is from the first column, e.g., the sub-block in the middle of the first column.

[0133] In at least one embodiment, a "L-like shape" may be generated using a subset of multiple motion vectors associated with the first row and / or first column of subblocks of the coding unit. For example, the L-like shape may be formed based on reconstructed samples obtained using motion vectors associated with one or more of the subset of subblocks in the first row of subblocks or the subset of subblocks in the first column of subblocks. That is, the first motion vector set may include a motion vector associated with each of the subblocks included in the first row of subblocks in the coding unit, and the second motion vector set may include a motion vector associated with each of the subblocks included in the first column of subblocks in the coding unit. The L-like shape may be formed based on reconstructed samples, which are based on one or more of the first subset of the first motion vector set or the second subset of the second motion vector set, that is, generated based on the first subset, the second subset, or both the first subset and the second subset. For the examples described above involving sub-blocks located in the middle of the first row of sub-blocks and / or the middle of the first column of sub-blocks, the described first motion vector subset and / or second motion vector subset may each include a motion vector associated with the corresponding sub-block located in the middle of the first row and / or the first column.

[0134] In at least one embodiment described above, only one pair of LIC parameters (a and b) is derived and used for the entire CU. In at least one other embodiment, the LIC tool for the affine model may involve multiple LIC parameter sets, for example, multiple pairs of LIC parameters for the linear LIC model may be generated for use with the affine motion model to more accurately correct the predicted samples. Since the LIC parameters can be derived in the same way at the decoder side, adding LIC parameter sets does not require adding syntax bits to be encoded in the bitstream, which means that there is no additional burden on the bit rate.

[0135] As an example, Fig.17 Another embodiment is shown, in which 1) several sub-blocks can be grouped into a larger sub-block (referred to herein as a "LIC group"); 2) a CU can be divided into multiple LIC groups, for example, into four LIC groups (top left, top right, bottom left, bottom right); 3) a pair of LIC parameters associated with each LIC group can be derived; 4) during motion compensation, the illumination changes of samples in the sub-block are corrected using the corresponding LIC parameters of the LIC group to which the sub-block belongs.

[0136] One or more embodiments described herein may also be applied when calculating a pair of LIC parameters for each LIC group. In an embodiment, an L-shape (Vcur_tl / Vcur_tr / Vcur_bl / Vcur_br) around the current LIC group is applied instead of an L-shape (Vcur) around the current CU. In an embodiment, a CU may be divided into more or less than 4 LIC groups. In an embodiment, a CU may be divided into different numbers of LIC groups, depending on the size of the CU. For example, if the size of the CU is 16×16 or 32×32, 4 LIC groups are generated; and a 64×64 CU may be divided into 8 LIC groups.

[0137] Fig.18 Another example of an embodiment of a portion of an encoder is shown. Fig.18 In the embodiment of the present invention, video information (such as video data including a picture portion) is processed based on an affine motion model at step 1810 to generate motion compensation information. At step 1820, a local illumination compensation (LIC) model is obtained, for example, parameters of a linear model are derived according to one or more aspects or embodiments described herein. Then, at step 1830, the video information is encoded based on the motion compensation information and the LIC model to generate encoded video information.

[0138] Fig.19 An example of an embodiment of a portion of a decoder according to one or more aspects of the present disclosure is shown in FIG. Fig.19 In the embodiment of the present invention, the encoded video information (such as video data including the encoded picture portion) is processed based on the affine motion model to generate motion compensation information at step 1910. At step 1920, a local illumination compensation (LIC) model is obtained, for example, parameters of a linear model are derived according to one or more aspects or embodiments described herein. Then, at step 1930, the video information is decoded based on the motion compensation information and the LIC model to generate a decoded picture portion.

[0139] Fig. 20 , Fig.21 and Fig. 22 An additional example of an embodiment of a portion of a decoder is shown. Fig. 20, motion compensation processing for the inter-mode encoded current CU occurs at step 2010. Step 2020 determines whether the inter mode is a merge mode. If so, then at step 2030, the LIC flag is inferred as described herein. If not, then at step 2080, the LIC flag is decoded. Both steps 2020 and 2080 are followed by step 2040, in which the state of the LIC flag is tested. If the LIC flag is false, then LIC is not applied, and processing with respect to LIC ends at step 2070. If it is determined at step 2040 that the LIC flag is true, then at step 2050, LIC parameters are derived. At step 2060, LIC is applied to the entire current CU based on those parameters, after which processing ends at step 2070.

[0140] exist Fig.21 In the step 2110, the LIC flag of the current CU is tested. If it is false, then the LIC processing is disabled or invalid for the current CU, and the motion compensation processing occurs at step 2170, and then the LIC related processing ends at step 2160. If the LIC flag is true at step 2110, the motion vector corresponding to one sub-block is calculated, determined, or obtained, for example, MV0 corresponding to the first sub-block. Next, by based on the motion vector and the associated reference block (e.g., V ref LIC parameters are derived from the MPEG-4 A / D scalar motion vector (MV0) to obtain the LIC model at step 2130. Then, motion compensation is applied at step 2140 and LIC is applied at step 2150. Steps 2140 and 2150 are repeated as a loop for all sub-blocks, thereby applying the LIC model to all sub-blocks based on the parameters determined from the single motion vector. After completing the loop for all sub-blocks, the process ends at step 2160.

[0141] exist Fig. 22In step 2210, the LIC flag of the current CU is tested. If false, the LIC process is disabled or invalid for the current CU, and the motion compensation process occurs at step 2270, and then the LIC related process ends at step 2260. If the LIC flag is true at step 2210, a plurality of motion vectors corresponding to a plurality of sub-blocks (e.g., a subset of sub-blocks in the first row and a subset of sub-blocks in the first column) are calculated, determined, or obtained at step 2220, which is performed by looping over the sub-blocks contained in the one or more subsets of sub-blocks. Step 2220 is followed by step 2230, in which LIC parameters are derived based on a plurality of reference blocks associated with the plurality of motion vectors. As described above, the plurality of reference blocks may have a data configuration represented as a quasi-L shape. Then, the motion compensation process occurs at step 2240, and the LIC process is applied at step 2250, in which the LIC is based on the LIC parameters determined at step 2230. Steps 2240 and 2250 are repeated as a loop for all sub-blocks, thereby applying the LIC model to all sub-blocks. After completing the loop for all sub-blocks, the process ends at step 2260.

[0142] When the LIC tool is activated, one or more embodiments of deriving the proposed LIC parameters for the affine model as described herein may also be performed for other sub-CU based motion vector predictions (i.e., “sub-block based temporal merging candidates”: optional temporal motion vector prediction (ATMVP) and spatio-temporal motion vector prediction (STMVP) and sub-block based temporal motion vector prediction (SbTMVP)).

[0143] In addition, although various aspects and embodiments have been described with respect to video information processing for inter-frame mode or inter-frame coding and linear models associated with LIC, one or more aspects, embodiments and features may also be applicable to intra-frame mode or intra-frame coding. For example, for a cross-component linear model (CCLM) in intra-frame coding, a luminance sample is used to predict the corresponding chrominance sample based on a linear model, and the parameters of the linear model may be derived or obtained according to one or more aspects described herein with respect to LIC.

[0144] Furthermore, if there are several sub-blocks within a CU, each sub-block may have a different corresponding reference sub-block. Therefore, one or more of the described embodiments may also be applied to this case.

[0145] This can be done by, for example Figure 3 These embodiments may be implemented by computer software implemented by the processor 1010 of the system 1000 in the embodiment of the present invention, or by hardware, or by a combination of hardware and software. As a non-limiting example, these embodiments may be implemented by one or more integrated circuits. Figure 3 The memory 1020 in the example of the system 1000 shown in FIG. 1 can be of any type suitable for the technical environment and can be implemented using any appropriate data storage technology, such as optical storage devices, magnetic storage devices, semiconductor-based storage devices, fixed memory, and removable memory as non-limiting examples. The processor 1010 can be of any type suitable for the technical environment and can include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor-based multi-core architecture as non-limiting examples.

[0146] The embodiments and aspects described herein may be implemented, for example, in a method or process, device, software program, data stream or signal. Even if only discussed in the context of a single form of embodiment (e.g., discussed only as a method), the embodiments of the features discussed may be implemented in other forms (e.g., devices or programs). For example, a device may be implemented in appropriate hardware, software, and firmware. For example, a method may be implemented in a device, such as, for example, in a processor (which generally refers to a processing device), which includes, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as, for example, a computer, a cellular phone, a portable / personal digital assistant (PDA), and other devices that facilitate information communication between end users.

[0147] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and variations thereof appearing in various places throughout this document are not necessarily all referring to the same embodiment.

[0148] Additionally, this document may refer to “determining” various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0149] Additionally, this document may refer to “accessing” various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0150] Additionally, this document may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing the information or retrieving the information (e.g., from a memory). Additionally, during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is often involved in one way or another.

[0151] It will be apparent to one of ordinary skill in the art that embodiments may generate various signals formatted to carry information, which may be stored or transmitted, for example. The information may include, for example, instructions for executing a method or data generated by one of the described embodiments. For example, a signal may be formatted to carry a bit stream of the described embodiments. For example, such a signal may be formatted as an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium, for example, a non-transitory computer-readable medium.

[0152] Generally speaking, at least one embodiment may relate to a method of encoding video information, comprising: processing the video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model.

[0153] At least one embodiment may relate to a device for encoding video information, comprising one or more processors, wherein the one or more processors are configured to: process the video information based on an affine motion model to generate motion compensation information; obtain a local illumination compensation model; and encode the video information based on the motion compensation information and the local illumination compensation model.

[0154] At least one embodiment may relate to a method of decoding video information, comprising: processing the video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and decoding the video information based on the motion compensation information and the local illumination compensation model.

[0155] At least one embodiment may relate to a device for decoding video information, comprising one or more processors, wherein the one or more processors are configured to: process the video information based on an affine motion model to generate motion compensation information; obtain a local illumination compensation model; and encode the video information based on the motion compensation information and the local illumination compensation model.

[0156] At least one embodiment may relate to a method or apparatus as described herein, wherein obtaining a local illumination compensation model comprises determining at least one model parameter of a linear model of illumination variations in video information based on at least one motion vector contained in motion compensation information.

[0157] An example of an embodiment of a method may include: processing video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination variation in the video information based on a plurality of motion vectors contained in the motion compensation information; and the video information includes a coding unit having a plurality of sub-blocks, the plurality of sub-blocks including a first row of sub-blocks and a first column of sub-blocks; and the plurality of motion vectors include a motion vector group associated with a corresponding sub-block in each sub-block contained in the first row of sub-blocks and each sub-block contained in the first column of sub-blocks; and determining the at least one model parameter includes obtaining a set of reconstructed samples forming an L-like shape based on the motion vector group, and evaluating local distortion based on the set of reconstructed samples.

[0158] Another example of an embodiment of a method may include: processing video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model; wherein the video information includes a coding unit having a plurality of sub-blocks, the plurality of sub-blocks including a first row of sub-blocks and a first column of sub-blocks; and the motion compensation information includes a first set of motion vectors associated with a corresponding sub-block in each sub-block included in the first row of sub-blocks and a second set of motion vectors associated with a corresponding sub-block in each sub-block included in the first column of sub-blocks; and obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination variation in the video information based on one or more of a first subset of the first set of motion vectors or a second subset of the second set of motion vectors; and determining the at least one model parameter includes obtaining a set of reconstructed samples forming an L-like shape based on one or more of the first subset of the first set of motion vectors or a second subset of the second set of motion vectors, and evaluating local distortion based on the set of reconstructed samples.

[0159] Another example of an embodiment of a method may include: processing video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector contained in the motion compensation information; and the video information includes a coding unit having a plurality of sub-blocks; and the first motion vector is associated with a first sub-block of the plurality of sub-blocks located in the upper left corner of the coding unit; and determining the at least one model parameter includes obtaining a set of reconstructed samples forming an L shape based on the first motion vector, and evaluating local distortion based on the set of reconstructed samples.

[0160] Another example of an embodiment of a method may include: processing video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector contained in the motion compensation information; and the first motion vector is associated with a center of a coding unit; and determining the at least one model parameter includes obtaining a set of reconstructed samples forming an L shape based on the first motion vector, and evaluating local distortion based on the set of reconstructed samples.

[0161] Another example of an embodiment of a method may include: processing video information based on a coding unit including multiple sub-blocks, thereby generating motion compensation information including multiple motion vectors associated with corresponding sub-blocks in the multiple sub-blocks; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination changes in the video information based on at least one of the multiple motion vectors; and the multiple sub-blocks included in the coding unit include a first row of sub-blocks and a first column of sub-blocks; and the multiple motion vectors include a motion vector group associated with corresponding sub-blocks in each sub-block included in the first row of sub-blocks and each sub-block included in the first column of sub-blocks; and determining the at least one model parameter includes obtaining a group of reconstructed samples forming an L-like shape based on the motion vector group, and evaluating local distortion based on the group of reconstructed samples.

[0162] Another example of an embodiment of a method may include: processing video information based on a coding unit including multiple sub-blocks to generate motion compensation information; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model; wherein the multiple sub-blocks included in the coding unit include a first row of sub-blocks and a first column of sub-blocks; and the motion compensation information includes a first group of motion vectors associated with a corresponding sub-block in each sub-block included in the first row of sub-blocks and a second group of motion vectors associated with a corresponding sub-block in each sub-block included in the first column of sub-blocks; and obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination variation in the video information based on one or more of a first subset of the first group of motion vectors and a second subset of the second group of motion vectors; and determining the at least one model parameter includes obtaining a group of reconstructed samples forming an L-like shape based on one or more of the first subset of the first group of motion vectors or the second subset of the second group of motion vectors, and evaluating local distortion based on the group of reconstructed samples.

[0163] Another example of an embodiment of a method may include: processing video information based on a coding unit including multiple sub-blocks to generate motion compensation information including multiple motion vectors associated with corresponding sub-blocks among the multiple sub-blocks; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector included in the motion compensation information; and the first motion vector is associated with a first sub-block among the multiple sub-blocks located in the upper left corner of the coding unit; and determining the at least one model parameter includes obtaining a set of reconstructed samples forming an L shape based on the first motion vector, and evaluating local distortion based on the set of reconstructed samples.

[0164] Another example of an embodiment of a method may include: processing video information based on a coding unit including multiple sub-blocks to generate motion compensation information including multiple motion vectors associated with corresponding sub-blocks in the multiple sub-blocks; obtaining a local illumination compensation model; and encoding the video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector included in the motion compensation information; and the first motion vector is associated with a center of the coding unit; and determining the at least one model parameter includes obtaining a set of reconstructed samples forming an L shape based on the first motion vector, and evaluating local distortion based on the set of reconstructed samples.

[0165] An example of an embodiment of an apparatus may include one or more processors, wherein the one or more processors are configured to: process video information based on an affine motion model to generate motion compensation information; obtain a local illumination compensation model; and encode the video information based on the motion compensation information and the local illumination compensation model; wherein, in order to obtain the local illumination compensation model, the one or more processors are further configured to determine at least one model parameter of a linear model of illumination variation in the video information based on multiple motion vectors contained in the motion compensation information; wherein the video information includes a coding unit having multiple sub-blocks, the multiple sub-blocks including a first row of sub-blocks and a first column of sub-blocks; and the multiple motion vectors include a motion vector group associated with a corresponding sub-block in each sub-block included in the first row of sub-blocks and each sub-block included in the first column of sub-blocks; and in order to determine the at least one model parameter, the one or more processors are further configured to obtain a group of reconstructed samples forming an L-like shape based on the motion vector group, and to evaluate local distortion based on the group of reconstructed samples.

[0166] Another example of an embodiment of an apparatus may include one or more processors, wherein the one or more processors are configured to: process video information based on an affine motion model to generate motion compensation information; obtain a local illumination compensation model; and encode the video information based on the motion compensation information and the local illumination compensation model; wherein the video information includes a coding unit having a plurality of sub-blocks, the plurality of sub-blocks including a first row of sub-blocks and a first column of sub-blocks; and the motion compensation information includes a first set of motion vectors associated with a corresponding sub-block in each sub-block included in the first row of sub-blocks and a second set of motion vectors associated with a corresponding sub-block in each sub-block included in the first column of sub-blocks; and in order to obtain the local illumination compensation model, the one or more processors are further configured to determine at least one model parameter of a linear model of illumination variation in the video information based on one or more of a first subset of the first set of motion vectors or a second subset of the second set of motion vectors; and in order to determine the at least one model parameter, the one or more processors are further configured to obtain a set of reconstructed samples forming an L-like shape based on one or more of the first subset of the first set of motion vectors or the second subset of the second set of motion vectors, and to evaluate local distortion based on the set of reconstructed samples.

[0167] Another example of an embodiment of an apparatus may include one or more processors, wherein the one or more processors are configured to: process video information based on an affine motion model to generate motion compensation information; obtain a local illumination compensation model; and encode the video information based on the motion compensation information and the local illumination compensation model; wherein, in order to obtain the local illumination compensation model, the one or more processors are further configured to determine at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector contained in the motion compensation information; and wherein the video information includes a coding unit having multiple sub-blocks; and the first motion vector is associated with a first sub-block of the multiple sub-blocks located in the upper left corner of the coding unit; and in order to determine the at least one model parameter, the one or more processors are further configured to obtain a set of reconstructed samples forming an L shape based on the first motion vector, and to evaluate local distortion based on the set of reconstructed samples.

[0168] Another example of an embodiment of an apparatus may include one or more processors, wherein the one or more processors are configured to: process video information based on an affine motion model to generate motion compensation information; obtain a local illumination compensation model; and encode the video information based on the motion compensation information and the local illumination compensation model; wherein, in order to obtain the local illumination compensation model, the one or more processors are further configured to determine at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector contained in the motion compensation information; wherein the first motion vector is associated with a center of a coding unit; and in order to determine the at least one model parameter, the one or more processors are further configured to obtain a set of reconstructed samples forming an L shape based on the first motion vector, and to evaluate local distortion based on the set of reconstructed samples.

[0169] Another example of an embodiment of an apparatus may include one or more processors, wherein the one or more processors are configured to: process video information based on a coding unit including multiple sub-blocks, thereby generating motion compensation information including multiple motion vectors associated with corresponding sub-blocks in the multiple sub-blocks; obtain a local illumination compensation model; and encode the video information based on the motion compensation information and the local illumination compensation model; wherein, in order to obtain the local illumination compensation model, the one or more processors are configured to determine at least one model parameter of a linear model of illumination variation in the video information based on at least one of the multiple motion vectors; and the multiple sub-blocks included in the coding unit include a first row of sub-blocks and a first column of sub-blocks; and the multiple motion vectors include a motion vector group associated with corresponding sub-blocks in each sub-block included in the first row of sub-blocks and each sub-block included in the first column of sub-blocks; and wherein, in order to determine the at least one model parameter, the one or more processors are further configured to obtain a group of reconstructed samples forming an L-like shape based on the motion vector group, and evaluate local distortion based on the group of reconstructed samples.

[0170] Another example of an embodiment of an apparatus may include one or more processors, wherein the one or more processors are configured to: process video information based on a coding unit including a plurality of subblocks to generate motion compensation information; obtain a local illumination compensation model; and encode the video information based on the motion compensation information and the local illumination compensation model; wherein the plurality of subblocks included in the coding unit include a first row of subblocks and a first column of subblocks; and the motion compensation information includes a first set of motion vectors associated with a corresponding subblock in each subblock included in the first row of subblocks and a second set of motion vectors associated with a corresponding subblock in each subblock included in the first column of subblocks; and in order to obtain the local illumination compensation model, the one or more processors are further configured to determine at least one model parameter of a linear model of illumination variation in the video information based on one or more of a first subset of the first set of motion vectors or a second subset of the second set of motion vectors; and in order to determine the at least one model parameter, the one or more processors are further configured to obtain a set of reconstructed samples forming an L-like shape based on one or more of the first subset of the first set of motion vectors or the second subset of the second set of motion vectors, and evaluate local distortion based on the set of reconstructed samples.

[0171] Another example of an embodiment of an apparatus may include one or more processors, wherein the one or more processors are configured to: process video information based on a coding unit including multiple sub-blocks to generate motion compensation information including multiple motion vectors associated with corresponding sub-blocks of the multiple sub-blocks; obtain a local illumination compensation model; and encode the video information based on the motion compensation information and the local illumination compensation model; wherein, in order to obtain the local illumination compensation model, the one or more processors are further configured to determine at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector contained in the motion compensation information; and the first motion vector is associated with a first sub-block of the multiple sub-blocks located in the upper left corner of the coding unit; and wherein, in order to determine the at least one model parameter, the one or more processors are further configured to obtain a set of reconstructed samples forming an L shape based on the first motion vector, and evaluate local distortion based on the set of reconstructed samples.

[0172] Another example of an embodiment of an apparatus may include one or more processors, wherein the one or more processors are configured to: process video information based on a coding unit including multiple sub-blocks to generate motion compensation information including multiple motion vectors associated with corresponding sub-blocks of the multiple sub-blocks; obtain a local illumination compensation model; and encode the video information based on the motion compensation information and the local illumination compensation model; wherein, in order to obtain the local illumination compensation model, the one or more processors are further configured to determine at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector contained in the motion compensation information; and the first motion vector is associated with a center of the coding unit; and in order to determine the at least one model parameter, the one or more processors are further configured to obtain a set of reconstructed samples forming an L shape based on the first motion vector, and evaluate local distortion based on the set of reconstructed samples.

[0173] An example of an embodiment of a method may include: processing encoded video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and decoding the encoded video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination changes in the video information based on multiple motion vectors contained in the motion compensation information; and the video information includes a coding unit having multiple sub-blocks, the multiple sub-blocks including a first row of sub-blocks and a first column of sub-blocks; and the multiple motion vectors include a motion vector group associated with a corresponding sub-block in each sub-block included in the first row of sub-blocks and each sub-block included in the first column of sub-blocks; and determining the at least one model parameter includes obtaining a group of reconstructed samples forming an L-like shape based on the motion vector group, and evaluating local distortion based on the group of reconstructed samples.

[0174] Another example of an embodiment of a method may include: processing encoded video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and decoding the encoded video information based on the motion compensation information and the local illumination compensation model; wherein the video information includes a coding unit having a plurality of sub-blocks, the plurality of sub-blocks including a first row of sub-blocks and a first column of sub-blocks; and the motion compensation information includes a first set of motion vectors associated with a corresponding sub-block in each sub-block included in the first row of sub-blocks and a second set of motion vectors associated with a corresponding sub-block in each sub-block included in the first column of sub-blocks; and obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination variation in the video information based on one or more of a first subset of the first set of motion vectors or a second subset of the second set of motion vectors; and determining the at least one model parameter includes obtaining a set of reconstructed samples forming an L-like shape based on one or more of the first subset of the first set of motion vectors or a second subset of the second set of motion vectors, and evaluating local distortion based on the set of reconstructed samples.

[0175] Another example of an embodiment of a method may include: processing encoded video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and decoding the encoded video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector contained in the motion compensation information; and the video information includes a coding unit having multiple sub-blocks; and the first motion vector is associated with a first sub-block of the multiple sub-blocks located in the upper left corner of the coding unit; and determining the at least one model parameter includes obtaining a set of reconstructed samples forming an L shape based on the first motion vector, and evaluating local distortion based on the set of reconstructed samples.

[0176] Another example of an embodiment of a method may include: processing encoded video information based on an affine motion model to generate motion compensation information; obtaining a local illumination compensation model; and decoding the encoded video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector contained in the motion compensation information; and the first motion vector is associated with a center of a coding unit; and determining the at least one model parameter includes obtaining a set of reconstructed samples forming an L shape based on the first motion vector, and evaluating local distortion based on the set of reconstructed samples.

[0177] Another example of an embodiment of a method may include: processing encoded video information based on a coding unit including multiple sub-blocks, thereby generating motion compensation information including multiple motion vectors associated with corresponding sub-blocks in the multiple sub-blocks; obtaining a local illumination compensation model; and decoding the encoded video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination changes in the video information based on at least one of the multiple motion vectors; and the multiple sub-blocks included in the coding unit include a first row of sub-blocks and a first column of sub-blocks; and the multiple motion vectors include a motion vector group associated with corresponding sub-blocks in each sub-block included in the first row of sub-blocks and each sub-block included in the first column of sub-blocks; and determining the at least one model parameter includes obtaining a group of reconstructed samples forming an L-like shape based on the motion vector group, and evaluating local distortion based on the group of reconstructed samples.

[0178] Another example of an embodiment of a method may include: processing encoded video information based on a coding unit including multiple sub-blocks to generate motion compensation information; obtaining a local illumination compensation model; and decoding the encoded video information based on the motion compensation information and the local illumination compensation model; wherein the multiple sub-blocks contained in the coding unit include a first row of sub-blocks and a first column of sub-blocks; and the motion compensation information includes a first group of motion vectors associated with a corresponding sub-block in each sub-block included in the first row of sub-blocks and a second group of motion vectors associated with a corresponding sub-block in each sub-block included in the first column of sub-blocks; and obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination variation in the video information based on one or more of a first subset of the first group of motion vectors or a second subset of the second group of motion vectors; and determining the at least one model parameter includes obtaining a group of reconstructed samples forming an L-like shape based on one or more of the first subset of the first group of motion vectors or the second subset of motion vectors, and evaluating local distortion based on the group of reconstructed samples.

[0179] Another example of an embodiment of a method may include: processing encoded video information based on a coding unit including multiple sub-blocks to generate motion compensation information including multiple motion vectors associated with corresponding sub-blocks among the multiple sub-blocks; obtaining a local illumination compensation model; and decoding the encoded video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector included in the motion compensation information; and the first motion vector is associated with a first sub-block among the multiple sub-blocks located in the upper left corner of the coding unit; and determining the at least one model parameter includes obtaining a set of reconstructed samples forming an L shape based on the first motion vector, and evaluating local distortion based on the set of reconstructed samples.

[0180] Another example of an embodiment of a method may include: processing encoded video information based on a coding unit including multiple sub-blocks to generate motion compensation information including multiple motion vectors associated with corresponding sub-blocks in the multiple sub-blocks; obtaining a local illumination compensation model; and decoding the encoded video information based on the motion compensation information and the local illumination compensation model; wherein obtaining the local illumination compensation model includes determining at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector included in the motion compensation information; and the first motion vector is associated with a center of the coding unit; and determining the at least one model parameter includes obtaining a set of reconstructed samples forming an L shape based on the first motion vector, and evaluating local distortion based on the set of reconstructed samples.

[0181] An example of an embodiment of an apparatus may include one or more processors, wherein the one or more processors are configured to: process encoded video information based on an affine motion model to generate motion compensation information; obtain a local illumination compensation model; and decode the encoded video information based on the motion compensation information and the local illumination compensation model; wherein, in order to obtain the local illumination compensation model, the one or more processors are further configured to determine at least one model parameter of a linear model of illumination changes in the video information based on multiple motion vectors contained in the motion compensation information; wherein the video information includes a coding unit having multiple sub-blocks, the multiple sub-blocks including a first row of sub-blocks and a first column of sub-blocks; and the multiple motion vectors include a motion vector group associated with each sub-block included in the first row of sub-blocks and a corresponding sub-block in each sub-block included in the first column of sub-blocks; and in order to determine the at least one model parameter, the one or more processors are further configured to obtain a group of reconstructed samples forming an L-like shape based on the motion vector group, and to evaluate local distortion based on the group of reconstructed samples.

[0182] Another example of an embodiment of an apparatus may include one or more processors, wherein the one or more processors are configured to: process encoded video information based on an affine motion model to generate motion compensation information; obtain a local illumination compensation model; and decode the encoded video information based on the motion compensation information and the local illumination compensation model; wherein the video information includes a coding unit having a plurality of subblocks, the plurality of subblocks including a first row of subblocks and a first column of subblocks; and the motion compensation information includes a first set of motion vectors associated with a corresponding subblock in each subblock included in the first row of subblocks and a second set of motion vectors associated with a corresponding subblock in each subblock included in the first column of subblocks; and to obtain the local illumination compensation model, the one or more processors are further configured to determine at least one model parameter of a linear model of illumination variation in the video information based on one or more of a first subset of the first set of motion vectors or a second subset of the second set of motion vectors; and to determine the at least one model parameter, the one or more processors are further configured to obtain a set of reconstructed samples forming an L-like shape based on one or more of the first subset of the first set of motion vectors or the second subset of the second set of motion vectors, and to evaluate local distortion based on the set of reconstructed samples.

[0183] Another example of an embodiment of an apparatus may include one or more processors, wherein the one or more processors are configured to: process encoded video information based on an affine motion model to generate motion compensation information; obtain a local illumination compensation model; and decode the encoded video information based on the motion compensation information and the local illumination compensation model; wherein, in order to obtain the local illumination compensation model, the one or more processors are further configured to determine at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector contained in the motion compensation information; and wherein the video information includes a coding unit having multiple sub-blocks; and the first motion vector is associated with a first sub-block of the multiple sub-blocks located in the upper left corner of the coding unit; and in order to determine the at least one model parameter, the one or more processors are further configured to obtain a set of reconstructed samples forming an L shape based on the first motion vector, and to evaluate local distortion based on the set of reconstructed samples.

[0184] Another example of an embodiment of an apparatus may include one or more processors, wherein the one or more processors are configured to: process encoded video information based on an affine motion model to generate motion compensation information; obtain a local illumination compensation model; and decode the encoded video information based on the motion compensation information and the local illumination compensation model; wherein, in order to obtain the local illumination compensation model, the one or more processors are further configured to determine at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector contained in the motion compensation information; wherein the first motion vector is associated with a center of a coding unit; and in order to determine the at least one model parameter, the one or more processors are further configured to obtain a set of reconstructed samples forming an L shape based on the first motion vector, and to evaluate local distortion based on the set of reconstructed samples.

[0185] Another example of an embodiment of an apparatus may include one or more processors, wherein the one or more processors are configured to: process encoded video information based on a coding unit including multiple sub-blocks, thereby generating motion compensation information including multiple motion vectors associated with corresponding sub-blocks in the multiple sub-blocks; obtain a local illumination compensation model; and decode the encoded video information based on the motion compensation information and the local illumination compensation model; wherein, in order to obtain the local illumination compensation model, the one or more processors are configured to determine at least one model parameter of a linear model of illumination variation in the video information based on at least one of the multiple motion vectors; and the multiple sub-blocks included in the coding unit include a first row of sub-blocks and a first column of sub-blocks; and the multiple motion vectors include a motion vector group associated with corresponding sub-blocks in each sub-block included in the first row of sub-blocks and each sub-block included in the first column of sub-blocks; and wherein, in order to determine the at least one model parameter, the one or more processors are further configured to obtain a group of reconstructed samples forming an L-like shape based on the motion vector group, and evaluate local distortion based on the group of reconstructed samples.

[0186] Another example of an embodiment of an apparatus may include one or more processors, wherein the one or more processors are configured to: process encoded video information based on a coding unit including a plurality of subblocks to generate motion compensation information; obtain a local illumination compensation model; and decode the encoded video information based on the motion compensation information and the local illumination compensation model; wherein the plurality of subblocks included in the coding unit include a first row of subblocks and a first column of subblocks; and the motion compensation information includes a first set of motion vectors associated with a corresponding subblock in each subblock included in the first row of subblocks and a second set of motion vectors associated with a corresponding subblock in each subblock included in the first column of subblocks; and in order to obtain the local illumination compensation model, the one or more processors are further configured to determine at least one model parameter of a linear model of illumination variation in the video information based on one or more of a first subset of the first set of motion vectors or a second subset of the second set of motion vectors; and in order to determine the at least one model parameter, the one or more processors are further configured to obtain a set of reconstructed samples forming an L-like shape based on one or more of the first subset of the first set of motion vectors or the second subset of the second set of motion vectors, and evaluate local distortion based on the set of reconstructed samples.

[0187] Another example of an embodiment of an apparatus may include one or more processors, wherein the one or more processors are configured to: process encoded video information based on a coding unit including multiple sub-blocks to generate motion compensation information including multiple motion vectors associated with corresponding sub-blocks of the multiple sub-blocks; obtain a local illumination compensation model; and decode the encoded video information based on the motion compensation information and the local illumination compensation model; wherein, in order to obtain the local illumination compensation model, the one or more processors are further configured to determine at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector contained in the motion compensation information; and the first motion vector is associated with a first sub-block of the multiple sub-blocks located in the upper left corner of the coding unit; and wherein, in order to determine the at least one model parameter, the one or more processors are further configured to obtain a set of reconstructed samples forming an L shape based on the first motion vector, and evaluate local distortion based on the set of reconstructed samples.

[0188] Another example of an embodiment of an apparatus may include one or more processors, wherein the one or more processors are configured to: process encoded video information based on a coding unit including multiple sub-blocks to generate motion compensation information including multiple motion vectors associated with corresponding sub-blocks of the multiple sub-blocks; obtain a local illumination compensation model; and decode the encoded video information based on the motion compensation information and the local illumination compensation model; wherein, in order to obtain the local illumination compensation model, the one or more processors are further configured to determine at least one model parameter of a linear model of illumination changes in the video information based on a first motion vector contained in the motion compensation information; and the first motion vector is associated with the center of the coding unit; and in order to determine the at least one model parameter, the one or more processors are further configured to obtain a set of reconstructed samples forming an L shape based on the first motion vector, and evaluate local distortion based on the set of reconstructed samples.

[0189] In a variation of at least one embodiment described herein involving a coding unit having a plurality of sub-blocks including a first row of sub-blocks and a first column of sub-blocks, a first motion vector subset includes a first motion vector corresponding to a sub-block located at a middle position of the first row of sub-blocks, and a second motion vector subset includes a second motion vector corresponding to a sub-block located at a middle position of the first column of sub-blocks.

[0190] In a variation of at least one embodiment described herein involving multiple sub-blocks of a coding unit, the multiple sub-blocks can be partitioned into multiple sub-block groups; and model parameters of a local illumination compensation model can be determined for each of the multiple sub-block groups; and encoding or decoding based on the local illumination compensation model can include processing video information associated with each sub-block group using at least one corresponding model parameter determined for each group.

[0191] In a variation of at least one embodiment described herein involving grouping subblocks, at least one of a first number of subblocks in each subblock group and a second number of groups formed for the coding unit is selected based on a size of the coding unit.

[0192] Variations of at least one of the embodiments described herein that involve encoding video information may include determining a rate-distortion metric associated with applying a local illumination compensation model; and providing a syntax element in the encoded video information having a value based on the rate-distortion metric.

[0193] In variations of at least one embodiment described herein involving at least one parameter of a linear model, the at least one model parameter may include a pair of first and second model parameters corresponding to a scaling factor and an offset.

[0194] Another example of an embodiment may involve a computer program product comprising computer instructions for performing any of the methods described herein when executed by one or more processors.

[0195] Another example of an embodiment may involve a non-transitory computer-readable medium storing executable program instructions to cause a computer to execute the instructions to perform any of the methods described herein.

[0196] Another example of an embodiment may involve a bitstream formatted to contain encoded video information produced by the method described herein.

[0197] Variations of the bitstream embodiments described herein may involve encoded video information comprising: an indicator indicating that the video information is encoded based on a local illumination compensation model and an affine motion model; and picture information encoded based on the local illumination compensation model and the affine motion model.

[0198] Another example of an embodiment may relate to a device comprising: an apparatus as described herein; and at least one of: (i) an antenna configured to receive a signal, the signal including data representing video information, (ii) a band limiter configured to limit the received signal to a frequency band including data representing the video information, and (iii) a display configured to display an image based on the video information.

[0199] At least one embodiment may be directed to a method or apparatus as described herein, wherein a second sub-block of the plurality of sub-blocks is located in the middle of a top row of sub-blocks, and a third sub-block of the plurality of sub-blocks is located in the middle of a left column of sub-blocks.

[0200] At least one embodiment may relate to a method or apparatus as described herein, wherein at least one model parameter of the linear model comprises a pair of first and second model parameters corresponding to a scaling factor and an offset, and processing video information based on the linear model comprises processing the multiple sub-blocks of the current coding unit based on the linear model using the scaling factor and the offset.

[0201] At least one embodiment may relate to a method or apparatus as described herein, wherein at least one model parameter of the linear model comprises a pair of first and second model parameters corresponding to a scaling factor and an offset, and obtaining at least one parameter of the linear model comprises: partitioning the plurality of sub-blocks of the current coding unit into a plurality of sub-block groups; determining a pair of model parameters for each of the plurality of sub-block groups to generate a plurality of pairs of parameters; and processing each of the sub-block groups of video information based on the linear model using a corresponding pair of the plurality of pairs of parameters.

[0202] At least one embodiment may relate to a method or apparatus for processing multiple sub-block groups as described herein, wherein at least one of a first number of sub-blocks in each sub-block group and a second number of sub-groups formed for the current coding unit is selected based on the size of the current coding unit.

[0203] At least one embodiment may be directed to a method or apparatus for encoding video information as described herein and further comprising determining a rate-distortion metric based on applying local illumination compensation; and providing a syntax element having a value in the encoded video information based on the rate-distortion metric.

[0204] At least one embodiment may relate to a bitstream formatted to include encoded video information, wherein the encoded video information includes: an indicator indicating that the video information is encoded based on a local illumination compensation model and an affine motion model; and picture information encoded based on the local illumination compensation model and the affine motion model.

[0205] At least one embodiment may relate to a device comprising an apparatus according to any embodiment described herein and further comprising at least one of the following: (i) an antenna configured to receive a signal, the signal including data representing video information, (ii) a band limiter configured to limit the received signal to a frequency band including data representing the video information, and (iii) a display configured to display an image based on the video information.

[0206] In at least one embodiment, when the LIC tool is activated, the proposed LIC parameters for the affine model can also be derived for other sub-CU based motion vector predictions (i.e., "sub-block based temporal merging candidates": optional temporal motion vector prediction (ATMVP) and spatio-temporal motion vector prediction (STMVP) and sub-block based temporal motion vector prediction (SbTMVP)).

[0207] At least one embodiment is directed to enabling compensation in conjunction with predictive encoding and / or decoding.

[0208] One or more embodiments of deriving the proposed LIC parameters as described herein may be applicable to deriving other parameters, such as parameters for scaling and / or offset and / or selection.

[0209] At least one other embodiment may involve modifying pixel values ​​of a predicted block (the block to which a motion vector points), wherein the modification may be performed through various filters (eg, illumination compensation and / or color compensation as described herein).

[0210] At least one other embodiment may involve predictor blocks based on motion vectors generated by various modes, including predictors determined by intra-coding.

[0211] At least one embodiment is directed to enabling a block-based local illumination compensation (LIC) tool when employing affine motion prediction for inter-mode coded coding units (CUs).

[0212] At least one embodiment is directed to activating the LIC tool for an inter-mode coded CU that employs an affine model to represent motion vectors.

[0213] At least one embodiment is directed to activating a LIC tool for an inter-mode coded CU, for example, using an affine model, and may include LIC flag decisions for affine AMVP and affine merge and corresponding LIC parameter derivation rules.

[0214] At least one embodiment is directed to how to activate the LIC tool and formulate related rules for inter-coded CUs using affine motion prediction in a way that provides good compression efficiency (rate-distortion performance) with minimal complexity increase in the coding design.

[0215] At least one embodiment includes:

[0216] Determine the LIC flag for inter-mode coded CUs using affine motion prediction. For affine AMVP, an iterative loop for the LIC tool may be applied to determine the LIC flag and signal the LIC flag to the bitstream. Otherwise, for affine merge, the LIC flag may be copied from a neighboring block in a similar manner to motion information copying in merge mode, e.g., the LIC flag is determined based on at least one affine control point associated with the neighboring block.

[0217] [Encoder / Decoder]

[0218] At least one embodiment includes:

[0219] Based on the determination that the LIC flag is true, formulate rules for deriving the corresponding LIC parameters. This usually involves several aspects. For example, using a single motion vector for the first sub-block, or considering the first row

[0220] / multiple motion vectors of sub-blocks in a column. As another example, a unique pair of LIC parameters is generated for the entire CU. As another example, multiple pairs of LIC parameters are derived. [Encoder / Decoder]

[0221] At least one embodiment improves the correction of block prediction samples based on accounting for illumination changes, for example, by enabling the LIC tool for inter-mode coded CUs using affine motion prediction.

[0222] Various embodiments described herein provide the advantage of increased coding efficiency.

[0223] At least one embodiment is based on the inventors' recognition that schemes involving disabling LIC in affine models cannot fully exploit the potential performance of block prediction samples obtained through affine motion compensation because illumination variations are not corrected by taking them into account.

[0224] In at least one embodiment, LIC parameters may be derived at the decoder side according to one or more embodiments for deriving these parameters at the encoder side without encoding additional bits into the bitstream, thereby not introducing additional burden on the bit rate.

[0225] At least one embodiment may involve inserting syntax elements in signaling that enable the decoder to derive parameters, such as LIC parameters, based on embodiments used at the encoder side.

[0226] In at least one embodiment, the method applied at the decoder may be selected based on one or more syntax elements inserted in the signaling.

[0227] In at least one embodiment, a bitstream or signal includes one or more of the described syntax elements, or variations thereof.

[0228] At least one embodiment is directed to creating and / or sending and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0229] A TV, set-top box, cellular phone, tablet computer, or other electronic device may implement any of the described embodiments.

[0230] A TV, set-top box, cellular phone, tablet computer, or other electronic device may implement any of the described embodiments and may display (eg, using a monitor, screen, or other type of display) the resulting images.

[0231] A TV, set-top box, cellular phone, tablet computer, or other electronic device may tune to a channel (eg, using a tuner) to receive a signal including an encoded image and process the image according to any of the described embodiments.

[0232] A TV, set-top box, cellular phone, tablet computer, or other electronic device may receive over the air (eg, using an antenna) a signal including an encoded image and process the image according to any of the described embodiments.

[0233] An embodiment may include a computer program product with a program code which, when executed, performs a method according to any of the embodiments described herein.

[0234] An embodiment may include a computer readable medium having program code stored thereon, which when executed performs a method according to any of the embodiments described herein.

[0235] Various other generalized and specificized inventions and claims are also supported and contemplated throughout this disclosure.

Claims

1. A video decoding device, comprising: The processor is configured as: obtaining an affine motion model associated with the block, the affine motion model comprising a plurality of control point motion vectors; obtaining local illumination compensation information associated with the block based on a plurality of control point motion vectors of an affine motion model associated with the block; and The block is decoded based on the affine motion model and the local illumination compensation information.

2. The video decoding device according to claim 1, wherein the processor is further configured to: obtaining a prediction of the block based on the affine motion model; and The prediction of the block is adjusted based on the local illumination compensation information, wherein the block is decoded based on the adjusted prediction block.

3. The video decoding device according to claim 1, wherein the block comprises a plurality of sub-blocks, and The processor is further configured to: Obtaining a plurality of prediction sub-blocks based on the affine motion model; and The plurality of predicted sub-blocks are adjusted based on the local illumination compensation information, wherein the block is decoded based on the plurality of adjusted predicted sub-blocks.

4. The video decoding device according to claim 1, wherein the block comprises a plurality of sub-blocks, and The processor is further configured to: determining, based on the plurality of control point motion vectors, a first motion vector associated with a first sub-block of the plurality of sub-blocks; deriving a set of local illumination compensation parameters based on the first motion vector and a template of the block; Obtaining a plurality of prediction sub-blocks based on the affine motion model; as well as The local illumination compensation parameter set is applied to the plurality of prediction sub-blocks.

5. The video decoding device according to claim 4, The processor configured to derive a set of local illumination compensation parameters based on the first motion vector and the template of the block is further configured to: determining, based on the first motion vector, at least one sample associated with a reference block; and At least one sample associated with the reference block is compared to a template of the block.

6. The video decoding device according to claim 5, Wherein the processor configured to compare at least one sample associated with the reference block with a template of the block is further configured to compare at least one sample associated with the reference block with one or more samples included in the template.

7. The video decoding device according to claim 1, wherein the block comprises a plurality of sub-blocks, and The processor is further configured to: determining, based on the plurality of control point motion vectors, a plurality of motion vectors associated with respective sub-blocks of the plurality of sub-blocks; obtaining a plurality of template samples based on a plurality of motion vectors associated with each sub-block; deriving a set of local illumination compensation parameters based on the plurality of template samples and the template of the block; Obtaining a plurality of prediction sub-blocks based on the affine motion model; as well as The local illumination compensation parameter set is applied to the plurality of prediction sub-blocks.

8. The video decoding device according to claim 7, The processor configured to obtain a plurality of template samples based on a plurality of motion vectors associated with each sub-block is further configured to: At least one of the plurality of template samples is determined for each of the corresponding sub-blocks in the respective sub-blocks based on the plurality of motion vectors. 9 . The video decoding apparatus according to claim 8 , wherein the plurality of template samples form an L-like shape.

10. The video decoding device according to claim 8, The processor configured to derive a set of local illumination compensation parameters based on the plurality of template samples and the template of the block is further configured to: Each of the plurality of template samples is compared to a corresponding sample contained in the template and associated with a respective sub-block.

11. A method for video decoding, comprising: obtaining an affine motion model associated with the block, the affine motion model comprising a plurality of control point motion vectors; obtaining local illumination compensation information associated with the block based on a plurality of control point motion vectors of an affine motion model associated with the block; and The block is decoded based on the affine motion model and the local illumination compensation information.

12. The method according to claim 11, further comprising: obtaining a prediction of the block based on the affine motion model; as well as The prediction of the block is adjusted based on the local illumination compensation information, wherein the block is decoded based on the adjusted prediction block.

13. The method according to claim 11, wherein the block comprises a plurality of sub-blocks, and The method further comprises: Obtaining a plurality of prediction sub-blocks based on the affine motion model; as well as The plurality of predicted sub-blocks are adjusted based on the local illumination compensation information, wherein the block is decoded based on the plurality of adjusted predicted sub-blocks.

14. The method according to claim 11, wherein the block comprises a plurality of sub-blocks, and The method further comprises: determining, based on the plurality of control point motion vectors, a first motion vector associated with a first sub-block of the plurality of sub-blocks; deriving a set of local illumination compensation parameters based on the first motion vector and a template of the block; Obtaining a plurality of prediction sub-blocks based on the affine motion model; as well as The local illumination compensation parameter set is applied to the plurality of prediction sub-blocks.

15. The method according to claim 14, The derivation of a local illumination compensation parameter set based on the first motion vector and the template of the block includes: determining, based on the first motion vector, at least one sample associated with a reference block; as well as At least one sample associated with the reference block is compared to a template of the block.

16. The method according to claim 15, Wherein comparing at least one sample associated with the reference block with a template of the block comprises comparing at least one sample associated with the reference block with one or more samples included in the template.

17. The method according to claim 11, wherein the block comprises a plurality of sub-blocks, and The method further comprises: determining, based on the plurality of control point motion vectors, a plurality of motion vectors associated with respective sub-blocks of the plurality of sub-blocks; obtaining a plurality of template samples based on a plurality of motion vectors associated with each sub-block; deriving a set of local illumination compensation parameters based on the plurality of template samples and the template of the block; Obtaining a plurality of prediction sub-blocks based on the affine motion model; as well as The local illumination compensation parameter set is applied to the plurality of prediction sub-blocks.

18. The method according to claim 17, The step of obtaining a plurality of template samples based on a plurality of motion vectors associated with each sub-block comprises: At least one of the plurality of template samples is determined for each of the corresponding sub-blocks in the respective sub-blocks based on the plurality of motion vectors. The method of claim 18 , wherein the plurality of template samples form an L-like shape.

20. A method for generating a local illumination compensation (LIC) mark, comprising: Grouping the plurality of sub-blocks into a sub-block group; Derive a pair of local illumination compensation LIC parameters; associating the pair of LIC parameters with the sub-chunk group; as well as Based on the LIC parameters associated with the sub-block group, illumination for the sub-blocks in the sub-block group is corrected.

Images