Illumination compensation for video encoding and decoding

Abstract

The present invention provides a video codec, which may involve: determining a motion compensation mode associated with encoding a first block of picture information, the motion compensation mode including a sub-block-based motion compensation prediction mode or a non-rectangular block partition mode; determining that a second block of picture information spatially adjacent to the first block is to be encoded or decoded based on illumination compensation; and encoding or decoding the first block based on the motion compensation mode and the illumination compensation.

Description

Technical Field

[0001] The present disclosure relates to video encoding and decoding. Background Art

[0002] To achieve high compression efficiency, image and video coding schemes, such as those defined by the HEVC (High Efficiency Video Coding) standard, typically employ prediction and transform coding to exploit spatial and temporal redundancies in video content. Generally, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlations, followed by transforming, quantizing, and entropy encoding the difference between the original block and the predicted block (typically expressed as a prediction error or prediction residual). To reconstruct the video, the compressed data is decoded using the inverse processes of prediction, transforming, quantizing, and entropy coding. New additions to video compression technology include various versions of reference software for Versatile Video Coding (VVC) and / or documentation, designated the Joint Exploration Model (JEM) developed by the Joint Video Exploration Team (JVET). Efforts such as the JEM aim to further improve existing standards such as HEVC. One area of improvement may involve a suite of tools available for video compression. For example, the proposed Local Illumination Compensation (LIC) tool can provide compensation for adverse effects that may be introduced by illumination variations during encoding and decoding. Summary of the Invention

[0003] Generally speaking, an example of an embodiment may be directed to a method comprising: determining a motion compensation mode for encoding a first block of picture information, the motion compensation mode comprising a sub-block-based motion compensation prediction mode or a non-rectangular block partitioning mode; determining that a second block of picture information spatially adjacent to the first block is to be decoded based on illumination compensation; and decoding the first block based on the motion compensation mode and the illumination compensation.

[0004] In general, another example of an embodiment may involve a method comprising: determining a motion compensation mode for encoding a first block of picture information, the motion compensation mode comprising a sub-block-based motion compensation prediction mode or a non-rectangular block partitioning mode; determining that a second block of picture information spatially adjacent to the first block is encoded based on illumination compensation; and encoding the first block based on the motion compensation mode and the illumination compensation.

[0005] In general, another example of an embodiment may involve an apparatus comprising one or more processors configured to determine a motion compensation mode for encoding a first block of picture information, the motion compensation mode comprising a sub-block-based motion compensation prediction mode or a non-rectangular block partitioning mode; determine a second block of picture information spatially adjacent to the first block to be decoded based on illumination compensation; and decode the first block based on the motion compensation mode and the illumination compensation.

[0006] In general, another example of an embodiment may involve an apparatus comprising one or more processors configured to determine a motion compensation mode for encoding a first block of picture information, the motion compensation mode comprising a sub-block-based motion compensation prediction mode or a non-rectangular block partitioning mode; determine that a second block of picture information spatially adjacent to the first block is to be encoded based on illumination compensation; and encode the first block based on the motion compensation mode and the illumination compensation.

[0007] In general, another example of an embodiment may involve a bitstream formatted to include encoded picture information, wherein the encoded video data is encoded by processing the picture information based on any one or more examples of the examples of embodiments of the method according to the present disclosure.

[0008] In general, one or more other examples of the embodiments may also provide a computer-readable storage medium, such as a non-volatile computer-readable storage medium, on which instructions for encoding or decoding picture information such as video data according to the method or apparatus described herein are stored. One or more embodiments may also provide a computer-readable storage medium on which a bitstream generated according to the method or apparatus described herein is stored. One or more embodiments may also provide a method and apparatus for transmitting or receiving a bitstream generated according to the method or apparatus described herein.

[0009] As explained below, various modifications and implementations are contemplated that may provide improvements to video encoding and / or decoding systems, including but not limited to one or more of increased compression efficiency and / or encoding efficiency and / or processing efficiency and / or reduced complexity.

[0010] The above presents a simplified summary of the subject matter in order to provide a basic understanding of some aspects of the present disclosure. This summary is not a comprehensive overview of the subject matter. It is not intended to identify key / important elements of the embodiments or to delineate the scope of the subject matter. Its sole purpose is to present some concepts of the subject matter in a simplified form as a prelude to the more detailed description provided below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] By attaching Figure 1 The present disclosure may be better understood with consideration of the following detailed description, wherein:

[0012] Figure 1 A block diagram illustrating an example of an implementation of a video encoder;

[0013] Figure 2 A block diagram illustrating an example of an implementation of a video decoder;

[0014] Figure 3 A block diagram illustrating an example of an embodiment of a system that provides video encoding and / or decoding;

[0015] Figure 4 An example of a coding tree unit (CTU) and coding tree concept (such as, for example, in HEVC) is shown;

[0016] FIG5 shows several examples of determining parameters of a local illumination compensation (LIC) model, such as the LIC model used in the Joint Exploration Model (JEM);

[0017] Figure 6 An example of the derivation of LIC parameters for bi-prediction is shown;

[0018] Figure 7 Another exemplary derivation of LIC parameters for bi-prediction is shown;

[0019] Figure 8 shows an example of the positions of left and top reference samples for a rectangular coding unit (CU);

[0020] Figure 9 An example of LIC parameter derivation involving restricted access to certain samples (e.g., intra samples that are unavailable due to standards such as pipeline design) is shown;

[0021] Figure 10 An example of an SbTMVP process (e.g., in Versatile Video Coding (VVC)) is shown;

[0022] Figure 11 An example of partitioning a CU into non-rectangular parts (e.g., prediction units (PUs) and associated diagonal weights between PUs) is shown;

[0023] Figure 12 Two examples of triangular CU partitions (left) and associated PU pairs (right) are shown;

[0024] Figure 13 An example of an implementation of LIC derivation and application for an SbTMVP merge block is shown; and

[0025] Figure 14 An example of an implementation of LIC derivation and application for triangle partitioning is shown;

[0026] Figure 15 An example of an implementation involving illumination compensation during encoding of picture information is shown; and

[0027] Figure 16 An example of an implementation involving illumination compensation during decoding of picture information is shown.

[0028] Throughout the drawings, identical reference numbers refer to identical or similar features. DETAILED DESCRIPTION

[0029] Recent efforts to improve video compression technology, such as those associated with the Joint Exploration Model (JEM) developed by the Joint Video Exploration Team (JVET), provide advanced features and tools. For example, such development efforts may include support for tools such as Local Illumination Compensation (LIC). LIC tools involve applying a LIC model to predict illumination changes that may occur between a prediction block and a corresponding reference block used for motion-compensated prediction. Using LIC involves determining parameters for the LIC model. However, as explained in more detail below, determining the model parameters can be problematic for certain operating modes of a video codec. One approach to addressing this issue may be to disable the use of LIC during such operating modes. In some cases, simply disabling the application of LIC can adversely affect compression efficiency. Generally speaking, various aspects and embodiments as described herein are directed to enabling the use of LIC during such potentially problematic situations.

[0030] This document describes various aspects, including tools, features, implementations, models, methods, and the like. Many of these aspects are described in detail, and often in a manner that may sound limited, at least to illustrate individual characteristics. However, this is for clarity and does not limit the application or scope of these aspects. In fact, all of the different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with aspects described in previous submissions.

[0031] The aspects described and contemplated in this document can be implemented in many different forms. Figure 1 、 Figure 2 and Figure 3 and other figures throughout this document provide some embodiments, but other embodiments are contemplated, and Figure 1 、 Figure 2 and Figure 3 The discussion does not limit the breadth of the embodiments. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, or computer-readable storage media. For example, the computer-readable storage medium can be a non-transitory computer-readable medium. The computer-readable storage medium can store thereon instructions for encoding or decoding video data according to any of the described methods and / or a bitstream generated according to any of the described methods.

[0032] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image", "picture" and "frame" are used interchangeably.

[0033] Various methods are described above, and each of the methods includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified or combined.

[0034] Various methods and other aspects described in this document may be used to modify one or more modules of a video encoder and / or decoder, such as, for example, the motion estimation module 170, the motion compensation module 175, and / or the motion compensation module 275 of the JVET or HEVC encoder 100 and decoder 200, as described in

[0026] , respectively. Figure 1 and Figure 2 Furthermore, aspects of the present invention are not limited to JVET or HEVC and are applicable, for example, to other standards and recommendations (whether pre-existing or developed in the future) and extensions of any such standards and recommendations (including JVET and HEVC). Unless otherwise indicated or technically excluded, the aspects described in this document may be used individually or in combination.

[0035] Various numerical values may be used in this document. Any specific numerical values are examples, and the described aspects are not limited to these specific values.

[0036] Figure 1 、 Figure 2 and Figure 3 Block diagrams illustrating examples of implementations of encoder 100, decoder 200, and system 1000 are respectively shown. Variations of encoder 100, decoder 200, and system 1000 are contemplated, but the examples described below are provided and described for clarity without necessarily describing all possible or contemplated variations.

[0037] exist Figure 1 In MPEG-4, before encoding, a video sequence may undergo pre-encoding processing (101), for example, applying a color transform to an input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and attached to the bitstream.

[0038] In an encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is partitioned (102) and processed in units such as CUs. Each unit is encoded, for example, using intra mode or inter mode. When a unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of intra mode or inter mode to use for encoding the unit and indicates the intra / inter decision by, for example, a prediction mode flag. The prediction residual is calculated, for example, by subtracting (110) the predicted block from the original image block.

[0039] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy encoded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., encode the residual directly without applying the transform or quantization process.

[0040] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. A loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0041] Figure 2 1 shows a block diagram of an example of a video decoder 200. In the decoder 200, the bitstream is decoded by the decoder elements as described below. The video decoder 200 generally performs the same operations as described above. Figure 1 The decoding process is the opposite of the encoding process. Figure 1 The encoder 100 in also typically performs video decoding as part of encoding the video data, for example, to provide a reference for further prediction.

[0042] In particular, the input to the decoder comprises a video bitstream which may be generated by a video encoder such as Figure 1The video encoder 100 generates a video image. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors and other encoded information. The picture partition information indicates how the picture is partitioned. Therefore, the decoder can divide the picture according to the decoded picture partition information (235). The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (255) to reconstruct the image block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). A loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0043] The decoded pictures may also undergo post-decoding processing (285), such as an inverse color transform (e.g., a transform from YCbCr 4:2:0 to RGB 4:4:4) or performing an inverse remapping of the remapping process performed in the pre-encoding process (101). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0044] Figure 3 1 shows a block diagram of a system in which various aspects and embodiments may be implemented. System 1000 may be embodied as a device including the components described below and configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. System 1000 may be communicatively coupled to other similar systems and communicated to other systems via, for example, Figure 3 The communication channels shown and as known to those skilled in the art are shown to implement one or more of the various aspects described in this document.

[0045] The system 1000 may include at least one processor 1010 configured to execute instructions loaded therein for implementing one or more of the various aspects described in this document. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits as known in the art. The system 1000 may include at least one memory 1020 (e.g., a volatile memory device, a non-volatile memory device). The system 1000 may include a storage device 1040, which may include non-volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, a magnetic disk drive, and / or an optical disk drive. As non-limiting examples, the storage device 1040 may include an internal storage device, an attached storage device, and / or a network-accessible storage device. The system 1000 may include an encoder / decoder module 1030, which is configured to process data to provide encoded video or decoded video.

[0046] The encoder / decoder module 1030 represents a module that can be included in a device to perform encoding and / or decoding functions. As is well known, a device can include one or both of an encoding module and a decoding module. In addition, the encoder / decoder module 1030 can be implemented as a separate component of the system 1000, or can be incorporated into the processor 1010 as a combination of hardware and software as known to those skilled in the art.

[0047] Program code to be loaded onto processor 1010 to perform various aspects described in this document may be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. Depending on the embodiment, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more of various items during execution of the processes described in this document, including, but not limited to, input video, decoded video, bitstreams, equations, formulas, matrices, variables, operations, and operation logic.

[0048] The system 1000 may include a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over the communication channel 1060. The communication interface may include, but is not limited to, a modem or a network card, and the communication channel may be implemented within a wired and / or wireless medium. The various components of the system 1000 may be connected or communicatively coupled together using various suitable connections, including, but not limited to, internal buses, wires, and printed circuit boards. For example, Figure 3 The system 1000 in FIG. 1 includes a communication network or bus 1140 .

[0049] As will be explained in detail below, aspects and embodiments according to the present disclosure may involve Figure 1 、 Figure 2 and Figure 3 Features of the system shown in , such as motion estimation features (e.g., Figure 1 Module 175) and motion compensation features such as Figure 1 Module 170 and / or Figure 2 Module 275).

[0050] For clarity, the following detailed description will describe various aspects with reference to implementations involving video compression techniques such as, for example, HEVC, JEM, and / or H.266. However, the described aspects are applicable to other video processing techniques and standards.

[0051] In the HEVC video compression standard, for encoding a picture, the frame is first split into large blocks designated as coding tree units (CTUs), which may be further split into smaller coding units (CUs), such as Figure 4 To encode a CU, a prediction unit (PU) is constructed from neighboring reconstructed samples (intra-frame prediction) or from previously reconstructed pictures stored in the decoded picture buffer (DPB) (inter-frame prediction). Next, the residual samples, calculated as the difference between the original samples and the PU samples, are transformed and quantized.

[0052] In inter prediction, motion compensated temporal prediction is employed to exploit the redundancy existing between consecutive pictures of a video. To this end, a motion vector is associated with the PU, and a reference index 0 (refIdx0) indicates which reference picture in the candidate picture list (LIST_0) to use.

[0053] In the Joint Exploration Model (JEM) and the VVC reference software developed by the JVET (Joint Video Exploration Team) team, some additional temporal prediction tools with associated parameters determined at the decoder side have been proposed. One such tool is Local Illumination Compensation (LIC). Basically, the purpose of LIC is to compensate for illumination changes that may occur between a prediction block and the reference block it uses for motion-compensated temporal prediction.

[0054] The use of LIC is usually signaled at the CU level through a flag (LIC flag) associated with each coding unit (CU) coded in inter mode. When the tool is activated and the LIC flag is true, the decoder calculates some prediction parameters based on the reconstructed picture samples located on the left and / or top of the current block to be predicted and the corresponding reference picture samples located on the left and / or top of the reference block. Figure 5A 、 Figure 5B and Figure 5C More specifically, Figure 5A、 Figure 5B and Figure 5C The LIC parameters are shown for a square CU ( Figure 5A ), rectangular CU( Figure 5B ) and sub-block based prediction ( Figure 5C ) is derived from neighboring reconstructed samples and corresponding reference samples transformed with a motion vector (MV). The description herein will refer to a set consisting of samples located in a row above the current (or reference) block and samples located in a column to the left of the current (or reference) block as an "L-shape" associated with the current or reference block. Examples of L-shapes are Figure 5A 、 Figure 5B and Figure 5C Shown in grey shading.

[0055] Some tools may be based on partitioning a CU or block into sub-blocks, e.g., as in Figure 5C 4 × 4 shown on the right. For some sub-block based tools, if the motion vectors associated with each sub-block are different, the reference L-shape can be constructed from non-adjacent samples, so that the MV indicates non-adjacent sub-blocks in the reference, as Figure 5C As shown. A group of non-adjacent sub-blocks such as Figure 5C The set of non-adjacent sub-blocks shown on the left can be referred to as a "quasi-L-shape". Examples of sub-block based motion compensation prediction modes are:

[0056] - affine merged motion predictions,

[0057] - Affine AMVP (Adaptive Motion Vector Prediction) motion prediction,

[0058] - Sub-block based temporal motion vector prediction (SbTMVP),

[0059] -Decoder-side motion vector correction (DMVR).

[0060] LIC may be based on an illumination variation model, such as a first-order linear illumination variation model using a scaling factor a and an offset b. In the case of a linear model, the model parameters a and b are derived from an L-shaped or quasi-L-shaped reconstruction and reference samples.

[0061] The LIC parameters may be selected based on any of a variety of methods. For example, the LIC parameters may be selected based on minimizing the error or local distortion between the current sample and the linearly modified reference sample for a local distortion defined as follows:

[0062]

[0063] in:

[0064] rec_cur(r) is the neighboring reconstructed sample in the current frame (Figure 5, right); and

[0065] rec_ref(r) is the reference sample constructed with motion compensation (MC) from the reference picture corresponding to the reconstructed sample r (Fig. 5, left), where s = r + mv. rec_cur(r) and rec_ref(r) are the co-located samples in the reconstructed and reference L-shapes, respectively.

[0066] The value of (a,b) can be obtained using least squares minimization:

[0067]

[0068] Note that for the sum term in Equation 2, the value of N can be further adjusted (incrementally reduced) to remain below the maximum allowed integer storage value (e.g., N<2 16 ). In addition, in Equation 2, the subsampling of the top sample set and the left sample set can be incremented for large blocks.

[0069] Another example of a method for deriving the LIC parameter may involve using the mean absolute deviation. For example, the sum of absolute differences (SAD) may be used to determine the LIC parameter as follows:

[0070]

[0071]

[0072] in:

[0073] - cur(r) is the neighboring reconstructed sample in the current frame (e.g. Figure 5B right side),

[0074] -ref(s) is the value from the reference picture (e.g. Figure 5B Left) Reference sample constructed with motion compensation (transformed by motion vector mv), and

[0075] -s=r+mv.

[0076] Another example of a method for deriving LIC parameters may involve using the minimum and maximum reconstructed sample values of the L-shape to determine the parameter "a". Once "a" is known, "b" may be derived as, for example, b = y mean -a*x mean (See the equation for “b” in Equation 2), where (x mean ;y mean ) are the average values of the reference sample and the reconstructed sample, respectively.

[0077] Once the LIC parameters are obtained for the current CU by the encoder or decoder, the prediction pred(current_block) of the current CU can be determined from the following equation for the unidirectional prediction case:

[0078] pred(current_block)=a×ref_block+b

[0079] Equation 3

[0080] where current_block is the current block to be predicted, pred(current_block) is the prediction of the current block, and ref_block is the reference block constructed with the regular motion compensation (MC) process and used for temporal prediction of the current block.

[0081] Note that a set of neighboring reconstructed samples and a set of reference samples (see the gray samples in FIG5 ) have the same number and the same pattern. The description herein will refer to “left samples” to refer to a set of neighboring reconstructed samples (or a set of neighboring reference samples) located to the left of the current block (or reference block), and will refer to “top samples” to refer to a set of neighboring reconstructed samples (or a set of neighboring reference samples) located at the top of the current block (or reference block). References to a “sample set” will refer to the union of the “left sample” set and the “top sample” set.

[0082] In the case of bi-prediction, the LIC process can be applied twice, as Figure 6 First, the LIC process is applied to the reference 0 prediction (LIST-0), as shown in Figure 6 300, 310, 320, 330 and 340 in FIG. Next, the LIC process is applied to the reference 1 prediction (LIST_1), as shown in FIG. Figure 6 The operations at 305, 315, 320, 335 and 345 may be the same as those at 300, 310, 330 and 340, respectively, i.e., those operations are simply repeated, but for illustration purposes, the operations at 305, 315, 335 and 345 are repeated. Figure 6 Next, the two predictions are combined as usual at 350 using either the default weighting (P=(P0+P1+1)>>1) or the Bi-Prediction Weighted Average (BPWA): P=(g0.P0+g1.P1+(1<<(s-1)))>>s). Figure 6 The method will be referred to herein as method-a.

[0083] This paper refers to it as Method-b and Figure 7 In the variant shown in , in case of bi-prediction, two normal predictions may first be performed and combined, followed by applying the LIC process once.

[0084] In another variation (method-c based on method-b), in the case of bi-prediction, the LIC-0 and LIC-1 parameters can be derived directly using least squares minimization, for example:

[0085]

[0086] In another variation, the set of reference (or reconstructed) samples to the left (or top) of the current block are not necessarily contiguous, as they may have been subsampled and / or some reconstructed samples may have been discarded according to some properties.

[0087] Some restrictions may apply to the choice of left and top samples for deriving LIC parameters (see grey samples in Figure 5):

[0088] R1) The number of samples in the sample set should be a power of 2 so that the partitioning can be performed using a right shift (e.g., Formula 2 or Formula 2bis)

[0089] R2) Left (N L ) samples and top (N T )The number of samples is the same and equal to

[0090] Ns (N = 2 * Ns) Equation 4

[0091] in:

[0092] n = min (cu height, cu width)

[0093] x=log2(n)

[0094] Ns=2 x

[0095] R3) The step size between left (step size V) samples or top (step size H) samples is equal to:

[0096] Step size V = cu height >> log2(Ns) Equation 5

[0097] Step size H = cu width >> log2(Ns)

[0098] Figure 8 An example of the positions of the left and top reference samples of a rectangular CU according to the aforementioned restrictions is shown in FIG.

[0099] Another example includes discarding reconstructed samples constructed using the intra prediction process, e.g. Figure 9 That is, Figure 9 LIC parameter calculation based on restricted access to samples is shown. Figure 9 In the , only gray samples can be used for parameter calculation; white samples are not available. For example, access to intra samples may be limited due to factors such as pipeline processing requirements.

[0100] In the sub-block-based temporal motion vector prediction (SbTMVP) method, similar to the temporal motion vector prediction (TMVP) in HEVC, the motion field in the collocated picture is used to improve the motion vector prediction and merge mode of the CU in the current picture. SbTMVP predicts motion at the sub-CU level. Motion shifting is applied before grabbing temporal motion information from the collocated picture, where the motion shift is obtained from the motion vector of one of the spatially neighboring blocks of the current CU instead of the collocated block.

[0101] Figure 10 An example of the SbTMVP process is shown in FIG. SbTMVP predicts the motion vectors of the sub-CUs within the current CU in two steps. In the first step, the spatial neighbors for ATMVP are Figure 10 The first spatial neighboring block with a motion vector that uses the collocated picture as its reference picture is identified. The identified motion vector is selected as the motion shift to be applied. If no such motion is identified from the spatial neighbors, the motion shift is set to (0, 0).

[0102] The second step involves applying the motion shifts from the spatial neighbors identified in the first step (e.g., Figure 10 MV of block A1 in the example shown in the lower part of ), and scale the motion information from the corresponding collocated sub-CU. In more detail, applying motion shifting involves adding the motion shifting information to the coordinates of the current block to obtain sub-CU level motion information (motion vector and reference index) from the collocated picture. Then, for each sub-CU, the motion information of its corresponding block in the collocated picture (the minimum motion grid covering the center sample) is used to derive the motion information of the sub-CU. After the motion information of the collocated sub-CU is identified, it is converted into the motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference picture of the temporal motion vector with the reference picture of the current CU.

[0103] The coding logic for additional SbTMVP merge candidates is the same as for other merge candidates. That is, for each CU in a P or B slice, an additional rate-distortion (RD) check is performed to decide whether to use the SbTMVP candidate.

[0104] In the case of non-rectangular prediction, the CU prediction consists of two non-rectangular predictions (PUs), as in Figure 11. In this mode, the block is split by a non-horizontal or non-vertical line, resulting in two parts. The coding mode is generally referred to as geometric partitioning (GEO) or triangular, and may also be generally referred to as non-rectangular mode in this article. The PUs have different motion vectors. In the case of merge mode, each vector comes from a different spatial or temporal neighbor. Figure 11 The lower part of shows the associated diagonal weightings between PUs.

[0105] It may happen that the default position of the sample set of the current CU is not suitable for the associated PU. Figure 12 The appropriate selection of neighboring samples used for triangle partitioning according to the PU is shown in . The sample set may be far away from the PU sample and discontinuous with the PU sample.

[0106] In this case, it is advantageous to use only samples that are spatially contiguous with the PU triangle for deriving LIC parameters. For example, in a triangle configuration such as Figure 12 In the case of the triangle configuration shown in the upper left of FIG), only the set of "top samples" for triangle "a" (N=NT) and only the set of "left samples" for triangle "b" (N=NL) are used. Figure 12 In the case of the triangle configuration example shown on the lower left side, both the “top sample” set and the “left sample” set are used for triangle “c”, and the LIC flag is inferred to be false for triangle “d”.

[0107] The same strategy can be applied to Figure 11 The four non-rectangular partitions shown in the upper right corner only keep samples of the same size as the PU size and the CU size. This means Figure 11 The non-rectangular PU partition examples 3 and 5 in will have the same sample set as the PU triangle of example 1, and the non-rectangular PU partition examples 4 and 6 will have the same sample set as the PU triangle of example 2. Figure 11 In the case of the left PU of partition examples 3 and 4, only the "left samples" and a subset of the "top samples" are used, where the subset of the "top samples" are the "top samples" that are continuous with the left PU. Figure 11 In the case of the right PU of partition example 3 in , a subset of “top samples” that are continuous with the right PU can be used. The same principle can be applied to select Figure 11 Reference samples with the “left samples” subset for PU partition examples 5 and 6 in .

[0108] In some of the scenarios presented above, the LIC process can be problematic, for example, reducing compression efficiency. For example, in the case of sub-block-based temporal motion vector prediction (SbTMVP), sub-block motion can differ and vary significantly between sub-blocks. Sub-blocks may also use different reference pictures. On the one hand, using the same LIC parameters for the entire CU (multiple sub-blocks) can produce visible artifacts and thus reduce compression efficiency. On the other hand, using sub-block neighbors to calculate the LIC parameters for each sub-block will create reconstruction dependencies between sub-blocks, which can make pipeline implementation problematic or impossible due to pipeline complexity constraints. In the case of triangular partitions, neighboring samples may not be available, and LIC cannot be applied efficiently. Therefore, in early experimental implementations of codecs such as VVC, the described problem patterns do not allow LIC, resulting in reduced compression efficiency.

[0109] In general, at least one embodiment described herein addresses such issues and provides LIC in such situations. That is, in general, at least one embodiment can provide LIC flag inference in the context of sub-block temporal motion vector prediction and triangle partitioning for both merge and classic inter modes. In general, at least one embodiment improves the derivation of sub-block-based temporal motion vector prediction (SbTMVP) and local illumination compensation (LIC) activation for triangle partitioning by efficiently controlling LIC usage and LIC parameter calculation. In general, at least one embodiment modifies LIC flag propagation in merge mode or controls LIC flag signaling when necessary. Depending on the prediction and / or sub-partitioning of the current CU, LIC can be inferred based on more information than just the LIC flags of neighboring blocks.

[0110] With SbTMVP, motion compensation is performed on a sub-block basis. For LIC to be compression-efficient, sub-block motion needs to be suppressed, such as motion within the same reference picture and / or similar motion. Furthermore, in current implementations of codecs such as VVC, LIC flags are not preserved between frames. Furthermore, motion on sub-blocks does not carry LIC information. Generally speaking, at least one embodiment addresses these limitations, thereby enabling expanded and / or improved use of LIC.

[0111] Figure 13 An example of at least one embodiment of LIC derivation according to the present disclosure is shown in FIG. Figure 13 In the embodiment of the present invention, at 1210, a spatial neighboring block A for motion shifting is determined. At 1220, motion vectors and reference indices of all sub-CUs are obtained. At 1230, if LIC is applied by spatial neighbor A for motion shifting, LIC is enabled. Thus, motion or motion shifting is determined based on temporal information, and LIC activation is determined based on spatial information.

[0112] In a variation, if all sub-block motion vectors are similar to those determined at 1240, then the LIC may be used for the current CU. For example, if all sub-block motion vectors use the same reference picture, have the same direction (unidirectional or bidirectional with the same reference index), and are close in magnitude to each other (e.g., the standard deviation of each component of the motion vector is less than a threshold (e.g., 5 pixels or samples)), then all sub-block motion vectors may be considered similar. Other metrics may be used, for example, to control complexity. One example of an alternative metric is to evaluate the vector v i The maximum value of the difference between the maximum and minimum values of each vector component and compare it to a limit or threshold, such as max(max(v i .x)-min(v i .x),max(v i .y)-min(v i Another example of a metric is to determine the average difference between the first vector (in scan order, i.e., the top left sub-block) and each sub-block vector and compare to a limit or threshold, e.g., average difference = sum(vi.x–v0.x+vi.y–V0.y) / (number of vectors).

[0113] If it is determined at 1240 that LIC has been disabled ("No" at 1240), the block is motion compensated at 1290 and the residual is transformed and coded as usual at 1280. If LIC is applied ("Yes" at 1240), LIC parameters are calculated at 1250, for example using Figure 5C The quasi-L-shaped method shown and described above is used for calculation. Next, at 1260, the block is motion compensated by the sub-block. Then, at 1270, LIC is applied to the entire CU. Finally, at 1280, the residual is transformed and encoded.

[0114] In general, at least one other example of an embodiment is compatible with and combinable with other methods for LIC parameter calculation (e.g., for sub-block based motion compensation such as LIC parameter derivation for affine or DMVR). In the case of affine mode, one or more embodiments are compatible with applying LIC compensation as described herein (including based on evaluating Figure 5C For the DMVR case, if the LIC process is initially inferred to be active or enabled (LIC flag inferred to be "true"), the LIC process may be disabled due to the DMVR process (sub-block modification of motion vectors at the decoder side), for example, if the motion vectors output from the DMVR process are not similar.

[0115] At least one other example of an implementation involving sub-CU motion similarity may additionally constrain motion vectors to be unidirectional for the LIC to be applied.

[0116] In at least one other example of an embodiment, if LIC information for temporal motion prediction is available, determining sub-CU motion similarity may additionally involve constraining the vectors to be LIC-enabled.

[0117] In at least one other example of an embodiment, the LIC flag may be signaled rather than derived from spatial neighbors. If the sub-CU motion constraints are satisfied, the LIC flag is encoded (encoder) or decoded (decoder), and LIC is applied, if appropriate. This means that the encoder operates to find the appropriate LIC flag value (i.e., on or off). For example, the encoder processes video data with both LIC on and LIC off, and evaluates the performance in each case based on a metric such as rate-distortion ratio (classic RDO (rate-distortion optimization)). The encoder then operates based on the LIC flag value or configuration that results in the best performance.

[0118] Now turning to the triangle partition case (available only in merge), the CU is split into two PUs, where each PU has its own motion, e.g., from different neighbors in the current implementation of VVC. The block is compensated twice (once for each motion vector), and then the two predictions are blended using weights according to the split direction, as in Figure 11 depicted in the lower half of the .

[0119] In general, at least one example of an implementation may involve LIC application, which depends on the neighbor vector LIC information, split direction, and PU index. Figure 14 Depicts an example of the derivation and application of LIC parameters for triangle partitioning. Figure 14 In , A and B refer to the spatially adjacent blocks used to derive the motion of the two PU triangular partitions. Figure 14At 1300 in FIGURE 1, vectors used as motion information for neighboring blocks A and B are determined. At 1305, a split direction (e.g., 45° or 135°) is determined. First, the LIC must be inferred from neighboring blocks used for motion prediction (steps 1310 and 1340 for A and B, respectively). For a given PU, if its neighbors have LIC enabled, the LIC for that PU may be considered. If the split direction (determined at 1305) and PU index (see Table 1 below) allow, as determined at steps 1315 and 1345 for A and B, respectively, LIC is enabled or used for the PU. If LIC is enabled or used, as determined at steps 1315 (and / or 1345), LIC parameters are calculated at step 1320 (and / or 1350), followed by motion compensation at step 1325 (and / or 1355), and LIC is applied to the motion-compensated PU at step 1330 (and / or 1360). If the PU does not have LIC (No at 1310 and / or 1340) and / or the triangle partition does not enable or permit LIC (No at 1315 and / or 1345), the PU is motion compensated (1370 and / or 1375). After 1330, 1360, 1370, or 1375, the two PUs are blended at 1380 (e.g., as Figure 11 ), then the residual and transform are calculated at 1390.

[0120] In at least one other example of an embodiment, LIC parameter derivation may have some constraints on the use of given reference samples, depending on the PU index, such as the example shown in Table 1.

[0121]

[0122] Table 1: LIC derivation for triangle partitioning. LIC activation and parameter derivation

[0123] In at least one other example of an embodiment, if the neighboring block for the current PU (A or B, depending on the triangle index) has a LIC, the LIC may be applied conventionally regardless of the split direction or PU index.

[0124] In at least one other example of an embodiment, the same LIC parameters may be used for both PUs based on an averaging model that involves first calculating the LIC parameters for each PU and then averaging the two LIC parameter sets.

[0125] Due to complexity issues, specific implementations such as VVC only implement uni-predictive LIC. However, triangle partitioning is implemented as bi-predictive. The bi-prediction pipeline uses one prediction with each triangle, which is then merged by pixel-by-pixel or sample-by-sample blending. Implementing LIC in such a context may involve computing the full bi-prediction before applying LIC to have only one LIC block in the pipeline (instead of one LIC block per prediction), such as in Figure 7 For the LIC triangle case, computing the full bi-prediction before applying LIC may involve applying the implementation example just described, i.e. deriving only one set of LIC parameters to apply to both predictions.

[0126] At least one other example of an embodiment may involve using the same LIC parameters for two PUs based only on the first PU. For example, in the case of a 45° split, the LIC parameters for the first PU are calculated and the same LIC parameters are used for the second PU. In a variation, the LIC parameters of the first PU are used for the LIC of the second PU only if the motion of the second PU is "similar" to that of the first PU. The "similar" property may include a combination of at least one of having the same reference picture or the difference between each component of the motion vector meeting a criterion (such as being less than a certain value, for example, 5 pixels or samples). In another variation, the LIC parameters of the first PU are used for the second PU, but the LIC is applied only if a neighbor associated with the second PU has a LIC.

[0127] In at least one other example of an embodiment, a LIC flag is signaled for each PU, rather than being derived from neighbors. If the split direction and PU index constraints are met, the LIC flag is encoded (decoded) and, if appropriate, applied. In one variant, only one LIC flag is signaled for both PUs.

[0128] In case of temporal motion vector prediction, the motion vector is predicted from the previously coded frame. The LIC flag can be inferred from this motion vector predictor. However, in specific implementations such as VVC, the LIC flag is not available for the temporal predictor as explained above. In this case, the LIC flag can be inferred from the spatial neighbors. For example, if the A0, A1, B0, B1 neighbors ( Figure 10 -top) has LIC enabled, then LIC is applied. In a variation, the condition for propagating LIC may involve requiring more than one neighbor to have LIC enabled. In another variation, LIC is applied if the spatial neighbor has LIC and the current PU's motion is "stopped" or similar to the given neighbor's motion, where "stopped" may be defined as having the same reference and vector differences meeting a specific criterion referred to herein as a similarity criterion (e.g., less than 5 pixels or samples per component difference).

[0129] Figure 15 Another example of an implementation is shown in FIG. Figure 15 In the embodiment of the present invention, at 1510, a motion compensation mode for encoding a first block of picture information is determined. For example, the motion compensation mode may include a sub-block based motion compensation prediction mode or a non-rectangular block partition mode. The non-rectangular block partition mode may be, for example, as described above with respect to Figure 11 and Figure 12 At 1520, a second block of picture information that is spatially adjacent to the first block is determined to be encoded based on illumination compensation. Then, at 1530, the first block is encoded based on the motion compensation mode and illumination compensation.

[0130] Figure 16 Another example of an implementation is shown in FIG. Figure 16 In the embodiment of the present invention, at 1610, a motion compensation mode for encoding the first block of picture information is determined. The motion compensation mode may include a sub-block based motion compensation prediction mode or a non-rectangular block partition mode. The non-rectangular block partition mode may be, for example, as described above with respect to Figure 11 and Figure 12 At 1620, a second block of picture information that is spatially adjacent to the first block is determined to be decoded based on illumination compensation. Then, at 1630, the first block is decoded based on the motion compensation mode and illumination compensation.

[0131] The embodiments described herein may be implemented, for example, by Figure 3 The processor 1010 of the system 1000 may be executed by computer software implemented by hardware or a combination of hardware and software. As a non-limiting example, these embodiments may be implemented by one or more integrated circuits. As a non-limiting example, Figure 3 The memory 1020 included in the example of system 1000 shown in FIG. 1 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, the processor 1010 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0132] The specific implementations and aspects described herein can be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of specific implementation (e.g., discussed only as a method), the specific implementation of the features discussed can also be implemented in other forms (e.g., a device or program). The device can be implemented in, for example, appropriate hardware, software, and firmware. The method can be implemented in, for example, a device (such as, for example, a processor) that generally refers to a processing device, which includes, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication equipment, such as, for example, a computer, a mobile phone, a portable / personal digital assistant ("PDA"), and other equipment that facilitates information communication between end users.

[0133] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations thereof in various places throughout this document are not necessarily all referring to the same embodiment.

[0134] Furthermore, this document may refer to “determining” various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0135] Additionally, this document may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0136] Additionally, this document may refer to "receiving" various pieces of information. Like "accessing," receiving is intended to be a broad term. Receiving information can include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" generally involves, in one way or another, an operation such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0137] It will be apparent to one of ordinary skill in the art that a specific implementation may generate various signals formatted to carry, for example, storable or transmittable information. The information may include, for example, instructions for executing a method or data generated by one of the specific implementations. For example, a signal may be formatted to carry a bit stream of the embodiment. Such signals may be formatted, for example, as electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or as baseband signals. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is known that the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium (e.g., a non-transitory computer-readable medium).

[0138] Various embodiments have been described. Embodiments may include one or more of the following features or entities, alone or in any combination, across various claim categories and types:

[0139] • Provide local illumination compensation (LIC) flag inference in the case of sub-block temporal motion vector prediction and triangle partitioning for both merge and classic inter modes in the encoder and / or decoder.

[0140] • Provide improved derivation of LIC activation for sub-block based temporal motion vector prediction (SbTMVP) and for triangle partitioning in an encoder and / or decoder by efficiently controlling LIC usage and LIC parameter calculation.

[0141] • Provision is made in the encoder and / or decoder to modify the LIC flag propagation in merge mode and / or control the LIC flag signalling.

[0142] • Providing prediction and / or sub-partitioning of the current CU in the encoder and / or decoder to infer the LIC based on more information than just the LIC flags of neighboring blocks.

[0143] • Providing in the encoder and / or decoder to enable LIC based on spatial neighbors for the motion shift to which LIC is applied.

[0144] Providing in an encoder and / or decoder enabling LIC based on evaluating a similarity criterion for a plurality of sub-block motion vectors, wherein the similarity criterion includes the plurality of sub-block motion vectors satisfying a reference picture criterion, and the plurality of sub-block motion vectors satisfying a direction criterion, and the plurality of sub-block motion vectors satisfying a magnitude criterion.

[0145] providing, in an encoder and / or decoder, a method for enabling LIC based on evaluating similarity criteria associated with a plurality of sub-block motion vectors, wherein the similarity criteria comprises a reference picture criterion based on the plurality of sub-block motion vectors all using one or more same reference pictures, and a direction criterion based on the plurality of sub-block motion vectors all having the same direction with one or more same reference indices, and evaluating a magnitude criterion, wherein the magnitude criterion comprises at least one of: a standard deviation of each component of the motion vector, or a magnitude criterion of the vector v i The maximum value of the difference between the maximum value and the maximum value of each vector component of i .x)-min(v i .x),max(max(v i .y)-min(v i .y)), or the average difference between the first vector (in scan order, i.e., the top-left sub-block) and each sub-block vector (e.g., average difference = sum(vi.x–v0.x+vi.y–V0.y) / (number of vectors)).

[0146] Providing processing, in an encoder and / or decoder, of video data, the video data comprising a coding unit having a first prediction unit and a second prediction unit, each having a corresponding prediction unit index, and wherein the first prediction unit and the second prediction unit are formed based on a triangular partition of the coding unit, wherein the processing comprises implementing local illumination compensation based on local illumination compensation parameters of neighboring blocks of the coding unit and a splitting direction of the triangular partition and the prediction unit index.

[0147] Providing processing, in an encoder and / or decoder, of video data, the video data comprising a coding unit having a first prediction unit and a second prediction unit, each having a corresponding prediction unit index, and wherein the first prediction unit and the second prediction unit are formed based on a triangular partition of the coding unit, wherein the processing comprises implementing local illumination compensation based on local illumination compensation parameters of neighboring blocks of the coding unit.

[0148] Providing processing, in an encoder and / or decoder, of video data, the video data comprising a coding unit having a first prediction unit and a second prediction unit, each having a corresponding prediction unit index, and wherein the first prediction unit and the second prediction unit are formed based on a triangular partition of the coding unit, wherein the processing comprises implementing local illumination compensation based on local illumination compensation parameters of neighboring blocks of the coding unit and a split direction of the triangular partition and the prediction unit index, and wherein after enabling local illumination compensation, at least one parameter of a local illumination compensation model is determined and at least one prediction unit is processed based on the local illumination compensation model and the at least one parameter.

[0149] Providing processing video data in an encoder and / or decoder, the video data including a coding unit having a first prediction unit and a second prediction unit, each having a corresponding prediction unit index, and wherein the first prediction unit and the second prediction unit are formed based on a triangular partition of the coding unit, wherein the processing includes implementing local illumination compensation based on local illumination compensation parameters of neighboring blocks of the coding unit and a split direction of the triangular partition and the prediction unit index, and wherein after enabling local illumination compensation, the first prediction unit is processed to determine a first value of at least one parameter of a local illumination compensation model, the second prediction unit is processed to determine a second value of the at least one parameter, and both the first prediction unit and the second prediction unit are processed to apply local illumination compensation based on using the local illumination compensation model and a third value of the at least one parameter corresponding to an average of the first and second values.

[0150] Providing processing, in an encoder and / or decoder, of video data, the video data comprising a coding unit having a first prediction unit and a second prediction unit, wherein the first prediction unit and the second prediction unit are formed based on a triangular partition of the coding unit, wherein the processing comprises determining a local illumination parameter based on the first prediction unit, and enabling local illumination compensation by applying local illumination compensation to both the first prediction unit and the second prediction unit based on and using a local illumination compensation parameter.

[0151] Providing processing, in an encoder and / or decoder, of video data comprising a coding unit having a first prediction unit and a second prediction unit, wherein the first prediction unit and the second prediction unit are formed based on a triangular partition of the coding unit, wherein the processing comprises determining a local illumination parameter based on the first prediction unit, and enabling local illumination compensation based on and applying local illumination compensation to the first prediction unit using a local illumination compensation parameter, wherein the local illumination compensation parameter is used to apply the local illumination compensation to the second prediction unit if a motion characteristic of the second prediction unit satisfies a similarity criterion when compared to a motion characteristic of the first prediction unit, wherein the similarity criterion comprises a combination of having a same reference picture and a difference between each component of a plurality of motion vectors satisfying a difference criterion.

[0152] Providing processing, in an encoder and / or decoder, of video data, the video data comprising a coding unit having a first prediction unit and a second prediction unit, wherein the first prediction unit and the second prediction unit are formed based on a triangular partition of the coding unit, wherein the processing comprises determining a local illumination parameter based on the first prediction unit, and enabling local illumination compensation by applying local illumination compensation to the first prediction unit based on and using a local illumination compensation parameter, wherein the local illumination compensation parameter is used to apply local illumination compensation to the second prediction unit if a neighbor associated with the second prediction unit has applied local illumination compensation.

[0153] Providing, in an encoder and / or decoder, processing video data based on temporal motion vector prediction, including predicting motion vectors from previously encoded frames, wherein the processing includes determining a flag indicating whether local illumination compensation is enabled, and the flag is inferred from spatial neighbors.

[0154] Providing, in an encoder and / or decoder, processing video data based on temporal motion vector prediction, including predicting motion vectors from previously encoded frames, wherein the processing includes determining a flag indicating whether local illumination compensation is enabled, and the flag is inferred based on more than one spatial neighbor having local illumination compensation enabled.

[0155] Providing, in an encoder and / or decoder, processing video data based on temporal motion vector prediction, including predicting motion vectors from previously encoded frames, wherein the processing includes determining a flag indicating whether local illumination compensation is enabled, and the flag is inferred from spatial neighbors based on whether local illumination compensation is enabled for the spatial neighbors, and motion characteristics of a current prediction unit satisfy a similarity criterion compared to motion characteristics of the spatial neighbors' motion, wherein the similarity criterion includes having a common reference and satisfying a difference criterion.

[0156] • Providing an encoder and / or decoder for processing video according to any of the embodiments, features or entities as described herein (alone or in any combination) based on providing reduced complexity and / or improved compression efficiency.

[0157] • Insertion in a signaling syntax element enables an encoder and / or decoder to provide encoding and / or decoding according to any of the embodiments, features, or entities as described herein (alone or in any combination).

[0158] • Based on these syntax elements, features or entities as described herein are selected, alone or in any combination, to be applied at the decoder.

[0159] • A bitstream or signal comprising one or more of the described syntax elements or variants thereof.

[0160] • Inserting an enable decoder in the signaling syntax element to provide decoding in a manner corresponding to the encoding method used by the encoder.

[0161] • Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more of said syntax elements or variants thereof.

[0162] A television, set-top box, cell phone, tablet or other electronic device that provides for applying encoding and / or decoding according to any of the embodiments, features or entities as described herein, either alone or in any combination.

[0163] A television, set-top box, cell phone, tablet, or other electronic device that performs encoding and / or decoding according to any of the embodiments, features, or entities as described herein (alone or in any combination) and displays the resulting image (e.g., using a monitor, screen, or other type of display).

[0164] A television, set-top box, cellular phone, tablet, or other electronic device that tunes (e.g., using a tuner) to a channel to receive a signal including an encoded image and performs encoding and / or decoding according to any of the embodiments, features, or entities as described herein (alone or in any combination).

[0165] A television, set-top box, cell phone, tablet, or other electronic device that receives (e.g., using an antenna) an over-the-air signal including an encoded image and performs encoding and / or decoding according to any of the embodiments, features, or entities as described herein (alone or in any combination).

[0166] A computer program product storing program code which, when executed by a computer, encodes and / or decodes according to any one of the embodiments, features or entities as described herein (alone or in any combination).

[0167] A non-transitory computer-readable medium comprising executable program instructions that cause a computer executing the instructions to perform encoding and / or decoding according to any of the embodiments, features, or entities described herein (alone or in any combination).

[0168] Various other generalized and specific embodiments are also supported and contemplated throughout this disclosure.

Claims

1. A method for video decoding, the method comprising: determining a sub-block based motion compensation mode to be used for a first block of picture information; enabling illumination compensation to be used during decoding based on a plurality of motion vectors associated with respective sub-blocks of a plurality of sub-blocks of the first block of picture information satisfying a similarity criterion; determining that a second block of picture information spatially adjacent to the first block is decoded based on illumination compensation; as well as A second block spatially adjacent to the first block based on determining picture information is decoded based on illumination compensation, and the first block is decoded based on the sub-block based motion compensation mode and the illumination compensation.

2. A method for video encoding, the method comprising: determining a sub-block based motion compensation mode to be used for a first block of picture information; enabling illumination compensation to be used during encoding based on a plurality of motion vectors associated with respective sub-blocks of a plurality of sub-blocks of the first block of picture information satisfying a similarity criterion; determining that a second block of picture information that is spatially adjacent to the first block is encoded based on illumination compensation; as well as Based on determining that a second block spatially adjacent to the first block of picture information is encoded based on illumination compensation, the first block is encoded based on the sub-block based motion compensation mode and the illumination compensation.

3. A device for video decoding, the device comprising: A processor configured to: determining a sub-block based motion compensation mode to be used for a first block of picture information; enabling illumination compensation to be used during decoding based on a plurality of motion vectors associated with respective sub-blocks of a plurality of sub-blocks of the first block of picture information satisfying a similarity criterion; determining that a second block of picture information spatially adjacent to the first block is decoded based on illumination compensation; as well as A second block spatially adjacent to the first block based on determining picture information is decoded based on illumination compensation, and the first block is decoded based on the sub-block based motion compensation mode and the illumination compensation.

4. A device for video encoding, the device comprising: A processor configured to: determining a sub-block based motion compensation mode to be used for a first block of picture information; enabling illumination compensation to be used during encoding based on a plurality of motion vectors associated with respective sub-blocks of a plurality of sub-blocks of the first block of picture information satisfying a similarity criterion; determining that a second block of picture information that is spatially adjacent to the first block is encoded based on illumination compensation; as well as Based on determining that a second block spatially adjacent to the first block of picture information is encoded based on illumination compensation, the first block is encoded based on the sub-block based motion compensation mode and the illumination compensation.

5. The method according to claim 1 or 2 or the apparatus according to claim 3 or 4, wherein the sub-block based motion compensation prediction mode comprises a sub-block based temporal motion vector prediction mode. 6 . The method according to claim 1 , wherein the plurality of subblocks include all subblocks of the first block of picture information, and the plurality of motion vectors include all the motion vectors of all the subblocks of the first block of picture information.

7. The method according to claim 1 or 2, wherein the similarity criteria comprises at least one of the following: - the plurality of motion vectors have one or more identical reference pictures, or - the plurality of motion vectors have the same direction, or - The plurality of motion vectors have a magnitude relationship. 8 . The method of claim 7 , wherein the plurality of motion vectors having the same direction include the plurality of motion vectors having a same reference index in a unidirectional direction or the plurality of motion vectors having a same reference index in a bidirectional direction.

9. The method of claim 7, wherein the magnitude relationship comprises a standard deviation of each component of the plurality of motion vectors being less than a certain value.

10. The method of claim 9, wherein the value is five samples.

11. A method according to claim 1 or 2, wherein the similarity criterion comprises evaluating a maximum value of a difference between a maximum value and a minimum value of each vector component of each motion vector in the plurality of motion vectors, and comparing the maximum value with a limit or threshold.

12. The method of claim 1 or 2, wherein the similarity criterion comprises determining an average difference between a first motion vector in scan order and each sub-block motion vector and comparing the average difference to a limit value.

13. The apparatus according to claim 3 or 4, wherein the plurality of subblocks include all subblocks of the first block of picture information, and the plurality of motion vectors include all the motion vectors of all the subblocks of the first block of picture information.

14. The apparatus according to claim 3 or 4, wherein the similarity criteria comprises at least one of the following: - the plurality of motion vectors have one or more identical reference pictures, or - the plurality of motion vectors have the same direction, or - The plurality of motion vectors have a magnitude relationship. 15 . The apparatus of claim 14 , wherein the plurality of motion vectors having the same direction include the plurality of motion vectors having a same reference index in a unidirectional direction or the plurality of motion vectors having a same reference index in a bidirectional direction.

16. The apparatus of claim 14, wherein the magnitude relationship comprises a standard deviation of each component of the plurality of motion vectors being less than a certain value. The apparatus of claim 16 , wherein the value is five samples.

18. An apparatus according to claim 3 or 4, wherein the similarity criterion comprises evaluating a maximum value of a difference between a maximum value and a minimum value of each vector component of each motion vector in the plurality of motion vectors, and comparing the maximum value with a limit or threshold.

19. Apparatus according to claim 3 or 4, wherein the similarity criterion comprises determining an average difference between a first motion vector in scan order and each sub-block motion vector and comparing the average difference to a limit value.

20. A computer program product stored on a computer readable medium, comprising program code instructions for implementing the steps of the method according to any one of claims 5, 6-11 or 12 when executed by at least one processor.

Images