Method and apparatus for local illumination compensation flag inheritance

Abstract

A method and apparatus for local illumination compensation flag inheritance are disclosed, the method comprising: for a current image block in merge mode, obtaining a local illumination compensation (LIC) flag and an LIC flag counter associated with the LIC flag, where the LIC flag indicates whether local illumination compensation is applied to the current image block; in response to determining that the LIC flag indicates that local illumination compensation is applied to the current image block, calculating LIC model parameters for the current image block; in response to determining that an LIC model parameter is close to a default value, decrementing the LIC flag counter for the current image block; and decoding the current image block by using the LIC model parameters. Local illumination compensation can be flexibly used. These methods and apparatus include separate local illumination compensation for components of a video block, as well as a flexible approach for illumination compensation flag inheritance.

Description

[0001] This application is a divisional application of the Chinese invention patent application with the application date of June 18, 2020, application number 202080056412.0, and invention name “Local Lighting Compensation Mark Inheritance”. Technical Field

[0002] This disclosure relates to video compression, and more particularly to performing video coding and decoding. Background Art

[0003] Many attempts have been made to improve the coding efficiency of block-based codecs. Local Illumination Compensation (LIC) is a tool introduced for this purpose. In the Joint Exploration Model (JEM) and VVC reference software developed by the JVET (Joint Video Exploration Team) group, some additional temporal prediction tools with associated parameters determined on the decoder side are being studied, including Local Illumination Compensation (LIC). Basically, the purpose of LIC is to compensate for the illumination changes that may occur between the predicted block and its reference block adopted by motion compensated temporal prediction. Summary of the invention

[0004] These and other drawbacks and deficiencies of the prior art are addressed by the presently described embodiments, which are directed to a method and apparatus for managing the tradeoff between the coding efficiency provided by a FRUC tool and its complexity.

[0005] According to one aspect of the described embodiments, a method is provided. The method includes the steps of determining local illumination compensation information to be used for one or more video components of a video block; encoding the video block using the local illumination compensation information; and indicating in a bitstream whether the illumination compensation information is used for the video block.

[0006] According to another aspect of the described embodiment, a second method is provided. The method includes the following steps: parsing a bitstream to obtain syntax information indicating local illumination compensation; and decoding at least one video block in the bitstream based on the syntax to conditionally perform local illumination compensation on components of the one video block.

[0007] According to another aspect of the described embodiments, a device is provided. The device includes a memory and a processor. The processor can be configured to encode or decode a portion of a video signal by any of the above methods.

[0008] According to another general aspect of at least one embodiment, there is provided an apparatus comprising an apparatus according to any of the decoding embodiments; and at least one of: (i) an antenna configured to receive a signal comprising a video block, (ii) a band limiter configured to limit the received signal to a frequency band comprising the video block, or (iii) a display configured to display an output representing the video block.

[0009] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that contains data content generated according to any of the described encoding embodiments or variations.

[0010] According to another general aspect of at least one embodiment, there is provided a signal comprising video data generated according to any of the described encoding embodiments or variations.

[0011] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.

[0012] According to another general aspect of at least one embodiment, there is provided a computer program product comprising instructions which, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variations.

[0013] These and other aspects, features and advantages of the present principles will become apparent from the following detailed description of exemplary embodiments, which is to be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Coding tree units and coding tree concepts for representing compressed HEVC pictures are shown.

[0015] Figure 2 It is shown that the LIC parameters in JEM are derived from neighboring reconstructed samples and the corresponding reference samples converted with MV (top: square CU; bottom: rectangular CU).

[0016] Figure 3 The LIC flag inference process is shown in the case of merge mode at the encoder (left side) and decoder (right side).

[0017] Figure 4 Derivation of LIC parameters and application to each prediction L0 and L1 are shown.

[0018] Figure 5 Derivation of LIC parameters and application from L0 and L1 to combined prediction are shown.

[0019] Figure 6shows how the LIC flag can be inherited multiple times in succession.

[0020] Figure 7 A general video compression scheme is shown.

[0021] Figure 8 A general video decompression scheme is shown.

[0022] Fig. 9 A flow chart of one embodiment of decoding a video block according to the present principles is shown.

[0023] Fig.10 It shows that the LIC flag can be inherited multiple times consecutively.

[0024] Fig.11 A flow chart of another embodiment of decoding a video block according to the present principles is shown.

[0025] Fig.12 Another example is shown where the LIC flag may be inherited multiple times consecutively.

[0026] Fig.13 It is a processor-based system for encoding / decoding.

[0027] Fig.14 One embodiment of a method for decoding video using at least one lighting compensation flag is shown.

[0028] Fig.15 One embodiment of a method for encoding video using at least one lighting compensation flag is shown.

[0029] Fig.16 One embodiment of an apparatus for encoding or decoding video using at least one lighting compensation flag is shown. DETAILED DESCRIPTION

[0030] The field of embodiments described herein is video compression, aiming to improve the video compression efficiency of prior art video coding schemes. They aim to improve compression efficiency compared to existing video compression systems based on hybrid architectures, and to achieve efficient local illumination compensation.

[0031] Block-based video compression

[0032] In HEVC (High Efficiency Video Coding, ITU-T H.265) or the common video codec (VVC of the Joint Video Experts Group), in order to decode an image, a frame is first split into large blocks (CTU = Coding Tree Unit), which may be further split into smaller coding units (CU), such as Figure 1 shown.

[0033] To decode a CU, a prediction block (PU = prediction unit) is constructed from neighboring reconstructed samples (intra-frame prediction) or from previously reconstructed samples of a picture stored in a decoded picture buffer (DPB) (inter-frame prediction). Next, the residual samples calculated as the difference between the original samples and the PU samples are transformed, quantized and entropy decoded.

[0034] In inter prediction, motion compensated temporal prediction is employed to exploit the redundancy present between consecutive images of a video. To do this, a motion vector is associated with the PU, and a reference index 0 (refldx0) is used that indicates which reference picture of LIST_0 to use.

[0035] Local lighting compensation

[0036] In the Joint Exploration Model (JEM) and the JVC reference software (developed by the JVET (Joint Video Exploration Team) group), some additional temporal prediction tools with associated parameters determined at the decoder side are being studied, including Local Illumination Compensation (LIC). Basically, the purpose of LIC is to compensate for illumination changes that may occur between a predicted block and its reference block adopted by motion compensated temporal prediction.

[0037] The use of LIC is usually signaled at CU level by a flag (LIC flag) associated with each coding unit (CU) coded in inter mode. When this tool is activated, the decoder calculates some prediction parameters ( Figure 2 ). In the considered prior art codec (JEM), the use of LIC for a given block depends on a flag associated with the block (called LIC flag). The LIC process is performed on all image components (e.g. illumination, chrominance components).

[0038] In the following, we will call the “L-shape” associated with the current block the set consisting of samples on the rows above the current block and / or samples on the columns to the left of the current block, as Figure 2 Another proposal would be to discard some reference samples (eg, use only reference samples from coding units / blocks coded in inter-coding mode).

[0039] LIC logo inheritance

[0040] The LIC flag may be explicitly decoded or inherited. When it is inherited, the LIC flag is derived from previously decoded parameters such as previously reconstructed LIC flag values, such as Figure 3 As depicted for the encoder (401) and decoder (402).

[0041] In merge or skip mode, a parameter candidate list is established (410). Parameters may include MV, reference index, unidirectional or bidirectional prediction, LIC flag. If the LIC flag is true (430), the LIC process applies: derive LIC parameters (440) and apply them (450) to the predicted PU samples (420).

[0042] LIC Model

[0043] In prior art approaches, the LIC model is based on a simple linear correction (Equation 1) applied to the regular current block prediction.

[0044] Y corr (x) = aY pred (x)+b

[0045] Equation 1. Linear LIC model.

[0046] Where: Y pred (x) is the predicted sample value at position x,

[0047] Y corr (x) is the corrected predicted sample value at position x,

[0048] (a, b) are the LIC parameters (sometimes called scale and offset).

[0049] However, the described embodiments are not limited to this simple model. For example, other models may be used, such as (Y corr =a.(Y pred ) 2 +bY pred +c) or (Y corr =a.log(Y pred )+b).

[0050] LIC parameter estimation

[0051] In the case of the standard LIC model (Equation 1), the LIC parameters (a, b) are weights and biases, and the weights and biases can be determined based on minimization of the error between the current sample and the linearly modified reference sample, which are defined as follows:

[0052]

[0053] Equation 2: LIC model distortion.

[0054] Where: rec_cur(r) is the current picture ( Figure 2 -right side),

[0055] rec_ref(s) is the image from the reference image ( Figure 2 - left side) constructed reference sample, where s = r + mv,

[0056] rec_cur(r) and rec_ref(r) are the co-located samples in the reconstructed L-shape and the reference L-shape, respectively.

[0057] The values ​​of (a, b) are obtained using least squares minimization (LSM) (Equation 3):

[0058]

[0059] Equation 3: (a, b) Derivation of LIC parameters.

[0060] Note that the value of N may be further adjusted (eg, incrementally decreased) to keep the summation term in Equation 3 below the maximum allowed integer storage value (eg, N < 2 16 ). For large blocks, subsampling of the top and left sample sets can also be increased.

[0061] Once the encoder or decoder obtains the LIC parameters for the current CU, the prediction pred(current_block) of the current CU includes the following (unidirectional prediction case):

[0062] pred(current_block)=a×ref_block+b (Equation 1)

[0063] Where current_block is the current block to be predicted, pred(current_block) is the prediction of the current block, and ref_block is a reference block constructed using a conventional motion compensation (MC) process and used for temporal prediction of the current block.

[0064] In the case of bidirectional prediction, the LIC process is applied twice, first on the reference 0 prediction (LIST-0) and second on the reference 1 prediction (LIST_1). Figure 4 Next, the two predictions are combined, usually using either the default weighting (P = (P0 + P1 + 1) >> 1) or the bidirectional prediction weighted average (BPWA) P = (g0.P0 + g1.P1 + (1 << (s-1))) >> s). This approach is referred to as method a.

[0065] In a variant (method b), in case of bidirectional prediction, the conventional predictions are first combined and then the next single LIC process is applied ( Figure 5 ).

[0066] Default LIC model

[0067] The estimation / calculation of the LIC parameters may produce values ​​corresponding to the identifier. In this case, for all "x" samples of the PU block, with Y corr (x) = Y pred (x). In the case of the LIC model described by Equation-1, this corresponds to (a; b) = (1; 0). In general, this may occur when there is no illumination change between the current and reference blocks, or when the "L-shape" is not available (e.g., a CU at an upper-left picture / slice boundary), or when the range of values ​​is too narrow to estimate the LIC parameters. In the following, we denote the "default model" as the LIC model that does not modify the PU sample values.

[0068] The LIC flag indicates whether the LIC process is applicable to the current CU for all components. However, it may happen that the estimated LIC parameters are default values ​​for at least one component. In case the LIC flag is explicitly coded, the encoder can select the value of the LIC flag (true or false) according to the best rate-distortion tradeoff (410). However, when the LIC flag is inherited, the encoder cannot easily control the LIC process because the LIC flag can be inherited multiple times in a row, such as Figure 6 As depicted in .

[0069] The present invention improves the derivation of component-based Local Illumination Compensation (LIC) flag parameters by deriving LIC flags. In addition, LIC flag inheritance can be conditioned on the actual derived LIC parameter values.

[0070] Embodiment 1:

[0071] Each component defines a LIC flag. C components (for example, for YUV or RGB, N C =3) and the LIC flag is explicitly decoded (e.g., AMVP mode), for N C The LIC logo is encoded.

[0072] In one variant, a single LIC flag value is encoded and each component LIC flag is equal to the decoded LIC flag.

[0073] Embodiment 2 (can be combined with embodiment 1):

[0074] One defines another __LIC_flag, which we denote as LIC_flag_out. The conventional LIC_flag indicates whether the LIC process (LIC parameter derivation and PU sample correction) is applicable to the current CU. LIC_flag_out indicates the value of the LIC flag that can be inherited by another CU (550).

[0075] In one variant, a value of "LIC_flag_out" is defined for each component C, and the derivation of "LIC_flag_out[C]" (570) depends on the calculated LIC parameter for component C. For example, if the calculated LIC parameter for component C is the default value, then LIC_flag_out[C]=false; otherwise, LIC_flag_out[C]=true. If the regular "LIC_flag" is false, then "LIC_flag_out" is false.

[0076] In one variant, in the case of the LIC model given by (Equation 1), if the ratio value is zero, then LIC_flag_out[C]=false.

[0077] Embodiment 2.1 (can be combined with Embodiment 1 and Embodiment 2):

[0078] If the regular "LIC_flag" of the current block is true, then the derivation of "LIC_flag_out" can depend on the "LIC_flag_out" information of the previous neighbor. For example, "majority", if most of the "LIC_flag_out" in the decoded neighboring blocks are true, then the "LIC_flag_out" of the current block is set to true. As Fig.10 shown, if the "LIC_flag_out" of neighboring N0 and N2 is true, while the "LIC_flag_out" of another neighbor N1 is false, then the "LIC_flag_out" of the current block is derived as true, which is the major value among its neighbors.

[0079] Embodiment 3 (can be combined with Embodiment 1 and Embodiment 2):

[0080] In another variant of emb-2, if the value of the LIC parameter is close to the default value, then LIC_flag_out[C]=false. For example, in the case of the LIC model (Equation 1), "close to default" (default a is 1 (i.e., 32>>5) can correspond to:

[0081] abs(a - 1)<th_a and abs(b)<th_b

[0082] where (th_a; th_b) are predetermined values (e.g., 0.1 and 10<<(bit depth_8))

[0083] Embodiment 4 (can be combined with Embodiments 1, 2 and 3):

[0084] In another variant, the derivation of "LIC_flag_out[C2]" (570) depends on the calculated LIC parameter for component C1, where C1≠C2. For example, C1 = luminance and C2 is a chrominance component (e.g., Cb or Cr).

[0085] Embodiment 5 (can be combined with Embodiments 1, 2, 3 and 4):

[0086] One value of "LIC_flag_out" is defined for each component type T, for example, one for luma and one for chroma (components Cb and Cr).

[0087] "LIC_flag" is decoded or inferred from the component type.

[0088] "LIC_flag_out" is used for luma components as before, while for chroma components, if at least one of the calculated LIC parameters for chroma Cb or Cr is default (or close to default), then "LIC_flag_out" is inferred to be false. In a variant, if the calculated LIC parameters for chroma Cb and Cr are both default (or close to default, emb-3), then the chroma component "LIC_flag_out" is inferred to be false.

[0089] Embodiment 6 (embodiment 1, 2 or 3):

[0090] This embodiment Fig.11 Described in.

[0091] "LIC_flag" is associated with "LIC_flag_counter". When combined with emb-1, there is one "LIC_flag_counter[C]" per component. When decoding a CU with an explicitly decoded LIC_flag, the value of "LIC_flag_counter" is initialized. If "LIC_flag=true", "LIC_flag_count" is initialized to "MaxLicCount" (e.g., "MaxLicCount=2") (620). If "LIC_flag=false", "LIC_flag_Counter" is initialized to 0 (630). In merge mode, "LIC_flag_counter" (610) is inherited.

[0092] (a) When combined with emb-2 or emb-3, if the LIC parameters are (close to) default, decrement "LIC_flag_counter" (unless "LIC_flag_counter" is zero) (640).

[0093] A CU decoded in merge mode inherits "LIC_flag_counter". If "LIC_flag_counter=0", LIC is disabled for the CU (650). In a variant, if the CU inherits "LIC_flag", then "LIC_flag_counter" is decremented by (b). One has only (a), only (b), or (a)+(b) (in this case, "LIC_flag_counter" can be decremented twice).

[0094] exist Fig.12 In the example, "MaxLicCount=3".

[0095] a. CUa has "LIC_flag" explicitly decoded and equal to true. The derived LIC parameters are not default, then "LIC_flag_counter" is equal to 3.

[0096] b. Cub is merged and inherits "LIC_flag_counter" from the upper CU (CUa). If the derived LIC parameters are not default, "LIC_flag_counter" remains equal to 3.

[0097] c. CUc is merged and inherits "LIC_flag_counter" from the left CU (CUb). The derived LIC parameters are default, then "LIC_flag_counter" is decremented and equal to 2.

[0098] d. CUd is merged and inherits "LIC_flag_counter" from the left CU (CUc). The derived LIC parameters are default, then "LIC_flag_counter" is decremented and equal to 1.

[0099] e. CUe is merged and inherits "LIC_flag_counter" from the left CU (CUd). The derived LIC parameters are defaulted, then "LIC_flag_counter" is decremented and equal to 0.

[0100] f. CUf is merged and inherits "LIC_flag_counter" from the left CU (CUe). "LIC_flag_counter" is equal to 0, then LIC is disabled ("LIC_flag" = false).

[0101] g. CUg is merged and inherits "LIC_flag_counter" from the left CU (CUf). "LIC_flag_counter" is equal to 0, then LIC is disabled ("LIC_flag" = false).

[0102] Embodiment 7 (embodiment 1):

[0103] In this embodiment, there is no LIC for chroma components. "LIC_flag" and "LIC_flag_out" for chroma components are always inferred to be false.

[0104] Embodiment 8:

[0105] In this embodiment, there is no LIC propagation (inheritance) for chroma components. "LIC_flag" is decoded in the bitstream for luma and chroma components, and LIC is applied as usual. But for chroma components, "LIC_flag_out" is always inferred to be false, regardless of the LIC parameter value. Therefore, the LIC for chroma components is always false for the merged blocks.

[0106] Note that for this embodiment, variations of a single "LIC_flag", one "LIC_flag" per component, or one "LIC_flag" per component type are possible. In the case of a single "LIC_flag", one flag is decoded per CU, and "LIC_flag_out" is always false for chroma components and as previously described for luma components.

[0107] Fig.15 One embodiment of a method 1500 according to the general aspects described herein is shown. The method begins at start block 1501 and control proceeds to block 1510 for determining local illumination compensation information to be used for one or more video components of a video block. Control proceeds from block 1510 to block 1520 to encode the video block using the local illumination compensation information. Control proceeds from block 1520 to block 1530 to indicate in a bitstream whether the illumination compensation information is used for the video block.

[0108] Fig.14 One embodiment of a method 1400 according to the general aspects described herein is shown. The method begins at start block 1401 and control proceeds to block 1410 for parsing a bitstream for syntax information indicating local illumination compensation. Control proceeds from block 1410 to block 1420 for decoding at least one video block in the bitstream based on the syntax to conditionally perform local illumination compensation on components of the one video block.

[0109] Fig.16One embodiment of an apparatus 1600 for compressing, encoding or decoding video using a coding or decoding tool is shown. The apparatus includes a processor 1610 and may be interconnected to a memory 1620 via at least one port. Both the processor 1610 and the memory 1620 may also have one or more additional interconnects to external connections.

[0110] The processor 1610 is also configured to insert or receive information in the bitstream and use various decoding tools to perform compression, encoding or decoding.

[0111] The present application describes a number of aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described as having specificity, and at least in order to illustrate individual characteristics, are usually described in a manner that may sound limited. However, this is for the purpose of describing clearly, and does not limit the application or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide other aspects. In addition, these aspects can also be combined and interchanged with the aspects described in earlier documents.

[0112] The aspects described and contemplated in this application can be implemented in many different forms. Figure 7 , 8 Some embodiments are provided in Figures 1 and 13, but other embodiments may be envisioned and discussed in the accompanying figures. Figure 7 , 8 and 13 do not limit the breadth of implementation. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the methods described, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the methods described.

[0113] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image", "picture" and "frame" are used interchangeably. Usually, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0114] Various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0115] The various methods and other aspects described in this application can be used to modify Figure 7 and Figure 8Modules of the video encoder 100 and decoder 200 shown, such as intra prediction, entropy coding and / or decoding modules (160, 360, 145, 330). In addition, aspects of the present invention are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations, whether pre-existing or developed in the future, and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this application can be used alone or in combination.

[0116] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes, and the described aspects are not limited to these specific values.

[0117] Figure 7 An encoder 100 is shown. Variations of the encoder 100 are contemplated, but for the sake of clarity, the encoder 100 is described below without describing all contemplated variations.

[0118] Before being encoded, the video sequence may undergo a pre-encoding process (101), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0119] In encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is partitioned (102) and processed in units such as CUs. Each unit is encoded using, for example, intra or inter mode. When the unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of intra mode or inter mode to use to encode the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. For example, a prediction residual is calculated by subtracting (110) the predicted block from the original image block.

[0120] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass the transform and quantization, i.e., directly decode the residual without applying a transform or quantization process.

[0121] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. An in-loop filter (165) is applied to the reconstructed image to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0122] Figure 8 2 shows a block diagram of a video decoder 200. In the decoder 200, a bitstream is decoded by decoder elements as described below. The video decoder 200 generally performs the same Figure 7 The encoding pass described in is a decoding pass that is the inverse of the encoding pass described in . The encoder 100 also typically performs video decoding as part of encoding the video data.

[0123] In particular, the input to the decoder includes a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other decoding information. Picture partition information indicates how the picture is partitioned. The decoder can therefore divide (235) the picture according to the decoded picture partition information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined (255) with the prediction block to reconstruct the image block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0124] The decoded picture may further undergo post-decoding processing (285), such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0125] Fig.13A block diagram of an example of a system in which various aspects and embodiments are implemented is shown. System 1000 can be implemented as a device including various components described below, and is configured to perform one or more aspects described herein. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances and servers. The elements of system 1000 can be implemented individually or in combination in a single integrated circuit (IC), multiple ICs and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed on multiple ICs and / or discrete components. In various embodiments, system 1000 is coupled to one or more other systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port communication. In various embodiments, system 1000 is configured to implement one or more aspects described herein.

[0126] The system 1000 includes at least one processor 1010, which is configured to execute instructions loaded therein, for implementing various aspects described herein, for example. The processor 1010 may include embedded memory, input and output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include a non-volatile memory and / or a volatile memory, including but not limited to an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a disk drive, and / or an optical drive. As a non-limiting example, the storage device 1040 may include an internal storage device, an attached storage device (including a removable and non-removable storage device), and / or a network-accessible storage device.

[0127] The system 1000 includes an encoder / decoder module 1030, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module (one or more) that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000 or may be incorporated into the processor 1010 as a combination of hardware and software as known to those skilled in the art.

[0128] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of the various items during the execution of the processes described herein. These stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0129] In some embodiments, memory within the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or a storage device 1040, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, an operating system for a television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations, such as working memory for MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by JVET (Joint Video Experts Group)).

[0130] Input to the elements of system 1000 may be provided through various input devices as shown in block 1130. Such input devices include, but are not limited to, the following: (i) an RF section that receives a radio frequency (RF) signal transmitted over the air, for example, by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Fig.13 Other examples not shown include composite video.

[0131] In various embodiments, the input device of block 1130 has associated corresponding input processing elements known in the art. For example, the RF portion may be associated with elements adapted to: (i) select a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) downconvert the selected signal, (iii) again bandlimit to a narrower frequency band to select a signal frequency band that may be referred to as a channel in some embodiments, (iv) demodulate the downconverted and bandlimited signal, (v) perform error correction, and (vi) demultiplex to select a desired data packet stream. The RF portion of various embodiments includes one or more elements to perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF portion may include a tuner that performs various of these functions, including, for example, downconverting a received signal to a lower frequency (e.g., an intermediate frequency or near baseband frequency) or baseband. In a set-top box embodiment, the RF part and its relevant input processing element receive the RF signal that sends by wired (for example, cable) medium, and filter to the frequency band of expectation again by filtering, down-conversion and perform frequency selection.Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements that perform similar or different functions.Adding element can include and insert element between existing element, for example insert amplifier and analog-to-digital converter.In various embodiments, the RF part comprises antenna.

[0132] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented as needed, for example, in a separate input processing IC or processor 1010. Similarly, various aspects of USB or HDMI interface processing may be implemented as needed, either in a separate interface IC or in processor 1010. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 1010 and encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on output devices.

[0133] The various components of system 1000 may be disposed within an integrated housing in which the various components may be interconnected and transmit data using suitable connection arrangements such as internal buses (including inter-IC (I2C) buses), wiring, and printed circuit boards known in the art.

[0134] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data through the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0135] In various embodiments, data is streamed or otherwise provided to the system 1000 using a wireless network, such as a Wi-Fi network, such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signals of these embodiments are received through a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that transmits data through an HDMI connection of the input block 1130 to provide streaming data to the system 1000. Still other embodiments use an RF connection of the input block 1130 to provide streaming data to the system 1000. As described above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use a wireless network other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0136] System 1000 can provide output signals to various output devices, including display 1100, speaker 1110 and other peripheral devices 1120. Display 1100 of various embodiments includes one or more of, for example, a touch screen display, an organic light emitting diode (OLED) display, a curved display and / or a foldable display. Display 1100 can be used for a television, a tablet computer, a laptop computer, a cellular phone (mobile phone) or other devices. Display 1100 can also be integrated with other components (for example, as in a smart phone), or separately (for example, an external monitor for a laptop computer). In each example of each embodiment, other peripheral devices 1120 include one or more of the following: an independent digital video disk (or digital versatile disk) (both of which can be referred to as DVR), a disk player, a stereo system and / or a lighting system. Various embodiments use one or more peripheral devices 1120, which provide functions based on the output of system 1000. For example, a disk player performs the function of playing the output of system 1000.

[0137] In various embodiments, control signals are transmitted between the system 1000 and the display 1100, speaker 1110, or other peripheral device 1120 using signaling such as AV.link, consumer electronics control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices may be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to the system 1000 via communication interface 1050 using communication channel 1060. The display 1100 and speaker 1110 may be integrated into a single unit in an electronic device (e.g., a television) along with other components of the system 1000. In various embodiments, the display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0138] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may alternatively be separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0139] These embodiments may be implemented by computer software implemented by the processor 1010, or hardware, or a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, as a non-limiting example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 1010 may be of any type suitable for the technical environment and may include, as a non-limiting example, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0140] Various implementations involve decoding. As used in this application, "decoding" may include, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also or alternatively includes a process performed by a decoder of the various implementations described in this application.

[0141] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0142] Various implementations involve encoding. In a manner similar to the above discussion of "decoding", "encoding" as used in this application may include, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such processes also or alternatively include processes performed by encoders of the various implementations described in this application.

[0143] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will become clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0144] Note that the syntax elements used here are descriptive terms. Therefore, they do not exclude the use of other syntax element names.

[0145] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0146] Various embodiments may involve parameter models or rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is usually considered, usually given a constraint on computational complexity. It can be measured by rate-distortion optimization (RDO) metric, or by least mean square (LMS), mean absolute error (MAE) or other such measurements. Rate-distortion optimization is usually formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different methods to solve the rate-distortion optimization problem. For example, these methods can be based on extensive testing of all coding options (including all considered modes or decoding parameter values), with a complete evaluation of their decoding costs and the associated distortion of the reconstructed signal after decoding and decoding. Faster methods can also be used to save decoding complexity, in particular, to calculate approximate distortion based on prediction or prediction residual signals rather than reconstructed signals. A mixture of these two methods can also be used, for example by using approximate distortion only for some possible coding options and full distortion for other coding options. Other methods only evaluate a subset of possible coding options. More generally, many approaches employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete assessment of both coding cost and associated distortion.

[0147] The implementations and aspects described herein may be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the features discussed may also be implemented in other forms (e.g., a device or program). For example, a device may be implemented with appropriate hardware, software, and firmware. The method may be implemented in, for example, a processor, which generally refers to a processing device (including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device). Processors also include communication devices, such as computers, cellular phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate information communication between end users.

[0148] References to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations in various places in this application do not necessarily all refer to the same embodiment.

[0149] Additionally, the present application may refer to "determining" various information. Determining information may include one or more of the following: estimating information, calculating information, predicting information, or retrieving information from a memory, for example.

[0150] Furthermore, the present application may involve "accessing" various information. Accessing information may include one or more of the following: for example, receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0151] Additionally, the present application may refer to "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, during operations such as storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is generally involved in one way or another.

[0152] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selection of only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to include selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to multiple items listed, as will be apparent to one of ordinary skill in this and related arts.

[0153] In addition, as used herein, the word "signaling" especially refers to indicating something to a corresponding decoder. For example, in some embodiments, the encoder signals a specific one of multiple transforms, decoding modes or flags. Thus, in one embodiment, the same transform, parameter or mode is used on both the encoder side and the decoder side. Therefore, for example, the encoder can send (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and other parameters, signaling can be used without sending (implicit signaling) to simply allow the decoder to know and select specific parameters. Bit saving is achieved in various embodiments by avoiding the transmission of any actual function. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to the corresponding decoder. Although the foregoing relates to the verb form of the word "signaling", the word "signal" can also be used as a noun in this article.

[0154] As will be apparent to one of ordinary skill in the art, implementations may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0155] We have described multiple embodiments across various claim categories and types. Features of these embodiments may be provided individually or in any combination. In addition, embodiments may include one or more of the following features, devices, or aspects, alone or in any combination, across various claim categories and types:

[0156] • A bitstream or signal comprising one or more of the described syntax elements, or variations thereof.

[0157] • A bitstream or signal comprising syntax conveying information generated according to any of the described embodiments.

[0158] • Creating and / or sending and / or receiving and / or decoding according to any of the embodiments described.

[0159] • A method, process, apparatus, medium storing instructions, medium storing data, or signal according to any of the described embodiments.

[0160] • Inserting syntax elements in the signaling that enable the decoder to determine the coding mode in a manner corresponding to that used by the encoder.

[0161] • Creating and / or sending and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0162] • A TV, set-top box, cell phone, tablet or other electronic device that performs the transformation method(s) according to any of the described embodiments.

[0163] A TV, set-top box, cell phone, tablet or other electronic device that performs the transformation method(s) determined according to any of the described embodiments and displays (e.g., using a monitor, screen or other type of display) the resulting image.

[0164] A TV, set-top box, cellular phone, tablet or other electronic device that selects, band-limits or tunes (e.g., using a tuner) a channel to receive a signal including an encoded image, and performs a transformation method(s) according to any of the described embodiments.

[0165] • A TV, set-top box, cell phone, tablet or other electronic device that receives over the air (eg, using an antenna) a signal including an encoded image and performs the transformation method(s).

Claims

1. A method comprising: For a current image block in a merge mode, obtaining a local illumination compensation (LIC) flag and a LIC flag counter associated with the LIC flag, wherein the LIC flag indicates whether local illumination compensation is applied to the current image block; In response to determining that the LIC flag indicates that local illumination compensation is applied to the current image block, calculating LIC model parameters for the current image block; In response to determining that the LIC model parameters are close to default values, decrementing the LIC flag counter for the current image block; as well as The current image block is decoded using the LIC model parameters.

2. The method according to claim 1, wherein for the current image block in the merge mode, obtaining a local illumination compensation (LIC) flag and a LIC flag counter associated with the LIC flag comprises: The LIC flag counter is inherited from a candidate block and the LIC flag is derived from the LIC flag counter. 3 . The method of claim 2 , further comprising decoding an index identifying the candidate block in a candidate block list.

4. The method of claim 2, wherein deriving the LIC flag from the LIC flag counter comprises: In response to determining that the LIC flag counter exceeds zero, the LIC flag is set to true.

5. The method according to claim 1, wherein for a current image block in a merge mode, obtaining a local illumination compensation (LIC) flag and a LIC flag counter associated with the LIC flag comprises: For a current image block in a merge mode, a local illumination compensation (LIC) flag of each component of the current image block and an LIC flag counter associated with the LIC flag are obtained.

6. An apparatus comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform: For a current image block in a merge mode, obtaining a local illumination compensation (LIC) flag and a LIC flag counter associated with the LIC flag, wherein the LIC flag indicates whether local illumination compensation is applied to the current image block; In response to determining that the LIC flag indicates that local illumination compensation is applied to the current image block, calculating LIC model parameters for the current image block; In response to determining that the LIC model parameters are close to default values, decrementing the LIC flag counter for the current image block; as well as The current image block is decoded using the LIC model parameters.

7. The apparatus according to claim 6, wherein for the current image block in the merge mode, obtaining a local illumination compensation (LIC) flag and a LIC flag counter associated with the LIC flag comprises: The LIC flag counter is inherited from a candidate block and the LIC flag is derived from the LIC flag counter.

8. The apparatus of claim 7, further comprising decoding an index identifying the candidate block in a candidate block list.

9. The apparatus of claim 7, wherein deriving the LIC flag from the LIC flag counter comprises: In response to determining that the LIC flag counter exceeds zero, the LIC flag is set to true.

10. The apparatus according to claim 6, wherein for a current image block in a merge mode, obtaining a local illumination compensation (LIC) flag and a LIC flag counter associated with the LIC flag comprises: For a current image block in a merge mode, a local illumination compensation (LIC) flag associated with each component of the current image block and a LIC flag counter associated with the LIC flag are obtained.

11. A method comprising: For a current image block in a merge mode, obtaining a local illumination compensation (LIC) flag and a LIC flag counter associated with the LIC flag, wherein the LIC flag indicates whether local illumination compensation is applied to the current image block; In response to determining that the LIC flag indicates that local illumination compensation is applied to the current image block, calculating LIC model parameters for the current image block; In response to determining that the LIC model parameters are close to default values, decrementing the LIC flag counter for the current image block; as well as The current image block is encoded using the LIC model parameters.

12. The method according to claim 11, wherein for a current image block in a merge mode, obtaining a local illumination compensation (LIC) flag and a LIC flag counter associated with the LIC flag comprises: The LIC flag counter is inherited from a candidate block and the LIC flag is derived from the LIC flag counter.

13. The method of claim 12, further comprising encoding an index identifying the candidate block in a candidate block list.

14. The method of claim 12, wherein deriving the LIC flag from the LIC flag counter comprises: In response to determining that the LIC flag counter exceeds zero, the LIC flag is set to true.

15. The method according to claim 11, characterized in that The step of obtaining, for a current image block in a merge mode, a local illumination compensation (LIC) flag and a LIC flag counter associated with the LIC flag comprises: obtaining, for the current image block in a merge mode, a local illumination compensation (LIC) flag of each component of the current image block and a LIC flag counter associated with the LIC flag.

16. An apparatus comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform: For a current image block in a merge mode, obtaining a local illumination compensation (LIC) flag and a LIC flag counter associated with the LIC flag, wherein the LIC flag indicates whether local illumination compensation is applied to the current image block; In response to determining that the LIC flag indicates that local illumination compensation is applied to the current image block, calculating LIC model parameters for the current image block; In response to determining that the LIC model parameters are close to default values, decrementing the LIC flag counter for the current image block; as well as The current image block is encoded using the LIC model parameters.

17. The apparatus according to claim 16, wherein for the current image block in the merge mode, obtaining a local illumination compensation (LIC) flag and a LIC flag counter associated with the LIC flag comprises: The LIC flag counter is inherited from a candidate block and the LIC flag is derived from the LIC flag counter.

18. The apparatus of claim 17, further comprising encoding an index identifying the candidate block in a candidate block list.

19. The apparatus of claim 17, wherein deriving the LIC flag from the LIC flag counter comprises: In response to determining that the LIC flag counter exceeds zero, the LIC flag is set to true.

20. The apparatus according to claim 16, wherein for a current image block in a merge mode, obtaining a local illumination compensation (LIC) flag and a LIC flag counter associated with the LIC flag comprises: A local illumination compensation (LIC) flag associated with each component of a current image block in a merge mode and a LIC flag counter associated with the LIC flag are obtained.

Images