Method and device for video processing and medium

Abstract

Embodiments of the present disclosure provide a solution for video processing. A method for video processing is presented. The method comprises: for a conversion between a current video block of a video and a bitstream of the video, obtaining values of a set of adjustment parameters associated with values of a set of model parameters of a local illumination compensation (LIC) model for encoding and decoding the current video block; updating the values of the set of model parameters based on the values of the set of adjustment parameters; and performing the conversion based on the updated values of the set of model parameters.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 358,463, filed on July 5, 2022, assigned to the assignee, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] Embodiments of the present disclosure generally relate to video processing techniques, and more particularly, to parameter refinement for illumination compensation in video encoding and decoding. Background Art

[0003] Nowadays, digital video capabilities are being applied to all aspects of people's lives. Various types of video compression technologies have been proposed for video encoding / decoding, such as MPEG-2, MPEG-4, ITU-TH.263, ITU-TH.264 / MPEG-4 Part 10 Advanced Video Codec (AVC), ITU-TH.265 High Efficiency Video Codec (HEVC) standard, and Versatile Video Codec (VVC) standard. However, it is generally expected to further improve the encoding and decoding quality of video encoding and decoding technologies. Summary of the invention

[0004] Embodiments of the present disclosure provide a solution for video processing.

[0005] In a first aspect, a method for video processing is provided. The method includes: for conversion between a current video block of a video and a bitstream of the video, obtaining values ​​of a set of adjustment parameters associated with values ​​of a set of model parameters of a local illumination compensation (LIC) model for encoding and decoding the current video block; updating values ​​of a set of model parameters based on the values ​​of the set of adjustment parameters; and performing conversion based on the updated values ​​of the set of model parameters.

[0006] According to the method of the first aspect of the present disclosure, one or more adjustment parameters are used to update the value of at least one model parameter of the LIC model. Compared with conventional solutions, the proposed method can refine the model parameters used for illumination compensation to obtain the optimal model parameters for the current video. In this way, the encoding and decoding quality can be improved.

[0007] In a second aspect, a device for video processing is provided. The device includes a processor and a non-volatile memory having instructions thereon. The instructions, when executed by the processor, cause the processor to perform the method according to the first aspect of the present disclosure.

[0008] In a third aspect, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions, and the instructions enable a processor to execute the method according to the first aspect of the present disclosure.

[0009] In a fourth aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by an apparatus for video processing. The method includes: obtaining values ​​of a set of adjustment parameters associated with values ​​of a set of model parameters of an LIC model for encoding and decoding a current video block of the video; updating values ​​of a set of model parameters based on the values ​​of the set of adjustment parameters; and generating a bitstream based on the updated values ​​of the set of model parameters.

[0010] In a fifth aspect, a method for storing a bitstream of a video is provided. The method includes: obtaining values ​​of a set of adjustment parameters associated with values ​​of a set of model parameters of an LIC model for encoding and decoding a current video block of the video; updating values ​​of a set of model parameters based on the values ​​of the set of adjustment parameters; generating a bitstream based on the updated values ​​of the set of model parameters; and storing the bitstream in a non-transitory computer-readable recording medium.

[0011] This summary is provided to introduce some concepts in a simplified form, which are further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become more apparent through the following detailed description with reference to the accompanying drawings. In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0013] Figure 1 A block diagram showing an example video encoding and decoding system according to some embodiments of the present disclosure is shown;

[0014] Figure 2 A block diagram of a first example video encoder according to some embodiments of the present disclosure is shown;

[0015] Figure 3 A block diagram of an example video decoder according to some embodiments of the present disclosure is shown;

[0016] Figure 4 The positions of spatial merging candidates are shown;

[0017] Figure 5 shows candidate pairs considered for redundancy check of spatial merging candidates;

[0018] Figure 6 A diagram showing motion vector scaling for temporal merging candidates is shown;

[0019] Figure 7Candidate positions for temporal merging candidates C0 and C1 are shown;

[0020] Figure 8 The MMVD search point is shown;

[0021] Fig. 9 A diagram for a symmetric MVD mode is shown;

[0022] Fig.10 An affine motion model based on control points is shown;

[0023] Fig.11 The affine MVF of each sub-block is shown;

[0024] Fig.12 The position of the inherited affine motion predictor is shown;

[0025] Fig.13 Control point motion vector inheritance is shown;

[0026] Fig.14 The positions of candidate positions for the constructed affine merge pattern are shown;

[0027] Fig.15 The use of motion vectors for the proposed combined method is shown;

[0028] Fig.16 Sub-block MV VSB and pixel Δv(i, j) are shown (grey arrows);

[0029] Fig.17A shows the spatially adjacent blocks used by ATVMP;

[0030] Fig. 17B shows the derivation of a sub-CU motion field by applying motion shifts from spatially neighboring blocks and scaling motion information from corresponding co-located sub-CUs;

[0031] Fig.18 shows the extended CU area used in BDOF;

[0032] Fig.19 Decoder side motion vector refinement is shown;

[0033] Fig. 20 An example of GPM segmentation grouped at the same angle is shown;

[0034] Fig.21 Unidirectional prediction MV selection for geometric partitioning mode is shown;

[0035] Fig. 22 An exemplary generation of warping weights w0 using a geometric partitioning pattern is shown;

[0036] Fig.23 The top and left neighboring blocks used in the derivation of CIIP weights are shown;

[0037] Fig.24 The spatial neighboring blocks used to derive spatial merging candidates are shown;

[0038] Fig.25 shows the template matching performed on the search area around the initial MV;

[0039] Fig.26 A diamond-shaped area in the search area is shown;

[0040] Fig. 27 shows the frequency response of the interpolation filter and the VVC interpolation filter at half-pixel phase;

[0041] Fig.28 The template in the reference picture and the reference sample of the template are shown;

[0042] Fig.29 A template of a block with sub-block motion using motion information of a sub-block of a current block and a reference sample of the template are shown;

[0043] Fig.30 A flowchart showing a method for video processing according to an embodiment of the present disclosure; and

[0044] Fig.31 A block diagram of a computing device is shown in which various embodiments of the present disclosure may be implemented.

[0045] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. DETAILED DESCRIPTION

[0046] The principle of the present invention is now described in conjunction with some embodiments. It should be understood that these embodiments are described only for illustrative purposes to help those skilled in the art understand and implement the present invention, and do not constitute any limitation to the scope of the present invention. The disclosure described herein can be implemented in various ways except for the way described below.

[0047] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0048] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but it is not necessary that every embodiment include the particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example, it is considered within the knowledge of those skilled in the art to affect embodiments related to such features, structures, or characteristics, for example, whether or not explicitly described.

[0049] It should be understood that although the terms "first" and "second" etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0050] The terms used herein are used only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include", "comprise", and / or "have", when used herein, specify the presence of stated features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof. Example Environment

[0051] Figure 1 1 is a block diagram illustrating an example video codec system 100 that may utilize the techniques of the present disclosure. As shown, the video codec system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a video encoding device, and the destination device 120 may also be referred to as a video decoding device. In operation, the source device 110 may be configured to generate encoded video data, and the destination device 120 may be configured to decode the encoded video data generated by the source device 110. The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0052] Video source 112 may include a source such as a video capture device. Examples of a video capture device include, but are not limited to, an interface that receives video data from a video content provider, a computer graphics system for generating video data, and / or combinations thereof.

[0053] The video data may include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a code stream. The code stream may include a bit sequence that forms a coded representation of the video data. The code stream may include coded pictures and associated data. The coded picture is a coded representation of the picture. The associated data may include a sequence parameter set, a picture parameter set, and other grammatical structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The coded video data may be directly transmitted to the destination device 120 via the network 130A via the I / O interface 116. The coded video data may also be stored on a storage medium / server 130B for access by the destination device 120.

[0054] The destination device 120 may include an I / O interface 126, a video decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain encoded video data from the source device 110 or the storage medium / server 130B. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to the user. The display device 122 may be integrated with the destination device 120, or may be outside the destination device 120, which is configured to be connected to an external display device interface.

[0055] The video encoder 114 and the video decoder 124 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other existing and / or further standards.

[0056] Figure 2 is a block diagram showing an example of a video encoder 200 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video encoder 114 in the system 100 is shown.

[0057] Video encoder 200 may be configured to implement any or all of the techniques of this disclosure. Figure 2 In the example of , video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video encoder 200. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0058] In some embodiments, the video encoder 200 may include a partitioning unit 201, a prediction unit 202, a residual generation unit 207, a transformation unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transformation unit 211, a reconstruction unit 212, a buffer 213 and an entropy coding and decoding unit 214, and the prediction unit 202 may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205 and an intra-frame prediction unit 206.

[0059] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, where at least one reference picture is a picture where the current video block is located.

[0060] Furthermore, although some components (such as the motion estimation unit 204 and the motion compensation unit 205) may be integrated, for the purpose of explanation, these components are described in detail below. Figure 2 are represented separately in the example.

[0061] The partitioning unit 201 may partition a picture into one or more video blocks. The video encoder 200 and the video decoder 300 may support various video block sizes.

[0062] The mode selection unit 203 may select one of a plurality of codec modes (intra-frame coding or inter-frame coding), for example, based on the error result, and provide the generated intra-frame codec block or inter-frame codec block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the codec block for use as a reference picture. In some examples, the mode selection unit 203 may select a combination of intra-frame and inter-frame prediction (CIIP) modes, where the prediction is based on an inter-frame prediction signal and an intra-frame prediction signal. In the case of inter-frame prediction, the mode selection unit 203 may also select a resolution for the motion vector (e.g., sub-pixel precision or integer pixel precision) for the block.

[0063] To perform inter-frame prediction on the current video block, the motion estimation unit 204 may generate motion information for the current video block by comparing the current video block with one or more reference frames from the buffer 213. The motion compensation unit 205 may determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from the buffer 213 other than the picture associated with the current video block.

[0064] The motion estimation unit 204 and the motion compensation unit 205 may perform different operations on the current video block, for example, depending on whether the current video block is in an I slice, a P slice, or a B slice. As used herein, an "I slice" may refer to a portion of a picture consisting of macroblocks, all of which are based on macroblocks within the same picture. Furthermore, as used herein, in some aspects, a "P slice" and a "B slice" may refer to a portion of a picture consisting of macroblocks that are independent of macroblocks in the same picture.

[0065] In some examples, the motion estimation unit 204 may perform unidirectional prediction on the current video block, and the motion estimation unit 204 may search the reference pictures of list 0 or list 1 to find the reference video block for the current video block. The motion estimation unit 204 may then generate a reference index and a motion vector, the reference index indicating the reference picture in list 0 or list 1 containing the reference video block, and the motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 204 may output the reference index, the prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0066] Alternatively, in other examples, the motion estimation unit 204 may perform bidirectional prediction on the current video block. The motion estimation unit 204 may search the reference pictures in list 0 for a reference video block for the current video block, and may also search the reference pictures in list 1 for another reference video block for the current video block. The motion estimation unit 204 may then generate a plurality of reference indexes and a plurality of motion vectors, the plurality of reference indexes indicating a plurality of reference pictures in list 0 and list 1 containing a plurality of reference video blocks, and the plurality of motion vectors indicating a plurality of spatial displacements between the plurality of reference video blocks and the current video block. The motion estimation unit 204 may output the plurality of reference indexes and the plurality of motion vectors of the current video block as motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the plurality of reference video blocks indicated by the motion information of the current video block.

[0067] In some examples, motion estimation unit 204 may output a complete set of motion information for use in a decoding process by a decoder. Alternatively, in some embodiments, motion estimation unit 204 may signal motion information of a current video block with reference to motion information of another video block. For example, motion estimation unit 204 may determine that motion information of a current video block is sufficiently similar to motion information of a neighboring video block.

[0068] In one example, motion estimation unit 204 may indicate to video decoder 300 a value in a syntax structure associated with the current video block that indicates that the current video block has the same motion information as another video block.

[0069] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0070] As discussed above, the video encoder 200 may signal motion vectors in a predictive manner.Two examples of prediction signaling techniques that may be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and merge mode signaling.

[0071] The intra prediction unit 206 may perform intra prediction on the current video block. When the intra prediction unit 206 performs intra prediction on the current video block, the intra prediction unit 206 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a prediction video block and various syntax elements.

[0072] The residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) the prediction video block(s) of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample portions of samples in the current video block.

[0073] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 207 may not perform a subtraction operation.

[0074] Transform processing unit 208 may generate one or more transform coefficient video blocks for a current video block by applying one or more transforms to the residual video block associated with the current video block.

[0075] After transform processing unit 208 generates a transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0076] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current video block for storage in the buffer 213.

[0077] After reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[0078] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the entropy coding unit 214 receives data, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy coded data and output a bitstream including the entropy coded data.

[0079] Figure 3 is a block diagram showing an example of a video decoder 300 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video decoder 124 in the system 100 is shown.

[0080] Video decoder 300 may be configured to perform any or all of the techniques of this disclosure. Figure 3 In the example of , video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video decoder 300. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0081] exist Figure 3 In the example of , the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, and a reconstruction unit 306 and a buffer 307. In some examples, the video decoder 300 can perform a decoding process that is generally opposite to the encoding process described with respect to the video encoder 200.

[0082] The entropy decoding unit 301 can retrieve the encoded code stream. The encoded code stream may include entropy encoded video data (e.g., encoded video data blocks). The entropy decoding unit 301 can decode the entropy encoded video data, and the motion compensation unit 302 can determine motion information from the entropy decoded video data, the motion information including motion vectors, motion vector precision, reference picture list indexes, and other motion information. The motion compensation unit 302 can determine the information, for example, by performing AMVP and merge mode. AMVP is used, including deriving several most likely candidates based on data and reference pictures of adjacent PBs. The motion information typically includes horizontal and vertical motion vector displacement values, one or two reference picture indexes, and in the case of prediction areas in B strips, also includes an identification of which reference picture list is associated with each index. As used herein, in some aspects, "merge mode" may refer to deriving motion information from blocks that are adjacent in space or time.

[0083] The motion compensation unit 302 may generate motion compensated blocks, possibly performing interpolation based on interpolation filters.Identifiers for the interpolation filters used with sub-pixel precision may be included in the syntax elements.

[0084] The motion compensation unit 302 may calculate interpolated values ​​for sub-integer pixels of a reference block using interpolation filters used by the video encoder 200 during encoding of the video block. The motion compensation unit 302 may determine the interpolation filters used by the video encoder 200 based on received syntax information, and the motion compensation unit 302 may use the interpolation filters to generate a prediction block.

[0085] The motion compensation unit 302 may use at least part of the syntax information to determine the size of blocks used to encode (multiple) frames and / or (multiple) slices of the encoded video sequence, partition information describing how each macroblock of a picture of the encoded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-frame coding block, and other information for decoding the encoded video sequence. As used herein, in some aspects, a "slice" may refer to a data structure that can be decoded independently of other slices of the same picture in terms of entropy coding, signal prediction, and residual signal reconstruction. A slice may be an entire picture, or it may be a region of a picture.

[0086] The intra prediction unit 303 may use, for example, an intra prediction mode received in a bitstream to form a prediction block from spatially neighboring blocks. The inverse quantization unit 304 inversely quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 305 applies an inverse transform.

[0087] The reconstruction unit 306 may obtain the decoded block, for example, by adding the residual block to the corresponding prediction block generated by the motion compensation unit 302 or the intra prediction unit 303. If necessary, a deblocking filter may also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in a buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction, and the buffer 307 also generates the decoded video for presentation on a display device.

[0088] Some example embodiments of the present disclosure will be described in detail below. It should be noted that the section titles used in this document are for ease of understanding, and the embodiments disclosed in the section are not limited to that section. In addition, although some embodiments are described with reference to general video codecs or other specific video codecs, the disclosed technology is also applicable to other video coding and decoding technologies. In addition, although some embodiments describe the video encoding steps in detail, it should be understood that the corresponding decoding steps of canceling the encoding will be implemented by the decoder. In addition, the term video processing includes video coding or compression, video decoding or decompression, and video transcoding, in which video pixels are represented from one compression format to another compression format or at different compression bit rates. 1. Overview The present disclosure relates to video coding techniques. Specifically, it relates to inter-frame prediction in video coding with special emphasis on sequences with illumination changes. These ideas can be applied to any image / video coding standard or non-standard image / video codec, such as next generation image / video coding standards, either alone or in various combinations. 2. Introduction Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, ISO / IEC developed MPEG-1 and MPEG-4 Visual, and the two organizations jointly developed H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Codec (AVC) and H.265 / HEVC (ITU-T and ISO / IEC, "High Efficiency Video Codec", Rec. ITU-T H.265 | ISO / IEC 23008-2 (Official)) standards. Since H.262, video codec standards have been based on a hybrid video codec structure that uses temporal prediction plus transform coding. To explore future video codec technologies beyond HEVC, VCEG and MPEG jointly founded the Joint Video Exploration Team (JVET) in 2015. The JVET meetings are held simultaneously every quarter, and the new video codec standard was officially named the Versatile Video Codec (VVC) at the JVET meeting in April 2018, and the first version of the VVC Test Model (VTM) was released at that time. The VVC working draft and the test model VTM are updated after each meeting. The VVC project achieved technical completion (FDIS) at the July 2020 meeting. ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC1 / SC29 / WG11) are studying the potential need for standardization of future video codec technologies, which have compression capabilities that significantly exceed the current VVC standard. This future standardization action may take the form of additional extensions to VVC or a completely new standard. The group works together to evaluate compression technology designs proposed by experts in the field. The first exploratory experiment (EE) was established during the JVET meeting named Enhanced Compression Model (ECM) from January 6 to 15, 2021. 2.1 Inter-frame prediction in VVC For each inter-predicted CU, motion parameters consisting of motion vector, reference picture index and reference picture list usage index, as well as additional information required by the new codec features of VVC for sample generation for inter-prediction. Motion parameters can be signaled explicitly or implicitly. When a CU is coded and decoded in skip mode, the CU is associated with one PU and has no significant residual coefficients, no coded motion vector increments or reference picture indexes. Merge mode is specified, whereby the motion parameters of the current CU are obtained from neighboring CUs, which include spatial candidates and temporal candidates, as well as additional arrangements introduced in VVC. Merge mode can be applied to any inter-predicted CU, not just skip mode. An alternative to merge mode is explicit transmission of motion parameters, where motion vectors, corresponding reference picture indexes for each reference picture list and reference picture list usage flags, as well as other required information, are explicitly signaled for each CU. In addition to the inter-frame coding features in HEVC, VVC includes several new and refined inter-frame prediction coding tools listed below: - Extended combined forecast; - High-precision (1 / 16 pixel) motion compensation and motion vector storage; -Merge mode with MVD (MMVD); - Symmetric MVD (SMVD) signaling; -Affine motion compensated prediction; - Sub-block based temporal motion vector prediction (SbTMVP); - Adaptive Motion Vector Resolution (AMVR); - Bidirectional prediction with CU level weight (BCW); - Bidirectional Optical Flow (BDOF); - Decoder side motion vector refinement (DMVR); - Geometric Partitioning Mode (GPM); - Combined inter- and intra-frame prediction (CIIP); -Reference image resampling. The following text provides details of those inter prediction methods specified in VVC. 2.1.1 Extended Merger Forecast In VVC, the merge candidate list is constructed by including the following five types of candidates in order: 1) Spatial MVP from spatially neighboring CUs. 2) Time MVP from the same CU. 3) History-based MVP from FIFO table. 4) Average MVP in pairs. 5) Zero MV. The size of the merge list is signaled in the sequence parameter set header, and the maximum allowed size of the merge list is 6. For each CU code in merge mode, the index of the best merge candidate is encoded and decoded using truncated unary binarization (TU). The first bin of the merge index is encoded and decoded using context, while bypass encoding and decoding is used for other bins. The derivation process of merge candidates for each category is provided in this section. As in HEVC, VVC also supports parallel derivation of merge candidate lists for all CUs within a certain size region. 2.1.1.1 Spatial Candidate Export The derivation of spatial merge candidates in VVC is the same as that in HEVC, except that the positions of the first two merge candidates are swapped. Figure 4 Among the candidates at the positions shown, up to four merge candidates are selected. The derivation order is B0, A0, B1, A1, and B2. Position B2 is considered only when one or more CUs at positions B0, A0, B1, and A1 are not available (for example, because it belongs to another slice or tile) or is intra-coded. After the candidate at position A1 is added, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving the coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only the possible candidate pairs are considered. Figure 5 The pairs are linked by arrows in , and candidates are added to the list only if the corresponding candidates used for redundancy checking do not have the same motion information. 2.1.1.2 Time Candidate Export In this step, only one candidate is added to the list. In particular, in the derivation of this temporal merge candidate, the scaled motion vector is derived based on the co-located CU belonging to the co-located reference picture. The reference picture list and reference index to be used for deriving the co-located CU are explicitly signaled in the slice header. Figure 6 As shown by the dotted line in , the scaled motion vector for the temporal merge candidate is obtained, which is scaled from the motion vector of the co-located CU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero. like Figure 7 As shown, the position of the temporal candidate is selected between candidates C0 and C1. If the CU at position C0 is not available, intra-coded, or outside the current row of the CTU, position C1 is used. Otherwise, position C0 is used in the derivation of the temporal merge candidate. 2.1.1.3 History-based Merge Candidate Export History-based MVP (HMVP) merge candidates are added to the merge list after spatial MVP and TMVP. In this method, the motion information of the previous codec block is stored in a table and used as the MVP of the current CU. A table with multiple HMVP candidates is maintained during the encoding / decoding process. The table is reset (cleared) when a new CTU row is encountered. Whenever there is a non-sub-block inter-coded CU, the associated motion information is added as a new HMVP candidate to the last entry of the table. The HMVP list size S is set to 6, which indicates that up to 5 history-based MVP (HMVP) candidates can be added to the table. When a new motion candidate is inserted into the table, a constrained first-in, first-out (FIFO) rule is used, where a redundancy check is first applied to find if there is an identical HMVP in the table. If found, the identical HMVP is removed from the table, and then all subsequent HMVP candidates are moved forward, and the identical HMVP is inserted into the last entry of the table. HMVP candidates can be used in the merge candidate list construction process. Check the nearest HMVP candidates in the table in order and insert them into the candidate list after the TMVP candidate. Apply redundancy check to spatial merge candidates or temporal merge candidates for HMVP candidates. To reduce the number of redundant checking operations, the following simplifications are introduced: 1. The last two entries in the table are redundancy checked as A1 and B1 space candidates respectively. 2. Once the total number of available merge candidates reaches the maximum allowed merge candidate minus 1, the merge candidate list construction process from HMVP is terminated. 2.1.1.4 Pairwise Average Merging Candidate Derivation Using the first two merge candidates, a pairwise average candidate is generated by averaging the predefined candidate pairs in the existing merge candidate list. The first merge candidate is defined as p0Cand, and the second merge candidate can be defined as p1Cand. The average motion vector is calculated for each reference list separately according to the availability of motion vectors of p0Cand and p1Cand. If two motion vectors are available in one list, they are averaged even if they point to different reference pictures, and their reference pictures are set to one of p0Cand; if only one motion vector is available, the motion vector is used directly; if no motion vector is available, this list is kept invalid. In addition, if the half-pixel interpolation filter index of p0Cand and p1Cand is different, it is set to 0. When the merge list is not full after adding pairwise average merge candidates, zero MVPs will be inserted at the end until the maximum number of merge candidates is encountered. 2.1.1.5 Merge Estimation Area The merge estimation region (MER) allows the merge candidate list to be derived independently for a CU in the same merge estimation region (MER). For generating the merge candidate list for the current CU, candidate blocks in the same MER as the current CU are not included. In addition, the update process for the history-based motion vector predictor candidate list is updated only when (xCb+cbWidth)>>Log2ParMrgLevel is greater than xCb>>Log2Par MrgLevel and (yCb+cbHeight)>>Log2parMrglevel is greater than (yCb>>Log2ParMrgLevel), and where (xCb, yCb) is the top left luma sample position of the current CU in the picture and (cbWidth, cbHeight) is the CU size. The MER size is selected at the encoder side and signaled in the sequence parameter set as log2_parallel_merge_level_minus2. 2.1.2 High-precision (1 / 16 pixel) motion compensation and motion vector storage VVC increases MV accuracy to 1 / 16 luma sample to improve prediction efficiency for slow motion video. This higher motion accuracy is particularly helpful for video content with local changes and non-translational motion, such as in the case of affine mode. For fractional position sample generation with higher MV accuracy, HEVC's 8-tap luma interpolation filter and 4-tap chroma interpolation filter are extended to 16 phases for luma and 32 phases for chroma. This extended filter set is applied to the MC process of inter-coded CUs except CUs in affine mode. For affine mode, a set of 6-tap luma interpolation filters with 16 phases is used for lower computational complexity and memory bandwidth savings. In VVC, the highest precision that motion vectors for non-affine CUs are explicitly signaled is quarter luma sample. In some inter prediction modes, such as affine mode, motion vectors can be signaled with 1 / 16 luma sample precision. In all inter-coded CUs with implicitly inferred MVs, MVs are derived with 1 / 16 luma sample precision, and motion compensated prediction is performed with 1 / 16 sample precision. For intra-motion domain storage, all motion vectors are stored with 1 / 16 luma sample precision. For the temporal motion domain storage used by TMVP and SbTVMP, motion domain compression is performed at an 8x8 size granularity compared to the 16x16 size granularity in HEVC. 2.1.3 Merge Mode with MVD (MMVD) In addition to the merge mode that uses implicitly derived motion information directly for prediction sample generation of the current CU, a merge mode with motion vector difference (MMVD) is introduced in VVC. The MMVD flag is signaled immediately after the regular merge flag is sent to specify whether the MMVD mode is used for the CU. In MMVD, after a merge candidate is selected, it is further refined by MVD information transmitted by signal. Further information includes a merge candidate flag, an index specifying the magnitude of motion, and an index indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the MV basis. The mmvd candidate flag is transmitted by signal to specify which one between the first and second merge candidates is used. The distance index specifies the motion magnitude information and indicates a predefined offset from the starting point. Figure 8 As shown, the offset is added to the horizontal component or the vertical component of the starting MV. The relationship between the distance index and the predefined offset is specified in Table 3-6. Table 3-6 - Relationship between distance index and predefined offset The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions, as shown in Table 3-7. Note that the meaning of the MVD symbol can change depending on the information of the starting MV. When the starting MV is a non-predicted MV or a bidirectionally predicted MV, where both lists point to the same side of the current picture (i.e., both referenced POCs are greater than the POC of the current picture, or both are less than the POC of the current picture), the symbol in Table 3-7 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectionally predicted MV, where the two MVs point to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), and the difference in POC in List 0 is greater than the difference in POC in List 1, the symbol in Table 3-7 specifies the sign of the MV offset added to the MV component of the starting MV of List 0, and the sign for List 1 MV has the opposite value. Otherwise, if the difference in POC in List 1 is greater than that in List 0, the symbol in Table 3-7 specifies the sign of the MV offset added to the List 1 MV component of the starting MV and the sign for List 0 MV has the opposite value. The MVD is scaled according to the difference in POC in each direction. If the POC difference in both lists is the same, no scaling is required. Otherwise, if the POC difference in list 0 is greater than the POC difference in list 1, the MVD for list 1 is scaled by defining the POC difference of L0 as td and the POC difference of L1 as tb, as Figure 7 If the POC difference of L1 is greater than L0, the MVD for list 0 is scaled in the same way. If the starting MV is unidirectionally predicted, the MVD is added to the available MVs. Table 3-7 - Signs of MV offsets specified by direction index Directions to IDX 00 01 10 11 x-axis + - N / A N / A y-axis N / A N / A + - 2.1.4 Symmetric MVD Encoding and Decoding In VVC, in addition to normal uni-prediction and bi-prediction mode MVD signaling, symmetric MVD mode for bi-prediction MVD signaling is applied. In symmetric MVD mode, the motion information including reference picture indexes of both list-0 and list-1 and MVD of list-1 is not signaled but derived. The decoding process of the symmetric MVD mode is as follows: 1. At the stripe level, the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 are derived as follows: -If mvd_11zero_flag is 1, BiDirPredFlag is set equal to 0. Otherwise, if the nearest reference picture in list-0 and the nearest reference picture in list-1 form a preceding pair of reference pictures or a following pair of reference pictures, BiDirPredFlag is set to 1, and both list-0 reference pictures and list-1 reference pictures are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. 2. At CU level, if the CU is bi-directionally predicted and BiDirPredFlag is equal to 1, the symmetry mode flag indicating whether the symmetry mode is used is explicitly signaled. When the symmetric mode flag is true, only mvp_10_flag, mvp_11_flag and MVD0 are explicitly signaled. The reference index for list-0 and list-1 are set equal to the reference picture pair, respectively. MVD1 is set equal to (-MVD0). The final motion vector is shown below. {(mvx0,mvy0)=(mvpx0+mvdx0,mvpy0+mvdy0)(mvx1,mvy1)=(mvpx1 -mvdx0,mvpy1-mvdy0) (3-14) In the encoder, symmetric MVD motion estimation starts with initial MV evaluation. A set of initial MV candidates includes MVs obtained from unidirectional prediction search, MVs obtained from bidirectional prediction search, and MVs from the AMVP list. The one with the lowest rate-distortion cost is selected as the initial MV for symmetric MVD motion search. 2.1.5 Affine Motion Compensated Prediction In HEVC, motion compensated prediction (MCP) only applies the translational motion model. In the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensated prediction is applied. Fig.10 As shown, the affine motion field of a block is described by motion information of two control points (4 parameters) or three control point motion vectors (6 parameters). For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as: For the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as: Where (mv 0x , mv 0y ) is the motion vector of the upper left control point, (mv 1x , mv 1y ) is the motion vector of the upper right control point, and (mv 2x , mv 2y ) is the motion vector of the lower left control point. To simplify motion compensated prediction, block-based affine transform prediction is applied. To derive the motion vector for each 4×4 luminance subblock, Fig.11 As shown, the motion vector of the center sample of each subblock is calculated according to the above equation and rounded to 1 / 16 fractional precision. Then the motion compensated interpolation filter is applied to generate a prediction of each subblock with a derived motion vector. The subblock size of the chroma component is also set to 4×4. The MV of a 4×4 chroma subblock is calculated as the average of the MVs of the four corresponding 4×4 luminance subblocks. Like translational motion inter prediction, there are two affine motion inter prediction modes: affine merge mode and affine AMVP mode. 2.1.5.1 Affine Merge Prediction AF_MERGE mode can be applied to CUs with width and height greater than or equal to 8. In this mode, the CPMV of the current CU is generated based on the motion information of spatially neighboring CUs. There can be up to five CPMVP candidates, and the index is transmitted by signal to indicate the CPMVP used for the current CU. The following three types of CPVM candidates are used to form the affine merge candidate list: - Inherited affine merge candidates inferred from the CPMV of neighboring CUs; - Affine merged candidate CPMVP constructed using the translation MV of the neighboring CU; -Zero MV. In VVC, there are at most two inherited affine candidates, which are derived from the affine motion models of neighboring blocks, one from the left neighboring CU and one from the upper neighboring CU. Fig.12 As shown. For the left predictor, the scan order is A0->A1, and for the upper predictor, the scan order is B0->B1->B2. Only the first successor candidate from each side is selected. Pruning check between two successor candidates is not performed. When a neighboring affine CU is identified, its control point motion vector is used to derive the CPMVP candidate in the affine merge list of the current CU. Fig.13 As shown, if the neighboring lower left block A is encoded and decoded in affine mode, the motion vectors v2, v3 and v4 of the upper left corner, upper right corner and lower left corner of the CU containing block A are obtained. When block A is encoded and decoded with a 4-parameter affine model, two CPMVs of the current CU are calculated based on v2 and v3. When block A is encoded and decoded with a 6-parameter affine model, three CPMVs of the current CU are calculated based on v2, v3 and v4. Constructing affine candidates means that candidates are constructed by combining the neighboring translation motion information of each control point. Fig.14 The specified spatial neighbors and the specified temporal neighbors are exported as shown. CPMV k (k=1, 2, 3, 4) indicates the kth control point. For CPMV1, check B2->B3->A2 blocks, and the MV of the first available block is used. For CPMV2, check B1->B0 blocks, and for CPMV3, check A1->A0 blocks. For TMVP, if available, use TMVP as CPMV4. After the MVs of the four control points are obtained, an affine merge candidate is constructed based on the motion information. The following combinations of control point MVs are used to construct in order: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3} A combination of 3 CPMVs constructs a 6-parameter affine merge candidate, and a combination of 2 CPMVs constructs a 4-parameter affine merge candidate. To avoid the motion scaling process, the relevant combination of control point MVs is discarded if the reference indices of the control points are different. After the inherited affine merge candidates and the constructed affine merge candidates are checked, if the list is still not full, a zero MV is inserted at the end of the list. 2.1.5.2 Affine AMVP Prediction Affine AMVP mode can be applied to CUs with width and height greater than or equal to 16. A CU-level affine flag is signaled in the bitstream to indicate whether affine AMVP mode is used, and then another flag is signaled to indicate whether it is 4-parameter affine or 6-parameter affine. In this mode, the difference between the CPMVP of the current CU and their predictor CPMVP is signaled in the bitstream. The affine AVMP candidate list size is 2, and it is generated in order using the following four types of CPVM candidates: - Inherited affine AMVP candidates inferred from the CPMV of neighboring CUs; - A constructed affine AMVP candidate CPMVP derived using the translation MV of the neighboring CU; -Translated MV from neighboring CU; -Zero MV. The order in which inherited affine AMVP candidates are checked is the same as that of inherited affine merge candidates. The only difference is that for AVMP candidates, only affine CUs with the same reference picture as in the current block are considered. No pruning process is applied when inserting the inherited affine motion predictor into the candidate list. The constructed AMVP candidates are Fig.14 The specified spatial neighbors shown are derived. The same check order as in the affine merge candidate construction is used. In addition, the reference picture index of the neighboring blocks is also checked. The first block in the check order is used, which is inter-coded and has the same reference picture as in the current CU. There is only one. When the current CU is coded with a 4-parameter affine mode and both mv0 and mv1 are available, they are added as a candidate in the affine AMVP list. When the current CU is coded with a 6-parameter affine mode and all three CPMVs are available, they are added as a candidate in the affine AMVP list. Otherwise, the constructed AMVP candidate is set to unavailable. If the affine AMVP list candidate is still less than 2 after the inherited affine AMVP candidate and the constructed AMVP candidate are checked, mv0, mv1, and mv2 are added in sequence as translation MVs to predict all control point MVs of the current CU when available. Finally, if the affine AMVP list is still not full, the zero MV is used to fill the affine AMVP list. 2.1.5.3 Affine Motion Information Storage In VVC, the CPMV of the affine CU is stored in a separate buffer. The stored CPMV is only used to generate the inherited CPMV in the affine merge mode and the inherited CPMV in the affine AMVP mode for the most recently coded CU. The sub-block MV derived from the CPMV is used for motion compensation, merging of translation MVs / MV derivation of AMVP lists, and deblocking. To avoid picture line buffering for additional CPMV, the affine motion data inheritance from the CU of the upper CTU is handled differently from the inheritance from the normal neighboring CU. If the candidate CU for affine motion data inheritance is in the upper CTU row, the lower left and lower right sub-block MVs in the line buffer are used for affine MVP derivation instead of CPMV. In this way, CPMV is only stored in the local buffer. If the candidate CU is 6-parameter affine coded, the affine model is downgraded to a 4-parameter model. Fig.15 As shown, along the top CTU boundary, the bottom left and bottom right sub-block motion vectors of the CU are used for affine inheritance of the CU in the bottom CTU. 2.1.5.4 Prediction Refinement with Optical Flow for Affine Mode Compared with pixel-based motion compensation, sub-block-based affine motion compensation can save memory access bandwidth and reduce computational complexity, but at the expense of prediction accuracy. In order to achieve finer motion compensation granularity, prediction refinement using optical flow (PROF) is used to refine sub-block-based affine motion compensation predictions without increasing the memory access bandwidth of motion compensation. In VVC, after sub-block-based affine motion compensation is performed, the brightness prediction samples are refined by adding the differences derived from the optical flow equation. PROF is described as the following four steps: Step 1) Sub-block based affine motion compensation is performed to generate sub-block predictions. Step 2) Using a 3-tap filter [-1, 0, 1], the spatial gradient g of the sub-block prediction x (i, j) and g y (i, j) is calculated at each sample position. The gradient calculation is exactly the same as the gradient calculation in BDOF. g x (i,j)=(I(i+1,j)>>shift1)-(I(i-1,j)>>shift1) (3-17) gy (i,j)=(I(l,j+1)>>shift1)-(I(i,j-1)>>shift1) (3-18) shift1 is used to control the precision of the gradient. The sub-block (i.e. 4x4) prediction is extended by one sample on each side of the gradient calculation. To avoid extra memory bandwidth and extra interpolation calculations, these extended samples on the extended boundary are copied from the nearest integer pixel position in the reference picture. Step 3) The brightness prediction refinement is calculated by following the optical flow equation. ΔI(i, j) = g x (i, j)*Δv x (i, j)+g y (i, j)*Δv y (i, j) (3-19) where Δv(i, j) is the difference between the sample MV calculated for the sample position (i, j), denoted as v(i, j), and the sub-block MV of the sub-block to which the sample (i, j) belongs, as Fig.16 As shown in the figure, Δv(i, j) is quantized in units of 1 / 32 brightness sample accuracy. Since the affine model parameters and the sample positions relative to the subblock center do not change from subblock to subblock, Δv(i, j) can be calculated for the first subblock and reused for other subblocks in the same CU. Let dx(i, j) and dy(i, j) be the distance from the sample position (i, j) to the subblock center (x SB ,y SB )’s horizontal and vertical offsets, Δv(x, y) can be derived by the following equations: {dx(i,j)=iX SB dy(i, j) = jy SB (3-20) {Δv x (i,j)=C*dx(i,j)+D*dy(i,j)Δv y (i,j)=E*dx(i,j)+F*dy(i,j) (3-21) To maintain accuracy, the sub-block (x SB ,y SB ) is calculated as ((W SB -1) / 2, (H SB -1) / 2), where W SB and H SB are the width and height of the sub-block respectively. For the 4-parameter affine model, For the 6-parameter affine model, Where (v 0x , v 0y ), (v 1x , v 1y ), (v 2x , v 2y ) are the control point motion vectors of the upper left, upper right and lower left, and w and h are the width and height of the CU. Step 4) Finally, the luma prediction refinement ΔI(i,j) is added to the sub-block prediction I(i,j). The final prediction I' is generated as follows. I′(i,j)=I(i,j)+ΔI(i,j) PROF is not applicable to affine codec CUs in two cases: 1) all control point MVs are the same, which indicates that the CU has only translational motion; 2) the affine motion parameters are larger than the specified limit, because the sub-block based affine MC is downgraded to CU based MC to avoid large memory access bandwidth requirements. A fast codec method is applied to reduce the codec complexity of affine motion estimation using PROF. PROF is not applied to the affine motion estimation stage in the following two cases: a) If the CU is not a root block and the parent block of the CU does not select the affine mode as its best mode, PROF is not applied because the possibility that the current CU selects the affine mode as the best mode is low; b) If the magnitudes of the four affine parameters (C, D, E, F) are all less than a predefined threshold and the current picture is not a low-latency picture, PROF is not applied because the improvement introduced by PROF for this case is small. In this way, affine motion estimation using PROF can be accelerated. 2.1.6 Sub-block based temporal motion vector prediction (Sb TMVP) VVC supports a sub-block-based temporal motion vector prediction (SbTMVP) method. Similar to the temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the co-located picture to improve the motion vector prediction and merge mode for the CU in the current picture. The same co-located picture used by TMVP is also used for SbTMVP. SbTMVP differs from TMVP in the following two main aspects: -TMVP predicts motion at CU level, but SbTMVP predicts motion at sub-CU level; -TMVP obtains the temporal motion vector from the co-located block in the co-located picture (the co-located block is the bottom-right or center block relative to the current CU), while SbTMVP applies motion shifting before obtaining the temporal motion information from the co-located picture, where the motion shifting is obtained from the motion vector of one of the spatial neighboring blocks from the current CU. SbTMVP process Fig.17A and Fig. 17B As shown. SbTMVP predicts the motion vector of the sub-CU in the current CU in two steps. In the first step, Fig.17A The spatial neighbors A1 in are checked. If A1 has a motion vector that uses the co-located picture as its reference picture, then that motion vector is selected as the motion shift to be applied. If no such motion is identified, the motion offset is set to (0, 0). In a second step, the motion shift identified in step 1 is applied (ie, added to the coordinates of the current block in the current picture 1820) to Fig. 17B The co-located picture shown obtains sub-CU level motion information (motion vector and reference index). Fig. 17B The example in assumes that the motion shift is set to the motion of block A1. Then, for each sub-CU, the motion information of the corresponding block in the co-located picture (the minimum motion grid covering the center sample) is used to derive the motion information for the sub-CU. After the motion information of the co-located sub-CU is identified, it is converted into a motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference picture of the temporal motion vector with the reference picture of the current CU. In VVC, a sub-block based merge list containing a combination of both SbTMVP candidates and affine merge candidates is used for signaling of sub-block based merge mode. The SbTMVP mode is enabled / disabled by a sequence parameter set (SPS) flag. If the SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry in the list of sub-block based merge candidates and is followed by the affine merge candidate. The size of the sub-block based merge list is signaled in the SPS, and the maximum allowed size of the sub-block based merge list is 5 in VVC. The sub-CU size used in SbTMVP is fixed to 8x8. Like the affine merge mode, the SbTMVP mode is only applicable to CUs whose width and height are greater than or equal to 8. The encoding logic of the additional SbTMVP merge candidate is the same as that for other merge candidates, ie, for each CU in a P slice or a B slice, an additional RD check is performed to decide whether to use the SbTMVP candidate. 2.1.7 Adaptive Motion Vector Resolution (AMVR) In HEVC, when use_integer in the slice header is equal to 0, the motion vector difference () (between the CU's motion vector and the predicted motion vector) is transmitted by signal in units of quarter luma samples. In VVC, the CU-level adaptive motion vector resolution (AMVR) scheme is introduced. AMVR allows the MVD of a CU to be encoded and decoded with different precisions. Depending on the mode for the current CU (normal AMVP mode or affine AVMP mode), the MVD of the current CU can be adaptively selected as follows: - Normal AMVP mode: quarter-brightness-sample, half-brightness-sample, whole-brightness-sample or quad-brightness-sample. - Affine AMVP modes: quarter-luminance-sample, whole-luminance-sample, or 1 / 16-luminance-sample. The CU-level MVD resolution indication is conditionally signaled if the current CU has at least one non-zero MVD component.If all MVD components (ie, both horizontal and vertical MVD for reference list L0 and reference list L1) are zero, a quarter-luminance-sample MVD resolution is inferred. For a CU with at least one non-zero MVD component, a first flag is transmitted by a signal to indicate whether quarter-luminance-sample MVD precision is used for the CU. If the first flag is 0, no further signaling is required, and quarter-luminance-sample MVD precision is used for the current CU. Otherwise, a second flag is transmitted by a signal to indicate that half-luminance-sample or other MVD precision (whole or four-luminance samples) is used for a normal AMVP CU. In the case of half-luminance-samples, a 6-tap interpolation filter is used for half-luminance sample positions instead of the default 8-tap interpolation filter. Otherwise, a third flag is transmitted by a signal to indicate whether whole-luminance-sample MVD precision or four-luminance-sample MVD precision is used for a normal AMVP CU. In the case of an affine AMVP CU, a second flag is used to indicate whether whole-luminance-sample MVD precision or 1 / 16 luminance-sample MVD precision is used. To ensure that the reconstructed MV has the expected precision (quarter-luma-sample, half-luma-sample, whole-luma-sample, or quad-luma-sample), the motion vector predictor for the CU is rounded to the same precision as the MVD before the MVD is added. The motion vector predictor is rounded towards zero (i.e., a negative motion vector predictor is rounded towards positive infinity, and a positive motion vector predictor is rounded towards negative infinity). The encoder uses RD check to determine the motion vector resolution for the current CU. In order to avoid always performing four CU-level RD checks for each MVD resolution, in VTM14, only the RD check of the MVD precision other than a quarter-brightness-sample is conditionally called. For the normal AVMP mode, the RD cost of the quarter-brightness-sample MVD precision and the whole-brightness-sample MV precision is calculated first. Then, the RD cost of the whole-brightness-sample MVD precision is compared with the RD cost of the quarter-brightness-sample MVD precision to determine whether it is necessary to further check the RD cost of the four-brightness-sample MVD precision. When the RD cost for the quarter-brightness-sample MVD precision is much less than the whole-brightness-sample MVD precision, the RD check of the four-brightness-sample MVD precision is skipped. Then, if the RD cost of the whole-brightness-sample MVD precision is significantly greater than the best RD cost of the previously tested MVD precision, the check of the half-brightness-sample MVD precision is skipped. For the affine AMVP mode, if the affine inter mode is not selected after checking the rate-distortion cost of the affine merge / skip mode, merge / skip mode, quarter-luma-sample MVD accuracy normal AMVP mode, and quarter-luma-sample MVD accuracy affine AMVP mode, the 1 / 16 luma-sample MV accuracy and 1-pixel MV accuracy affine inter modes are not checked. In addition, the affine parameters obtained in the quarter-luma-sample MV accuracy affine inter mode are used as the starting search points for the 1 / 16 luma-sample and quarter-luma-sample MV accuracy affine inter modes. 2.1.8 Bidirectional Prediction with CU-Level Weights (BCW) In HEVC, a bidirectional prediction signal is generated by averaging two prediction signals obtained from two different reference pictures and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging to allow weighted averaging of the two prediction signals. P bi-pred =((8-w)*P0+w*P1+4)>>3 (3-24) Five weights, w∈{-2, 3, 4, 5, 10}, are allowed in weighted average bidirectional prediction. For each bidirectional prediction CU, the weight w is determined in one of two ways: 1) For non-merged CUs, the weight index is signaled after the motion vector difference; 2) For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applied to CUs with 256 or more luma samples (i.e., the CU width multiplied by the CU height is greater than or equal to 256). For low-latency pictures, all 5 weights will be used. For non-low-latency pictures, only 3 weights (w∈{3, 4, 5}) are used. -At the encoder, a fast search algorithm is applied to find the weight index without significantly increasing the encoder complexity. These algorithms are summarized below. Reference can be made to the VTM software and document JVET-L0646 for further details. When combined with AMVR, the unequal weights of 1-pixel and 4-pixel motion vector accuracy are only conditionally checked if the current picture is a low-latency picture. - When combined with affine, affine ME will be performed for unequal weights if and only if the affine mode is selected as the current best mode. - Unequal weights are only conditionally checked when the two reference pictures in bidirectional prediction are the same. - Unequal weights are not searched when certain conditions are met, which depend on the POC distance between the current picture and its reference pictures, the codec QP and the temporal level. The BCW weight index is encoded using one context encoded binary followed by a bypass encoded binary. The first context encoded binary indicates whether equal weights are used; and if unequal weights are used, an additional binary is signaled using the bypass codec to indicate which unequal weights are used. Weighted prediction (WP) is a codec tool supported by the H.264 / AVC and HEVC standards for efficient encoding and decoding of video content in fading conditions. Support for WP has also been added to the VVC standard. WP allows weighting parameters (weights and offsets) to be transmitted by signal for each reference picture list L0 and each reference picture in list L1. Then, during motion compensation, the weights and offsets of the corresponding reference pictures are applied. WP and BCW are designed for different types of video content. In order to avoid interaction between WP and BCW (which will complicate the VVC decoder design), if the CU uses WP, the BCW weight index is not transmitted by signal, and w is inferred to be 4 (ie, equal weights are applied). For the merged CU, the weight index is inferred from the neighboring blocks based on the merge candidate index. This can be applied to the normal merge mode and the inherited affine merge mode. For the constructed affine merge mode, affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index of the CU using the constructed affine merge mode is simply set equal to the BCW index of the first control point MV. In VVC, CIIP and BCW cannot be applied jointly to a CU. When a CU is encoded and decoded using CIIP mode, the BCW index of the current CU is set to 2, for example, equal weight. 2.1.9 Bidirectional Optical Flow (BDOF) The Bidirectional Optical Flow (BDOF) tool is included in VVC. BDOF, formerly known as BIO, is included in JEM. Compared to the JEM version, BDOF in VVC is a simpler version that requires much less computation, especially in terms of the number of multiplications and the size of the multipliers. BDOF is used to refine the bidirectional prediction signal of a CU at the 4×4 sub-block level. BDOF is applied to a CU if all of the following conditions are met: - The CU is encoded using "true" bi-prediction mode, ie, one of the two reference pictures precedes the current picture in display order, and the other of the two reference pictures follows the current picture in display order. - The distances from the two reference pictures to the current picture (ie, the POC difference) are the same. - Both reference images are short-term reference images. -The CU is not encoded or decoded using affine mode or SbTMVP merge mode. -CU has more than 64 luma samples. - CU height and CU width are both greater than or equal to 8 luma samples. -BCW weight index indicates equal weight. - The current CU does not have WP enabled. -CIIP mode is not used for the current CU. BDOF is applied only to the luma component. As its name suggests, BDOF mode is based on the concept of optical flow, which assumes that the motion of objects is smooth. For each 4×4 sub-block, motion refinement (v x , v y ) is calculated by minimizing the difference between L0 prediction samples and L1 prediction samples. Then motion refinement is used to adjust the bi-prediction sample values ​​in the 4x4 sub-block. The following steps are applied in the BDOF process. First, by directly calculating the difference between two adjacent samples, the horizontal and vertical gradients of the two prediction signals, and k=0,1,is calculated, that is, Among them I (k) (i, j) is the sample value at the prediction signal coordinate (i, j) in list k, k=0,1, and shift1 is calculated based on the luma bit depth bitDepth as shift1=max(6, bitDepth-6). Then, the autocorrelations and cross-correlations of the gradients S1, S2, S3, S5, and S6 are calculated as follows: in, where Ω is a 6×6 window around a 4×4 sub-block, and n a and n b The values ​​are set to min(1, bitDepth-11) and min(4, bitDepth-8) respectively. Then the cross-correlation and autocorrelation terms are used to refine the motion (v x , v y ) is exported using: in th′ BIO =2 max(5,BD-7) . is the round-down (floor) function, and Based on the motion refinement and gradients, the following adjustments are computed for each sample in the 4×4 sub-block: Finally, the BDOF samples of the CU are calculated by adjusting the bidirectional prediction samples as shown below: pred BDOF (x, y) = (I (0) (x,y)+I (1) (x, y) + b(x, y) + o offset )>>shift (3-30) These values ​​are chosen so that the multipliers in the BDOF process do not exceed 15 bits and the maximum bit width of the intermediate parameters in the BDOF process remains within 32 bits. In order to derive the gradient value, some prediction samples I in the list k (k=0, 1) outside the current CU boundary are (k) (i, j) needs to be generated. Fig.18 As shown, BDOF in VVC uses an extended row / column around the boundary of the CU. In order to control the computational complexity of generating prediction samples outside the boundary, the prediction samples in the extended area (white positions) are generated by directly taking the reference samples at nearby integer positions (using the floor() operation on the coordinates) without interpolation, and a conventional 8-head motion compensated interpolation filter is used to generate prediction samples within the CU (gray positions). These extended sample values ​​are only used for gradient calculations. For the rest of the steps in the BDOF process, if any sample values ​​and gradient values ​​outside the CU boundary are needed, these sample values ​​and gradient values ​​are filled (i.e. repeated) from their nearest neighbors. When the width and / or height of a CU is greater than 16 luma samples, it will be divided into sub-blocks with width and / or height equal to 16 luma samples, and the sub-block boundaries are regarded as CU boundaries in the BDOF process. The maximum unit size of the BDOF process is limited to 16x16. For each sub-block, the BDOF process can be skipped. When the SAD between the initial L0 prediction sample and the L1 prediction sample is less than a threshold, the BDOF process is not applied to the sub-block. The threshold is set equal to (8*W*(H>>1), where W represents the sub-block width and H represents the sub-block height. In order to avoid the additional complexity of SAD calculation, the SAD between the initial L0 prediction sample and the L1 prediction sample calculated in the DVMR process is reused here. If BCW is enabled for the current block, that is, the BCW weight index indicates unequal weights, bidirectional optical flow is disabled. Similarly, if WP is enabled for the current block, that is, luma_weight_lx_flag of any of the two reference pictures is 1, BDOF is also disabled; BDOF is also disabled when the CU is encoded and decoded using symmetric MVD mode or CIIP mode. 2.1.10 Decoder-side Motion Vector Refinement (DMVR) To improve the accuracy of the merge mode MV, decoder-side motion vector refinement based on bilateral matching (BM) is applied in VVC. In the bidirectional prediction operation, the refined MV is searched around the initial MV in the reference picture list L0 and the reference picture list L1. The BM method calculates the distortion between two candidate blocks in the reference picture list L0 and the list L1. Fig.19 As shown, based on each MV candidate around the initial MV, the SAD between the red blocks is calculated. The MV candidate with the lowest SAD becomes the refined MV and is used to generate the bidirectional prediction signal. In VVC, the application of DMVR is limited and only applies to CUs with the following modes and capabilities: - CU level merge mode with bi-predictive MV. -Relative to the current picture, one reference picture is in the past and the other reference picture is in the future. - The distances from the two reference pictures to the current picture (ie, the POC difference) are the same. - Both reference images are short-term reference images. -CU has more than 64 luma samples. - CU height and CU width are both greater than or equal to 8 luma samples. -BCW weight index indicates equal weight. -Currently the block is not WP enabled. -CIIP mode is not used for the current block. The refined MV derived by the DMVR process is used to generate inter-frame prediction samples and is also used for temporal motion vector prediction of future picture encoding and decoding. The original MV is used in the deblocking process and is also used for spatial motion vector prediction of future CU encoding and decoding. Additional features of DMVR are mentioned in the following sub-clauses. 2.1.10.1 Search Scheme In DVMR, the search point is around the initial MV, and the MV offset obeys the MV difference mirror rule. In other words, any point examined by DMVR represented by the candidate MV pair (MV0, MV1) follows the following two equations: MV0′=MV0+MV_offset (3-31) MV1′=MV1-MV-offset (3-32) Wherein, MV_offset represents the refinement offset between the initial MV and the refinement MV in one of the reference pictures. The refinement search range is two integer luma samples starting from the initial MV. The search includes an integer sample offset search phase and a fractional sample refinement phase. The integer sample offset search uses a 25-point full search. First, the SAD of the initial MV pair is calculated. If the SAD of the initial MV pair is less than the threshold, the integer sampling stage of DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search stage. In order to reduce the impact of DMVR refinement uncertainty, it is proposed to support the original MV in the DMVR process. The SAD between the reference blocks referenced by the initial MV candidates is reduced by 1 / 4 of the SAD value. The integer sample search is followed by fractional sample refinement. To save computational complexity, fractional sample refinement is derived using the parametric error surface equations instead of doing an additional search using SAD comparisons. Fractional sample refinement is conditionally called based on the output of the integer sample search stage. When the integer sample search stage ends with a center with minimum SAD in either the first or second iteration search, fractional sample refinement is further applied. In the sub-pixel offset estimation based on the parameter error surface, the cost of the center position and the costs of the four neighboring positions from the center are used to fit a two-dimensional parabolic error surface equation of the following form E(x, y) = A(x min ) 2 +B(yy min ) 2 +C (3-33) Where (x min ,y min) corresponds to the fractional position with the minimum cost, and C corresponds to the minimum cost value. By solving the above equation using the cost values ​​of the five search points, (x min ,y min ) is calculated as: x min =(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0))) (3-34) y min =(E(0,-1)-E(0,1)) / (2((E(0,-1)+E(0,1)-2E(0,0))) (3-35). x min and min The value of is automatically clamped between -8 and 8, since all cost values ​​are positive, with a minimum value of E(0,0). This corresponds to a half-pixel offset with 1 / 16-pixel MV precision in VVC. The calculated fraction (x min ,y min ) is added to the integer distance refinement MV to obtain the sub-pixel accurate refinement δMV. 2.1.10.2 Bilinear interpolation and sample padding In VVC, the resolution of MV is 1 / 16 luma sample. Samples at fractional positions are interpolated using an 8-tap interpolation filter. In DMVR, the search points are around the initial fractional pixel MV with integer sample offsets, so the samples at these fractional positions need to be interpolated for the DMVR search process. In order to reduce the computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in DMVR. Another important effect is that by using a bilinear filter, DVMR does not access more reference samples than the normal motion compensation process within the 2-sample search range. After the refined MV is obtained through the DMVR search process, an ordinary 8-tap interpolation filter is applied to generate the final prediction. In order not to access more reference samples for the normal MC process, samples will be filled from those available samples, which are not needed for the interpolation process based on the original MV, but are needed for the interpolation process based on the refined MV. 2.1.10.3 Maximum DMVR Processing Unit When the width and / or height of a CU is greater than 16 luma samples, it will be further split into sub-blocks with width and / or height equal to 16 luma samples. The maximum unit size for the DMVR search process is limited to 16x16. 2.1.11 Geometric Partitioning Mode (GPM) In VVC, geometric partitioning mode is supported for inter prediction. The geometric partitioning mode is signaled using a CU level flag as a merge mode. Other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. For each possible CU size w×h=2 m ×2 n , where m, n∈{3…6} Excluding 8x64 and 64x8, the geometric partitioning mode supports a total of 64 partitions. When this mode is used, the CU is split into two parts by a geometrically positioned straight line ( Fig.19 ). The position of the partition line is mathematically derived from the angle and offset parameters of the specific partition. Each part of the geometric partition in the CU uses its own motion for inter-frame prediction; each partition only allows unidirectional prediction, that is, each part has a motion vector and a reference index. Unidirectional prediction motion constraints are applied to ensure that, as with traditional bidirectional prediction, only two motion compensated predictions are required for each CU. The unidirectional prediction motion for each partition is derived using the process described in 3.4.11.1. If a geometric partitioning mode is used for the current CU, a geometric partitioning index indicating the partitioning mode (angle and offset) of the geometric partitioning and two merge indices (one for each partition) are further transmitted through a signal. The number of maximum GPM candidate sizes is explicitly transmitted through a signal in the SPS, and the syntax binarization for the GPM merge indices is specified. After predicting each part of the geometric partitioning, the sample values ​​along the edges of the geometric partitioning are adjusted using a hybrid process with adaptive weights as in 3.4.11.2. This is the prediction signal for the entire CU, and the transformation and quantization processes will be applied to the entire CU as in other prediction modes. Finally, the motion field of the CU predicted using the geometric partitioning mode is stored as in 3.4.11.3. 2.1.11.1 One-way prediction candidate list construction The unidirectional prediction candidate list is derived directly from the merge candidate list constructed according to the extended merge prediction process in 3.4.1. Denote n as the index of the unidirectional prediction motion in the geometric unidirectional prediction candidate list. The LX motion vector of the nth extended merge candidate is used as the nth unidirectional prediction motion vector of the geometric partitioning mode, with X equal to the parity of n. These motion vectors are Fig.21 If the corresponding LX motion vector of the n-th extended merge candidate does not exist, the L(1-X) motion vector of the same candidate is used instead as the unidirectional prediction motion vector of the geometric partitioning mode. 2.1.11.2 Blending Along Geometric Partition Edges After predicting each part of the geometric partition using its own motion, blending is applied to the two prediction signals to derive samples around the geometric partition edges. The blending weights for each position of the CU are derived based on the distance between the individual position and the partition edge. The distance from position (x, y) to the partition edge is derived as: ρ x,j ={i%16=8 or (i%16≠0 and h≥w)0 otherwise ±(j×w)>>2 (3-38) ρ y，j ={i%16=8 or (i%16≠0 and h≥w)±(j×h)>>2 otherwise 0 (3-39) where i, j are the angle and offset indices of the geometric partition, which depend on the geometric partition index transmitted by the signal. x，j and ρ y，j The sign of depends on the angle index i. The weight of each part of the geometric partition is derived as follows: wIdxL(x,y)=partIdx? 32+d(x,y):32-d(x,y) (3-40) w1(x,y)=1-w0(x,y) (3-42) partIdx depends on the angle index i. An example of weight w0 is Fig. 22 is shown in the figure. 2.1.11.3 Motion Field Storage for Geometry Partitioning Mode Mv1 from the first part of the geometric partitioning, Mv2 from the second part of the geometric partitioning, and the combination Mv of Mv1 and Mv2 are stored in the motion domain of the CU encoded in the geometric partitioning mode. The type of motion vector stored for each individual position in the motion field is determined as: sType=abs(motionIdx)<32?2:(motionIdx≤0?(1-partIdx):partIdx) (3-43) where motionIdx is equal to d(4x+2,4y+2), which is recalculated according to equation (3-36). partIdx depends on the angle index i. If sType is equal to 0 or 1, then Mv0 or Mv1 is stored in the corresponding motion field, otherwise, if sTType is equal to 2, the combined Mv from Mv0 and Mv2 is stored. The combined Mv is generated using the following process: 1) If Mv1 and Mv2 are from different reference picture lists (one from L0 and the other from L1), then Mv1 and Mv2 are simply combined to form a bidirectional prediction motion vector. 2) Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored. 2.1.12 Combined Inter and Intra Prediction (CIIP) In VVC, when a CU is encoded and decoded in merge mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and if both the CU width and the CU height are less than 128 luma samples, an additional flag is signaled to indicate whether the combined inter / intra prediction (CIIP) mode is applied to the current CU. As its name indicates, CIIP prediction combines the inter prediction signal with the intra prediction signal. The inter prediction signal P in CIIP mode inter is derived using the same inter-prediction process as applied to the normal merge mode; and the intra-prediction signal P intra It is derived after the conventional intra prediction process using planar mode. Then, the intra prediction signal and the inter prediction signal are combined using weighted averaging, where the weight values ​​are calculated depending on the coding mode of the top and left neighboring blocks (e.g. Fig.23 ) as follows: - If top neighboring is available and is intra-coded, set isIntraTop to 1, otherwise set isIntraTop to 0; - If the left neighbor is available and is intra-coded, set isIntraLeft to 1, otherwise set isIntralLeft to 0; - If (isIntraLeft + isIntraTop) is equal to 2, then wt is set to 3; - Otherwise, if (isIntraLeft+isIntraTop) is equal to 1, wt is set to 2; - Otherwise, set wt to 1. The CIIP forecast is constructed as follows: P CIIP =((4-wt)*P inter +wt*P intra +2)>>2 (3-43) 2.1.13 Reference Picture Resampling (RPR) In HEVC, the spatial resolution of a picture cannot be changed unless a new sequence with a new SPS is started, within an IRAP picture. VVC enables the resolution of pictures within a sequence to be changed at a position without encoding or decoding IRAP pictures that are always intra-coded. This feature is sometimes called reference picture resampling (RPR) because it requires resampling the reference pictures used for inter prediction when the reference pictures have a different resolution than the current picture being decoded. To avoid additional processing steps, the RPR process in VVC is designed to be embedded in the motion compensation process and performed at the block level. In the motion compensation stage, scaling is used together with motion information to locate reference samples in the reference picture for the difference process. In VVC, the scaling ratio is limited to be greater than or equal to 1 / 2 (2x downsampling from the reference picture to the current picture) and less than or equal to 8 (8x upsampling). Three sets of resampling filters with different frequency cutoffs are specified to handle various scaling ratios between the reference picture and the current picture. The three sets of resampling filters are applied to scaling ratios from 1 / 2 to 1 / 1.75, from 1 / 1.75 to 1 / 1.25, and from 1 / 1.25 to 8, respectively. Each set of resampling filters has 16 phases for luminance and 32 phases for chrominance, which is the same as the case of motion compensated interpolation filters. It is worth noting that in the case of scaling ratios ranging from 1 / 1.25 to 8, the filter set of normal MC interpolation is used. In fact, the conventional MC difference process is a special case of a resampling process with scaling ratios ranging from 1 / 1.25 to 8. In addition to conventional translation block motion, the affine mode has three sets of 6-tap interpolation filters for the luminance component to cover different scaling ratios in the RPR. Exports the horizontal and vertical scale ratios based on the image width and height and the left, right, top, and bottom scale offsets specified for the reference and current images. To support this feature, the picture resolution and the corresponding consistency window are signaled in the PPS instead of in the SPS, where the maximum picture resolution is signaled. 2.1.14 Miscellaneous inter-frame prediction aspects 2.2 To reduce memory bandwidth, 4x4 size CUs that are inter-coded are not allowed in VVC. For 4x8 or 8x4 CUs that are inter-coded, only unidirectional mode is allowed. When the motion information from merge mode is bidirectional, it is converted to unidirectional by keeping only list 0 motion information. 3. Inter-frame prediction tools under study in ECM 3.1 Local Illumination Compensation (LIC) LIC is an inter-prediction technique to model local illumination variations between the current block and its prediction block as a function of the current block template and the reference block template. The parameters of the function can be represented by a scale α and an offset β, which form a linear equation, i.e., α*p[x]+β to compensate for illumination variations, where p[x] is the reference sample pointed to by MV at position x on the reference picture. Since α and β can be derived based on the current block template and the reference block template, no signaling overhead is required for them, except that the AMVP mode signals a LIC flag to indicate the use of LIC. Local illumination compensation is used for uni-directionally predicted inter CUs with the following modifications. Neighboring samples within the frame can be used to derive LIC parameters; Disable LIC for blocks with less than 32 luma samples; For both non-sub-block and affine modes, LIC parameter derivation is performed based on the template block samples corresponding to the current CU instead of the partial template block samples corresponding to the first top-left 16x16 unit; Generate samples of the reference block template by using MC with the block MV without rounding it to integer pixel precision. 3.2 Non-adjacent spatial candidates Non-adjacent spatial merge candidates are inserted after TMVP in the regular merge candidate list. The pattern of spatial merge candidates is as follows: Fig.24 The distance between non-adjacent spatial candidates and the current codec block is based on the width and height of the current codec block. The line buffer limit is not applied. 3.3 Template Matching (TM) Template Matching (TM) is a decoder-side MV derivation method used to refine the motion information of the current CU by finding the closest match between a template in the current picture (i.e., the top and / or left neighboring blocks of the current CU) and a block in a reference picture (i.e., the same size as the template). Fig.25 As shown, in the [-8, +8] pixel search range, a better MV is searched around the initial motion of the current CU. The template matching method is used with the following modifications: the search step size is determined based on the AMVR mode, and the TM can cascade the bilateral matching process in the merge mode. In AMVP mode, MVP candidates are determined based on template matching error to pick the one that achieves the minimum difference between the current block template and the reference block template, and then TM performs MV refinement only on that specific MVP candidate. TM refines that MVP candidate by using an iterative diamond search, starting with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode) within a search range of [-8, +8] pixels. The AMVP candidate can be further refined by using a cross search with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode), followed by half-pixels and quarter-pixels, depending on the AMVR mode as specified in Table 1. This search process ensures that the MVP candidate still maintains the same MV accuracy as indicated by the AMVR mode after the TM process. Table 1 - Search patterns of AMVR and merge patterns with AMVR In merge mode, a similar search method is applied to the merge candidates indicated by the merge index. As shown in Table 1, TM can be performed all the way down to 1 / 8 pixel MVD accuracy, or skip accuracy beyond half-pixel MVD accuracy, depending on whether an alternative interpolation filter is used based on the merged motion information (used when AMVR is half-pixel mode). In addition, when TM mode is enabled, template matching can work as an independent process between block-based and sub-block-based bilateral matching (BM) methods or as an additional MV refinement process, depending on whether BM can be enabled based on its enabling condition check. 3.4 Multi-pass decoder-side motion vector refinement Multiple passes of decoder side motion vector refinement are applied. In the first pass, bilateral matching (BM) is applied to one codec block. In the second pass, BM is applied to each 16x16 sub-block within the codec block. In the third pass, the MV in each 8x8 sub-block is refined by applying bidirectional optical flow (BDOF). The refined MVs are stored for both spatial and temporal motion vector prediction. 3.4.1 First pass - Block-based bilateral matching MV refinement In the first pass, refined MVs are derived by applying BM to the codec blocks. Similar to decoder-side motion vector refinement (DMVR) in bidirectional prediction, refined MVs are searched around two initial MVs (MV0 and MV1) in reference picture lists L0 and L1. Refined MVs (MV0_pass1 and MV1_pass1) are derived around the initiating MVs based on the minimum bilateral matching cost between the two reference blocks in L0 and L1. BM performs a local search to derive integer sample precision intDeltaMV. The local search applies a 3×3 square search pattern, looping in the search range [-sHor, sHor] in the horizontal direction and the search range [-sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block scale and the maximum values ​​of sHor and sVer are 8. The bilateral matching cost is calculated as: bilCost = mvDistanceCost + sadCost. When the block size cbW*cbH is greater than 64, the MRSAD cost function is applied to remove the DC effect of the distortion between the reference blocks. When the bilCost of the center point of the 3×3 search pattern has the minimum cost, the intDeltaMV local search terminates. Otherwise, the current minimum cost search point becomes the new center point of the 3×3 search pattern and continues to search for the minimum cost until it reaches the end of the search range. The existing fractional sample refinement is further applied to derive the final deltaMV. Then, the refined MV after the first pass is exported as: MV0_pass1 = MV0 + deltaMV, MV1_pass1=MV1-deltaMV. 3.4.2 Second pass - sub-block based bilateral matching MV refinement In the second pass, refined MVs are derived by applying BM to 16×16 grid sub-blocks. For each sub-block, the refined MVs are searched around the two MVs (MV0_pass1 and MV1_pass1) obtained in the first pass in the reference picture lists L0 and L1. The refined MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the two reference sub-blocks in L0 and L1. For each subblock, BM performs a full search to derive integer sample precision intDeltaMV. The full search has a search range of [-sHor, sHor] in the horizontal direction and a search range of [-sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block scale and the maximum values ​​of sHor and sVert are 8. The bilateral matching cost is calculated by applying the cost factor to the SATD cost between two reference sub-blocks, such as: bilCost = satdCost * costFactor. The search area (2*sHor+1)*(2*sVer+1) is divided into 5 diamond search areas, such as Fig.26As shown. Each search area is assigned a costFactor determined by the distance (intDeltaMV) between each search point and the starting MV, and each diamond area is processed in order starting from the center of the search area. In each area, the search points are processed in raster scan order, starting from the upper left corner of the area and going to the lower right corner. When the minimum bilCost in the current search area is less than or equal to the threshold of sbW*sbH, the integer pixel (int-pel) full search is terminated, otherwise, the int-pel full search continues to the next search area until all search points are checked. The existing VVC DMVR fractional sample refinement is further applied to derive the final deltaMV (sbIdx2). Then, the refined MV of the second pass is derived as: ·MV0_pass2(sbIdx2)=MV0_pass1+deltaMV(sbIdx2); ·MV1_pass2(sbIdx2)=MV1_pass1-deltaMV(sbIdx2). 3.4.3 The third pass - sub-block-based bidirectional optical flow MV refinement In the third pass, refined MVs are derived by applying BDOF to 8×8 grid sub-blocks. For each 8×8 sub-block, BDOF refinement is applied to derive scaled Vx and Vy without clipping from the refined MV of the parent sub-block of the second pass. The derived bioMv(Vx, Vy) is rounded to 1 / 16 sample precision and clipped between -32 and 32. The refined MVs of the third pass (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) are derived as: ·MV0_pass3(sbIdx3)=MV0_pass2(sbIdx2)+bioMv, ·MV1_pass3(sbIdx3)=MV0_pass2(sbIdx2)-BioMv. 3.5OBMC When OBMC is applied, the top and left boundary pixels of the CU are refined using the motion information of neighboring blocks with weighted prediction. The conditions under which OBMC should not be applied are as follows: When OBMC is disabled at SPS level; When the current block has intra mode or IBC mode; When the current block applies LIC; When the current luminance block area is less than or equal to 32 Sub-block boundary OBMC is performed by applying the same blending to the top, left, bottom and right sub-block boundary pixels using the motion information of the neighboring sub-blocks. It is enabled for sub-block based codec tools: Affine AMVP mode; Affine merge mode and sub-block based temporal motion vector prediction (SbTMVP); Sub-block based bilateral matching. 3.6 Sample-based BDOF In sample-based BDOF, instead of deriving motion refinement (Vx, Vy) on a block basis, it is performed per sample. The codec block is divided into 8×8 sub-blocks. For each sub-block, whether to apply BDOF is determined by checking the SAD between two reference sub-blocks against a threshold. If it is decided to apply BDOF to a sub-block, for each sample in the sub-block, Vx and Vy are derived using a sliding 5×5 window and applying the existing BDOF process for each sliding window. The derived motion refinement (Vx, Vy) is applied to adjust the bidirectionally predicted sample value of the window center sample. 3.7 Interpolation The 8-tap interpolation filter used in VVC is replaced by a 12-tap filter. The interpolation filter is derived from a sinc function, whose frequency response is truncated at the Nyquist frequency and clipped by a cosine window function. Table 2 gives the filter coefficients for all 16 phases. Fig. 27 The frequency response of the interpolation filter is compared with the VVC interpolation filter, all at half-pixel phase. Table 2. Filter coefficients for 12-tap interpolation filter 3.8 Multiple Hypothesis Prediction (MHP) In the multi-hypothesis inter prediction mode, one or more additional motion compensated prediction signals are signaled in addition to the conventional bidirectional prediction signal. The resulting overall prediction signal is obtained by sample-by-sample weighted superposition. bi and the first additional inter-frame prediction signal / hypothesis h3, the resulting prediction signal is as follows: p3=(1-α)pb i +αh3. The weighting factor α is specified by a new syntax element add_hyp_weight_idx according to the following mapping. add_hypweight_idx α 0 1 / 4 1 -1 / 8 Similarly, more than one additional prediction signal may be used. The resulting overall prediction signal is iteratively accumulated using each additional prediction signal. pn+1 =(1-α n+1 ) n +α n+1 h n+1 The resulting overall prediction signal is obtained as the final p n (i.e., the p with the largest index n n ). Within this EE, up to two additional prediction signals may be used (ie, n is limited to 2). The motion parameters for each additional prediction hypothesis can be signaled implicitly by specifying the reference index, motion vector predictor index and motion vector difference or by specifying a merge index. A separate multi-hypothesis merge flag distinguishes these two signaling modes. For inter-AMVP mode, MHP is only applied if unequal weights in BCW are selected in bi-prediction mode. A combination of MHP and BDOF is possible, however BDOF is only applied to the bidirectional prediction signal part of the prediction signal (ie the ordinary first two hypotheses). 3.9 Adaptive Reordering of Merged Candidates with Template Matching (ARMC-TM) The merge candidates are adaptively re-ranked using Template Matching (TM). The re-ranking method is applied to the regular merge mode, Template Matching (TM) merge mode, and Affine merge mode (excluding SbTMVP candidates). For TM merge mode, the merge candidates are re-ranked before the refinement process. After constructing the merge candidate list, the merge candidates are divided into several subgroups. For the normal merge mode and the TM merge mode, the subgroup size is set to 5. For the affine merge mode, the subgroup size is set to 3. The merge candidates in each subgroup are sorted in ascending order according to the cost value based on template matching. For simplicity, the merge candidates in the last but not the first subgroup are not reordered. The template matching cost of the merge candidate is measured by the sum of absolute differences (SAD) between the samples of the template of the current block and its corresponding reference samples. The template includes a set of reconstructed samples adjacent to the current block. The reference samples of the template are located by the motion information of the merge candidate. When the merge candidate uses bidirectional prediction, the reference sample of the merge candidate template is also composed of Fig.28 The bidirectional prediction shown in is generated. For a sub-block based merge candidate with a sub-block size equal to Wsub×Hsub, the above template includes several sub-templates with a size of Wsub×1, and the left template includes several sub-templates with a size of 1×Hsub. Fig.29 As shown, the motion information of the sub-blocks in the first row and the first column of the current block is used to derive the reference samples of each sub-template. 3.10 Geometric Partitioning Mode (GPM) with Merged Motion Vector Difference (MMVD) GPM in VVC is extended by applying motion vector refinement on top of the existing GPM unidirectional MV. A flag for the GPM CU is first signaled to specify whether this mode is used. If the mode is used, each geometric partition of the GPM CU can further decide whether to signal MVD. If MVD is signaled for a geometric partition, after selecting a GPM merge candidate, the motion of the partition is further refined by the signaled MVD information. All other procedures remain the same as GPM. MVD is signaled as a pair of distance and direction, similar to MMVD. GPM has 9 candidate distances (1 / 4-pixel, 1 / 2-pixel, 1-pixel, 2-pixel, 3-pixel, 4-pixel, 6-pixel, 8-pixel, 16-pixel) and 8 candidate directions (four horizontal / vertical directions and four diagonal directions) involved in MMVD (GPM-MMVD). In addition, when pic_fpel_mmvd_enabled_llag is equal to 1, MVD is shifted left by 2 in MMVD. 3.11 Geometric Partitioning Pattern (GPM) with Template Matching (TM) Template matching is applied to GPM. When GPM mode is enabled for a CU, a CU level flag is signaled to indicate whether TM is applied to both geometry partitions. TM is used to refine the motion information of each geometry partition. When TM is selected, a template is constructed using left, top, or left and top neighboring samples depending on the partition angle, as shown in Table 3. The motion is then refined by minimizing the difference between the current template and the template in the reference picture using the same search pattern with merge mode of the half-pixel interpolation filter disabled. Table 3 is for the templates of the 1st and 2nd geometric partitions, where A indicates using the upper samples, L indicates using the left samples, and L+A indicates using both the left samples and the upper samples. Divide Angle 0 2 3 4 5 8 11 12 13 14 First division A A A A L+A L+A L+A L+A A. A. Second division L+A L+A L+A L L L L L+A L+A L+A Divide Angle 16 18 19 20 21 24 27 28 29 30 First division A A A A L+A L+A L+A L+A A A Second division L+A L+A L+A L L L L L+A L+A L+A The GPM candidate list is constructed as follows: 1. The interleaved list 0 MV candidates and list 1 MV candidates are directly derived from the regular merge candidate list, where the list 0 MV candidates have higher priority than the list 1 MV candidates. A pruning method with adaptive threshold based on the current CU size is applied to remove redundant MV candidates. 2. The interleaved list 1 MV candidates and list 0 MV candidates are further derived directly from the regular merge candidate list, where the list 1 MV candidates have higher priority than the list 0 MV candidates. The same pruning method with adaptive threshold is also applied to remove redundant MV candidates. 3. Zero MV candidates are filled until the GPM candidate list is full. 4. GPM-MMVD and GPM-TM are specific to one GPM CU. This is done by first signaling the GPM-MMVD syntax. When both GPM-MMVD control flags are equal to false (i.e., GPM-MMVD is disabled for both GPM partitions), a GPM-TM flag is signaled to indicate whether template matching is applied to both GPM partitions. Otherwise (at least one GPM-MMVD flag is equal to true), the value of the GPM-TM flag is inferred to be false. 4. Question Existing designs for illumination compensation have template-based model parameter derivation. However, the derived parameters may not be optimal for the current block. This motivates an approach to refine the model parameters for illumination compensation. 5. Detailed solution In order to solve the above problems and some unmentioned problems, the methods outlined below are disclosed. The solutions should be considered as examples to explain the general concept and should not be interpreted in a narrow manner. In addition, these solutions can be applied individually or in combination in any way. Although a solution for template-based derivation of LIC parameters is disclosed, the disclosed method is applicable to other design methods of LIC, such as reconstructing a portion of a block to determine LIC parameters. The disclosed method is applicable to other scenarios using illumination compensation, such as inter-view prediction in multi-view coding. The term "block" may refer to a codec block (CB), Cu, PU, ​​PB. 1. Regarding a method for refining model parameters for illumination compensation, the following method is disclosed. 1) In one example, for a block encoded with LIC, a value Δα and / or a value Δβ may be applied, so that the updated LIC model parameters may be α+Δα and / or β+Δβ. a. In one example, additional operations such as shifting or clipping may be applied to α+Δα and / or β+Δβ to obtain the final α and / or β in the LIC. b. Blocks can have specific modes, such as AMVP or Affine AMVP or Merging or Sub-block Merging. c. In one example, the selection of the refinement parameter value depends on factors such as: i. For example, the values ​​of Δα and Δβ and the total number of these values ​​depend on various factors, such as block size, QP value, etc. ii. For example, different values ​​are used at the sequence / picture / strip level. iii. For example, dynamically updating values ​​based on previously observed statistics. d. In one example, refinement parameters are used under certain conditions like certain sequence or frame or block size and QP value. e. In one example, the above method is applied to all color planes or a subset of them. i. In one example, the refinement parameters for each plane are different. ii. In one example, all or a subset of color planes share the same refinement parameters. f. Δα and / or Δβ may be signaled at picture / slice / tile / CTU / CU / PU / TU level. i. In one example, Δα and / or Δβ may be selected from the candidate set and an index may be signaled to indicate which one was selected. ii. In one example, Δα and / or Δβ may be in the form of K*step, where step is a predefined number, such as 1 / 64 or 1 / 32, and K may be signaled to indicate Δα and / or Δβ. iii. The candidate set may depend on codec information, such as slice / picture type, temporal layer, QP, block size, codec mode, color component, etc. iv. Δα and / or Δβ can be encoded and decoded in a predictive manner. a) Δα and / or Δβ can be predicted from Δα and / or Δβ of neighboring blocks. v. In one example, the syntax elements used to indicate Δα and / or Δβ may be binarized as fixed length codes, (truncated) unary codes, exponential Golomb codes, etc. vi. In one example, the syntax element indicating Δα and / or Δβ may be encoded using at least one coding context in arithmetic coding. a) Alternatively, the syntax elements indicating Δα and / or Δβ may be coded using bypass coding. vii. In one example, the sign and absolute value of Δα and / or Δβ may be encoded and decoded separately. viii. In one example, the candidate set of Δα and / or Δβ may be signaled, for example, in a slice header / picture header / PPS / SPS / APS / etc. 2) In one example, for a block encoded with an MV candidate list having at least one candidate (denoted as X candidate) with the LIC flag enabled, at least one additional candidate may be inserted into the candidate list. a. In one example, a block may be encoded and decoded using a specific mode such as merging, MMVD, CIIP, affine merging, sub-block merging, template matching merging, etc. b. The MV candidate list can be a merge list, an affine merge list, a template matching merge list, etc. c. In one example, the new merge candidate may inherit all the information of the X candidate and additionally apply refinement parameters Δα and / or Δβ. i. For example, the selection and use of Δα and / or Δβ follows similar basic principles as disclosed in the previous item 1. ii. For example, only a subset of candidates are considered for constructing new merge candidates with refined parameters. iii. For example, the selection of Δα and / or Δβ is further conditioned on the type of merging mode, such as affine mmvd, regular merging, etc. d. Additional candidates may be added to the list after (only) the X candidates. e. Multiple additional candidates may be added for X candidates with different Δα and / or Δβ. f. If there is more than one candidate with the LIC flag in the list, multiple X candidates may be considered. 3) In one example, for merge mode, instead of adding new merge candidates, refinement parameters similar to AMVP mode are applied, and the value and usage may further depend on: i. For example, the type of merge mode; ii. For example, factors such as block size, QP, and other conditions as listed for AMVP mode. 4) In one example, a syntax element (eg, a flag) may be signaled in the bitstream to specify whether the disclosed parameters are ultimately selected for the current block codec. a. Furthermore, the syntax elements may depend on whether the block is AMVP coded or merge coded. b. Alternatively, furthermore, the syntax element may depend on whether the block is uni-directionally predicted or bi-directionally predicted. c. Alternatively, furthermore, the syntax elements may depend on the type of merge prediction used. d. Alternatively, furthermore, syntax elements are signaled only for a subset or all of the Y, U, V planes. e. Alternatively, furthermore, syntax elements may be signaled conditionally. f. Whether / how to signal syntax elements may depend on the dimensions of the current block. i. For example, the syntax element may be signaled only if the current block is larger than a predefined size. ii. For example, the syntax element may be signaled only if the sum of the width and height of the current block is greater than a predefined threshold. iii. Greater than can be replaced by “less than” or “not greater than” or “not less than”. g. Whether / how to signal syntax elements may depend on the QP selected to encode the current block. i. For example, the syntax element may be signaled only if the QP of the current block is greater than a predefined threshold. ii. Greater than can be replaced by “less than” or “not greater than” or “not less than”. h. In one example, syntax elements may be binarized into fixed length codes, (truncated) unary codes, exponential Golomb codes, etc. i. In one example, the syntax element may be encoded using at least one codec context in arithmetic coding. i. Alternatively, the syntax elements may be encoded and decoded using bypass codec. General 5) Whether and / or how to apply the above disclosed method can be applied at the sequence level / picture group level / picture level / Signaled at slice level / tile group level, e.g. in sequence header / picture header / SPS / VPS / DPS / DCI / PPS / PPS / Slice Header / Tile Group Header. 6) Whether and / or how to apply the methods disclosed above may be signaled at PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU row / slice / tile / sub-picture / other kind of region containing more than one sample or pixel. 7) Whether and / or how to apply the above disclosed method may depend on the coded information, such as block size, color format, single / dual tree partitioning, color component, slice / picture type.

[0089] More details related to parameter refinement for illumination compensation in video encoding and decoding of embodiments of the present disclosure will be described below. The embodiments of the present disclosure should be considered as examples to explain general concepts and should not be interpreted in a narrow manner. In addition, these embodiments can be applied individually or in combination in any way.

[0090] As used herein, the term "block" may refer to a color component, a sub-picture, a slice, a tile, a codec tree unit (CTU), a CTU row, a CTU group, a codec unit (CU), a prediction unit (PU), a transform unit (TU), a codec tree block (CTB), a codec block (CB), a prediction block (PB), a transform block (TB), a sub-block of a video block, a sub-region within a video block, a video processing unit including multiple samples / pixels, etc. A block may be rectangular or non-rectangular.

[0091] Fig.303000 is a flowchart of a method 3000 for video processing according to some embodiments of the present disclosure. The method 3000 may be implemented during a conversion between a current video block of a video and a bitstream of the video. Fig.30 As shown, method 3000 starts at 3002, where a set of adjustment parameter values ​​are obtained. The set of adjustment parameters are associated with a set of model parameter values ​​for a local illumination compensation (LIC) model for encoding and decoding a current video block. It should be understood that a set of adjustment parameters may include one or more adjustment parameters, and a set of model parameters may include one or more adjustment parameters.

[0092] As an example and not limitation, the values ​​of a set of model parameters can be determined based on the template of the current video block and the template of the reference video block of the current video block. It should be understood that the values ​​of a set of model parameters can also be obtained in any other suitable manner. The scope of the present disclosure is not limited in this regard.

[0093] In some embodiments, a set of model parameters may include at least one of the following: a scaling amount of the LIC model, or an offset of the LIC model. Accordingly, a set of adjustment parameters may include at least one of the following: a scaling amount adjustment parameter for updating a value of the scaling amount, or an offset adjustment parameter for updating a value of the offset. It should be understood that a set of model parameters may also include any other suitable parameters. The scope of the present disclosure is not limited in this respect.

[0094] In some embodiments, a set of adjustment parameter values ​​may be determined at the encoder and decoder, respectively. Alternatively, a set of adjustment parameter values ​​may be determined at the encoder and sent to the decoder with a signal. The decoder may obtain a set of adjustment parameter values ​​by parsing the bitstream.

[0095] At 3004, the values ​​of a set of model parameters are updated based on the values ​​of the set of adjustment parameters. As an example and not limitation, the value of the scaling amount can be updated based on the sum of the value of the scaling amount and the value of the scaling amount adjustment parameter. Additionally or alternatively, the value of the offset can be updated based on the sum of the value of the offset and the value of the offset adjustment parameter. It should be understood that the values ​​of a set of model parameters can also be updated in any other suitable manner. The scope of the present disclosure is not limited in this regard.

[0096] At 3006, a conversion is performed based on the updated values ​​of a set of model parameters. In some embodiments, the conversion can be performed with the aid of a LIC model having updated values ​​for the set of model parameters. Alternatively, a target value for the set of model parameters can be obtained by shifting or clipping at least one of the updated values. Furthermore, the conversion can be performed based on the target value.

[0097] In some embodiments, the conversion may include encoding the current video block into a bitstream. Alternatively or additionally, the conversion may include decoding the current video block from the bitstream. It should be understood that the above illustrations and / or examples are described for illustrative purposes only. The scope of the present disclosure is not limited in this regard.

[0098] In view of the above situation, one or more adjustment parameters are used to update the value of at least one model parameter of the LIC model. Compared with conventional solutions, the proposed method can refine the model parameters for illumination compensation to obtain the optimal model parameters for the current video. In this way, the encoding and decoding quality can be improved.

[0099] In some embodiments, the current video block may be encoded and decoded using an advanced motion vector prediction (AMVP) mode, an affine AMVP mode, a merge mode, a sub-block merge mode, or the like.

[0100] In some embodiments, the set of adjustment parameters may include a first adjustment parameter. At 3002, a value of the first adjustment parameter may be selected from a plurality of candidate values ​​for the first adjustment parameter based on a size of a current video block, a quantization parameter (QP) value used to encode the current video block, and / or the like. As an example and not limitation, the number of candidate values ​​in the plurality of candidate values ​​may depend on the size of the current video block, the QP value used to encode the current video block, and the like.

[0101] In some embodiments, the values ​​of a set of adjustment parameters may depend on the level of the current video block, that is, different values ​​for a set of adjustment parameters are adopted at the sequence level, the picture level or the slice level.

[0102] In some embodiments, the video may further include a first video block that was encoded using the LIC mode prior to the current video block. At 3002, values ​​of a set of adjustment parameters may be determined based on statistics of the first video block and values ​​of a set of adjustment parameters used to encode the first video block. In other words, the values ​​of the set of adjustment parameters are dynamically updated based on previously observed statistics.

[0103] In some embodiments, the video may further include a second video block different from the current video block. The information about whether to update the values ​​of a set of model parameters for encoding and decoding the second video block may depend on a sequence of the second video blocks, a frame of the second video block, a size of the second video block, a QP value for encoding and decoding the second video block, and / or the like.

[0104] In some embodiments, the video may be associated with a plurality of color planes. In addition, values ​​of a set of model parameters for encoding and decoding the video block associated with at least one color plane of the plurality of color planes may be updated. In some embodiments, the values ​​of the set of adjustment parameters may be different for each color plane of the plurality of color planes. In some alternative embodiments, the values ​​of the set of adjustment parameters may be the same for all color planes of the plurality of color planes. Alternatively, the values ​​of the set of adjustment parameters may be the same for a subset of the plurality of color planes.

[0105] In some embodiments, the set of adjustment parameters may include a second adjustment parameter. In addition, a value for the second adjustment parameter or an indication of a value for the second adjustment parameter may be indicated in the bitstream at a picture level, a slice level, a tile level, a codec tree unit (CTU) level, a codec unit (CU) level, a prediction unit (PU) level, or a transform unit (TU) level, etc.

[0106] In some embodiments, the value for the second adjustment parameter may be selected from a set of candidate values. The indication may include an index to the value for the second adjustment parameter. Alternatively, the value for the second adjustment parameter may be equal to the product of a predefined number and K. K may be a real number and may be included in the indication. As an example and not limitation, the predefined number may be 1 / 64 or 1 / 32.

[0107] In some embodiments, a set of candidate values ​​may depend on codec information of the current video block. As an example and not limitation, the codec information may include slice type, picture type, temporal layer, QP, block size, codec mode, or color component, etc.

[0108] In some embodiments, the value or indication for the second adjustment parameter may be encoded in a predictive manner.As an example and not limitation, the value for the second adjustment parameter may be determined based on the value for the second adjustment parameter of a neighboring video block used to encode the current video block.

[0109] In some embodiments, the syntax element indicating the value for the second adjustment parameter may be binarized into a fixed length code, a unary code, a truncated unary code, or an exponential Golomb code, etc. In some embodiments, the syntax element indicating the value for the second adjustment parameter may be coded using at least one codec context in arithmetic coding. Alternatively, the syntax element for the value of the second adjustment parameter may be coded using bypass coding.

[0110] In some embodiments, the sign and the absolute value of the second adjustment parameter may be encoded and decoded separately. In some embodiments, a set of candidate values ​​may be indicated in a slice header, a picture header, a picture parameter set (PPS), a sequence parameter set (SPS), or an adaptation parameter set (APS), etc.

[0111] In some embodiments, if the motion vector (MV) candidate list for the current video block includes at least one candidate coded and decoded using the LIC mode, at least one additional candidate may be added to the MV candidate list. For example, the at least one additional candidate may be generated based on the at least one candidate. It should be noted that if the at least one candidate is coded and decoded using the LIC mode, the LIC flag is enabled for the at least one candidate.

[0112] In some embodiments, the current video block may be encoded and decoded using a merge mode, a merge mode with motion vector difference (MMVD) mode, a combined inter-intra prediction (CIIP) mode, an affine merge mode, a sub-block merge mode, or a template matching merge mode, etc. In some embodiments, the MV candidate may be a merge list, an affine merge list, or a template matching merge list, etc.

[0113] In some embodiments, at least one additional candidate may include a first additional candidate corresponding to a first candidate in the at least one candidate. The first additional candidate may be determined based on information of the first candidate and a reference value of a set of model parameters. The reference value may be obtained by using an adjustment value of a set of model parameters for encoding and decoding the first candidate using a set of adjustment parameters. In other words, the additional candidate may inherit all information of the corresponding candidate and additionally apply the adjustment parameters to the model parameters.

[0114] In some embodiments, the values ​​of a set of tuning parameters used to determine additional candidates may be selected and used in a manner similar to the values ​​of a set of tuning parameters used to update the values ​​of a set of model parameters, as described above.

[0115] In some embodiments, the at least one candidate may include multiple candidates encoded and decoded using the LIC mode. In addition, at least one additional candidate may be generated based on a portion of the multiple candidates.

[0116] In some embodiments, the current video block may be encoded using a merge mode. In addition, values ​​of a set of adjustment parameters may be selected based on the type of merge mode.

[0117] In some embodiments, the first additional candidate may be added to the MV candidate list after the first candidate. In one example, the first additional candidate may follow the first candidate in the MV candidate list. Additionally, the first additional candidate may immediately follow the first candidate in the MV candidate list.

[0118] In some embodiments, the at least one additional candidate may include multiple additional candidates determined based on the first additional candidate having different values ​​of a set of adjustment parameters. In other words, more than one additional candidate may be generated based on one of the at least one candidate using different values ​​of a set of adjustment parameters.

[0119] In some embodiments, the at least one candidate may include multiple candidates encoded using the LIC mode.One or more additional candidates may be determined for each of the multiple candidates and added to the MV candidate list.

[0120] In some embodiments, if the motion candidate list for merge mode used to encode the current video block includes a second candidate encoded using LIC mode, a second additional candidate can be determined based on information of the second candidate and a reference value of a set of model parameters. The reference value can be obtained by adjusting the values ​​of a set of model parameters used to encode the second candidate using a set of adjustment parameters. The values ​​of the set of adjustment parameters can be indicated in the bitstream. For example, the values ​​of the set of adjustment parameters can be determined and used based on the type of merge mode, the size of the current video block, or QP and / or the like.

[0121] In some embodiments, information about at least one of the following may depend on the dimension of the current video block or the QP used to encode the current video block: whether to indicate a first syntax element in the bitstream, or how to indicate the first syntax element in the bitstream. The first syntax element may indicate first information about whether a value of a set of adjustment parameters is used to encode the current video block. As an example and not limitation, the first information may depend on whether the current video block is encoded and decoded using AMVP mode or merge mode. In another example, the first information may depend on whether the current video block is unidirectionally predicted or bidirectionally predicted. Alternatively, in the case where the current video block is encoded and decoded using merge prediction mode, the first information may depend on the type of merge prediction mode.

[0122] In some embodiments, the first syntax element may be indicated in the bitstream. In one example, the first syntax element may be indicated in the bitstream for a Y color plane. Additionally or alternatively, the first syntax element may be indicated in the bitstream for a U color plane. Additionally or alternatively, the first syntax element may be indicated in the bitstream for a V color plane.

[0123] In some embodiments, if the current video block is larger than a predefined size, the first syntax element may be indicated in the bitstream. Additionally or alternatively, if the sum of the width and height of the current video block is larger than a predefined threshold, the first syntax element may be indicated in the bitstream.

[0124] In some alternative embodiments, if the current video block is smaller than a predefined size, the first syntax element may be indicated in the bitstream. Additionally or alternatively, if the sum of the width and height of the current video block is smaller than a predefined threshold, the first syntax element may be indicated in the bitstream.

[0125] In some alternative embodiments, the first syntax element may be indicated in the bitstream, such as if the current video block is greater than or equal to a predefined size. Additionally or alternatively, the first syntax element may be indicated in the bitstream if the sum of the width and height of the current video block is greater than or equal to a predefined threshold.

[0126] In some alternative embodiments, the first syntax element may be indicated in the bitstream if the current video block is less than or equal to a predefined size. Additionally or alternatively, the first syntax element may be indicated in the bitstream if the sum of the width and height of the current video block is less than or equal to a predefined threshold.

[0127] In some embodiments, if the QP is greater than a predefined threshold, the first syntax element may be indicated in the bitstream. Alternatively, if the QP is less than a predefined threshold, the first syntax element may be indicated in the bitstream. In some alternative embodiments, if the QP is greater than or equal to the predefined threshold, the first syntax element may be indicated in the bitstream. In some further embodiments, if the QP is less than or equal to the predefined threshold, the first syntax element may be indicated in the bitstream.

[0128] In some embodiments, the first syntax element may be binarized as one of the following: a fixed length code, a unary code, a truncated unary code, or an exponential Golomb code, etc. In some embodiments, the first syntax element may be encoded using at least one codec context in arithmetic coding. Alternatively, the first syntax element may be encoded using bypass coding.

[0129] In some embodiments, whether and / or how to apply a method may be indicated at a sequence level, a picture group level, a picture level, a slice level, or a tile group level, etc. In some embodiments, whether and / or how to apply a method may be indicated in a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header, etc.

[0130] In some embodiments, whether and / or how to apply a method may be indicated at a region containing more than one sample or pixel. By way of example and not limitation, a region may include a prediction block (PB), a transform block (TB), a codec block (CB), a prediction unit (PU), a transform unit (TU), a codec unit (CU), a virtual pipeline data unit (VPDU), a codec tree unit (CTU), a CTU row, a slice, a tile, and / or a sub-picture.

[0131] In some embodiments, whether and / or how to apply the method may depend on the encoded information. As an example and not limitation, the encoded information may include block size, color format, single tree partitioning, dual tree partitioning, color component, slice type, or picture type and / or the like.

[0132] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a device for video processing. In the method, a set of adjustment parameter values ​​are obtained. The set of adjustment parameter values ​​are associated with a set of model parameter values ​​of an LIC model for a current video block of the encoded video. The set of model parameter values ​​are updated based on the set of adjustment parameter values. In addition, a bitstream is generated based on the updated values ​​of the set of model parameters.

[0133] According to yet another embodiment of the present disclosure, a method for storing a bitstream of a video is provided. In the method, a set of adjustment parameter values ​​are obtained. The set of adjustment parameter values ​​are associated with a set of model parameter values ​​of an LIC model for encoding and decoding a current video block of the video. The set of model parameter values ​​are updated based on the set of adjustment parameter values. In addition, a bitstream is generated based on the updated values ​​of the set of model parameters and stored in a non-transitory computer-readable recording medium.

[0134] Implementations of the present disclosure may be described in view of the following clauses, and features of the following clauses may be combined in any reasonable manner.

[0135] Clause 1. A method for video processing, comprising: for a conversion between a current video block of a video and a bitstream of the video, obtaining values ​​of a set of adjustment parameters associated with values ​​of a set of model parameters of a local illumination compensation (LIC) model used to encode and decode the current video block; updating the values ​​of the set of model parameters based on the values ​​of the set of adjustment parameters; and performing the conversion based on the updated values ​​of the set of model parameters.

[0136] Clause 2. The method of clause 1, wherein the set of model parameters comprises at least one of: a scaling amount of the LIC model, or an offset of the LIC model.

[0137] Clause 3. The method of clause 2, wherein the set of adjustment parameters comprises at least one of: a scaling adjustment parameter for updating a value of the scaling amount, or an offset adjustment parameter for updating a value of the offset.

[0138] Clause 4. A method according to Clause 3, wherein updating the values ​​of the set of model parameters includes at least one of the following: updating the value of the scaling amount based on the sum of the value of the scaling amount and the value of the scaling amount adjustment parameter, or updating the value of the offset based on the sum of the value of the offset and the value of the offset adjustment parameter.

[0139] Clause 5. A method according to any one of clauses 1 to 4, wherein performing the conversion based on the updated values ​​comprises: obtaining target values ​​of the set of model parameters by shifting or clipping at least one of the updated values; and performing the conversion based on the target values.

[0140] Clause 6. The method of any one of clauses 1 to 5, wherein the current video block is encoded using one of: an advanced motion vector prediction (AMVP) mode, an affine AMVP mode, a merge mode, or a sub-block merge mode.

[0141] Clause 7. A method according to any one of clauses 1 to 5, wherein the set of adjustment parameters includes a first adjustment parameter, and obtaining the value of the set of adjustment parameters includes: selecting a value for the first adjustment parameter from multiple candidate values ​​for the first adjustment parameter based on at least one of the following: the size of the current video block, or a quantization parameter (QP) value used to encode and decode the current video block.

[0142] Clause 8. The method of clause 7, wherein the number of candidate values ​​in the plurality of candidate values ​​depends on at least one of: the size of the current video block, or the QP value used to encode and decode the current video block.

[0143] Clause 9. The method of any one of clauses 1 to 5, wherein the values ​​of the set of adjustment parameters depend on a level of the current video block.

[0144] Clause 10. A method according to any one of clauses 1 to 5, wherein the video also includes a first video block encoded and decoded using the LIC mode before the current video block, and obtaining the values ​​of the set of adjustment parameters includes: determining the values ​​of the set of adjustment parameters based on statistics of the first video block and the values ​​of the set of adjustment parameters used to encode and decode the first video block.

[0145] Clause 11. A method according to any one of clauses 1 to 10, wherein the video also includes a second video block different from the current video block, and information about whether to update the values ​​of the set of model parameters used to encode and decode the second video block depends on at least one of the following: a sequence of the second video blocks, a frame of the second video block, a size of the second video block, or a QP value used to encode and decode the second video block.

[0146] Clause 12. A method according to any one of clauses 1 to 11, wherein the video is associated with multiple color planes and the values ​​of the set of model parameters for encoding and decoding a video block associated with at least one color plane of the multiple color planes are updated.

[0147] Clause 13. The method of clause 12, wherein the values ​​of the set of adjustment parameters are different for each color plane of the plurality of color planes.

[0148] Clause 14. The method of clause 12, wherein the values ​​of the set of adjustment parameters are the same for all color planes of the plurality of color planes.

[0149] Clause 15. The method of clause 12, wherein the values ​​of the set of adjustment parameters are the same for a subset of the plurality of color planes.

[0150] Clause 16. A method according to any one of clauses 1 to 15, wherein the set of adjustment parameters includes a second adjustment parameter, and the value of the second adjustment parameter or an indication of the value of the second adjustment parameter is indicated in the bitstream at one of the following: picture level, slice level, tile level, codec tree unit (CTU) level, codec unit (CU) level, prediction unit (PU) level, or transform unit (TU) level.

[0151] Clause 17. The method of Clause 16, wherein the value for the second adjustment parameter is selected from a set of candidate values, and the indication comprises an index for the value for the second adjustment parameter.

[0152] Clause 18. The method of clause 16, wherein the value for the second adjustment parameter is equal to a product of a predefined number and K, K being a real number and included in the indication.

[0153] Clause 19. The method of clause 18, wherein the predefined number is 1 / 64 or 1 / 32.

[0154] Clause 20. The method of clause 17, wherein the set of candidate values ​​depends on codec information of the current video block.

[0155] Clause 21. The method of clause 20, wherein the codec information comprises at least one of: slice type, picture type, temporal layer, QP, block size, codec mode, or color component.

[0156] Clause 22. The method of clause 16, wherein the value or the indication for the second adjustment parameter is encoded in a predictive manner.

[0157] Clause 23. The method of clause 22, wherein the value for the second adjustment parameter is determined based on a value for the second adjustment parameter of a neighboring video block used to encode the current video block.

[0158] Clause 24. The method of clause 16, wherein a syntax element indicating the value for the second adjustment parameter is binarized as one of: a fixed length code, a unary code, a truncated unary code, or an Exponential Golomb code.

[0159] Clause 25. The method of clause 16, wherein a syntax element indicative of the value for the second adjustment parameter is encoded using at least one codec context in arithmetic coding.

[0160] Clause 26. The method of clause 16, wherein a syntax element indicative of the value for the second adaptation parameter is encoded using bypass encoding.

[0161] Clause 27. The method of clause 16, wherein the sign and the absolute value of the second adjustment parameter are separately encoded and decoded.

[0162] Clause 28. A method according to any one of clauses 17 or 20 to 21, wherein the set of candidate values ​​is indicated in one of the following: a slice header, a picture header, a picture parameter set (PPS), a sequence parameter set (SPS), or an adaptation parameter set (APS).

[0163] Clause 29. The method of any one of clauses 1 to 28, wherein if a motion vector (MV) candidate list for the current video block includes at least one candidate coded using LIC mode, at least one additional candidate is added to the MV candidate list.

[0164] Clause 30. A method according to clause 29, wherein the current video block is encoded and decoded using one of the following: merge mode, merge mode with motion vector difference (MMVD) mode, combined inter-frame intra-frame prediction (CIIP) mode, affine merge mode, sub-block merge mode, or template matching merge mode.

[0165] Clause 31. A method according to any one of clauses 29 to 30, wherein the MV candidate is one of: A merge list, an affine merge list, or a template matching merge list.

[0166] Clause 32. A method according to any one of clauses 29 to 31, wherein the at least one additional candidate includes a first additional candidate corresponding to a first candidate among the at least one candidate, and the first additional candidate is determined based on information of the first candidate and a reference value of the set of model parameters, and the reference value is obtained by using the adjustment value of the set of model parameters for encoding and decoding the first candidate using the set of adjustment parameters.

[0167] Clause 33. The method of clause 32, wherein values ​​of the set of tuning parameters are selected and used in a manner similar to the values ​​of the set of tuning parameters used to update the values ​​of the set of model parameters.

[0168] Clause 34. The method of any of clauses 29 to 33, wherein the at least one candidate comprises a plurality of candidates encoded using the LIC mode, and the at least one additional candidate is generated based on a portion of the plurality of candidates.

[0169] Clause 35. The method of clause 32, wherein the current video block is encoded using a merge mode, and values ​​of the set of adjustment parameters are selected based on a type of the merge mode.

[0170] Clause 36. The method of clause 32, wherein the first additional candidate is added to the MV candidate list after the first candidate.

[0171] Clause 37. The method of clause 32, wherein the first additional candidate immediately follows the first candidate in the MV candidate list.

[0172] Clause 38. The method of any of clauses 32 to 37, wherein the at least one additional candidate comprises a plurality of additional candidates determined based on the first additional candidates having different values ​​of the set of adjustment parameters.

[0173] Clause 39. A method according to any one of clauses 29 to 38, wherein the at least one candidate comprises a plurality of candidates encoded using LIC mode, and one or more additional candidates are determined for each of the plurality of candidates and added to the MV candidate list.

[0174] Clause 40. A method according to any one of clauses 1 to 28, wherein if the motion candidate list for merge mode used for encoding and decoding the current video block includes a second candidate encoded using LIC mode, a second additional candidate is determined based on information of the second candidate and reference values ​​of the set of model parameters, the reference values ​​are obtained by adjusting values ​​of the set of model parameters used for encoding and decoding the second candidate using the set of adjustment parameters, and the values ​​of the set of adjustment parameters are indicated in the bitstream.

[0175] Clause 41. The method of clause 40, wherein the values ​​of the set of adjustment parameters are determined and used based on at least one of: the type of the merge mode, the size of the current video block, or the QP.

[0176] Clause 42. A method according to any one of clauses 1 to 41, wherein information about at least one of the following items depends on the dimension of the current video block or the QP used to encode and decode the current video block: whether to indicate a first syntax element in the bitstream, the first syntax element indicating first information about whether the value of the set of adjustment parameters is used to encode and decode the current video block, or how to indicate the first syntax element in the bitstream.

[0177] Clause 43. The method of clause 42, wherein the first syntax element is indicated in the bitstream.

[0178] Clause 44. The method of any one of clauses 42 to 43, wherein the first information depends on whether the current video block is encoded using AMVP mode or merge mode.

[0179] Clause 45. The method of any one of clauses 42 to 43, wherein the first information depends on whether the current video block is uni-directionally predicted or bi-directionally predicted.

[0180] Clause 46. The method of any of clauses 42 to 43, wherein the current video block is encoded using a merge prediction mode, and the first information depends on a type of the merge prediction mode.

[0181] Clause 47. A method according to any one of clauses 42 to 46, wherein the first syntax element is indicated in the bitstream for at least one of: a Y color plane, a U color plane, or a V color plane.

[0182] Clause 48. The method of any one of clauses 42 to 47, wherein the first syntax element is indicated in the bitstream if the current video block is larger than a predefined size.

[0183] Clause 49. The method of any one of clauses 42 to 47, wherein the first syntax element is indicated in the bitstream if a sum of a width and a height of the current video block is greater than a predefined threshold.

[0184] Clause 50. The method of any one of clauses 42 to 47, wherein the first syntax element is indicated in the bitstream if the current video block is smaller than a predefined size.

[0185] Clause 51. The method of any one of clauses 42 to 47, wherein the first syntax element is indicated in the bitstream if a sum of a width and a height of the current video block is less than a predefined threshold.

[0186] Clause 52. The method of any one of clauses 42 to 47, wherein the first syntax element is indicated in the bitstream if the current video block is greater than or equal to a predefined size.

[0187] Clause 53. The method of any one of clauses 42 to 47, wherein the first syntax element is indicated in the bitstream if a sum of a width and a height of the current video block is greater than or equal to a predefined threshold.

[0188] Clause 54. The method of any one of clauses 42 to 47, wherein the first syntax element is indicated in the bitstream if the current video block is less than or equal to a predefined size.

[0189] Clause 55. The method of any one of clauses 42 to 47, wherein the first syntax element is indicated in the bitstream if a sum of a width and a height of the current video block is less than or equal to a predefined threshold.

[0190] Clause 56. The method of any one of clauses 42 to 47, wherein the first syntax element is indicated in the bitstream if the QP is greater than a predefined threshold.

[0191] Clause 57. The method of any one of clauses 42 to 47, wherein the first syntax element is indicated in the bitstream if the QP is less than a predefined threshold.

[0192] Clause 58. The method of any one of clauses 42 to 47, wherein the first syntax element is indicated in the bitstream if the QP is greater than or equal to a predefined threshold.

[0193] Clause 59. The method of any one of clauses 42 to 47, wherein the first syntax element is indicated in the bitstream if the QP is less than or equal to a predefined threshold.

[0194] Clause 60. The method of any one of clauses 42 to 59, wherein the first syntax element is binarized into one of: a fixed length code, a unary code, a truncated unary code, or an Exponential Golomb code.

[0195] Clause 61. A method according to any of clauses 42 to 59, wherein the first syntax element is encoded using at least one codec context in arithmetic coding.

[0196] Clause 62. The method of any one of clauses 42 to 59, wherein the first syntax element is encoded using bypass codec.

[0197] Clause 63. A method according to any one of clauses 1 to 62, wherein whether and / or how to apply the method is indicated at one of: sequence level, group of picture level, picture level, slice level, or tile group level.

[0198] Clause 64. A method according to any one of clauses 1 to 62, wherein whether and / or how the method is applied is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.

[0199] Clause 65. A method according to any one of clauses 1 to 62, wherein whether and / or how to apply the method is indicated at an area containing more than one sample or pixel.

[0200] Clause 66. A method according to clause 65, wherein the region includes at least one of the following: a prediction block (PB), a transform block (TB), a codec block (CB), a prediction unit (PU), a transform unit (TU), a codec unit (CU), a virtual pipeline data unit (VPDU), a codec tree unit (CTU), a CTU row, a slice, a tile, or a sub-picture.

[0201] Clause 67. The method of any one of clauses 1 to 62, wherein whether and / or how to apply the method depends on the encoded information.

[0202] Clause 68. The method of clause 67, wherein the encoded information comprises at least one of: block size, color format, single tree partitioning, dual tree partitioning, color component, slice type, or picture type.

[0203] Clause 69. The method of any one of clauses 1 to 68, wherein the converting comprises encoding the current video block into the bitstream.

[0204] Clause 70. The method of any one of clauses 1 to 68, wherein the converting comprises decoding the current video block from the bitstream.

[0205] Clause 71. An apparatus for video processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of clauses 1 to 70.

[0206] Clause 72. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method of any one of clauses 1 to 70.

[0207] Clause 73. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by an apparatus for video processing, wherein the method comprises: obtaining values ​​of a set of adjustment parameters associated with values ​​of a set of model parameters of an LIC model used to encode and decode a current video block of the video; updating the values ​​of the set of model parameters based on the values ​​of the set of adjustment parameters; and generating the bitstream based on the updated values ​​of the set of model parameters.

[0208] Clause 74. A method for storing a bitstream of a video, comprising: obtaining values ​​of a set of adjustment parameters associated with values ​​of a set of model parameters of an LIC model used to encode and decode a current video block of the video; updating the values ​​of the set of model parameters based on the values ​​of the set of adjustment parameters; generating the bitstream based on the updated values ​​of the set of model parameters; and storing the bitstream in a non-transitory computer-readable recording medium. Example Device

[0209] Fig.31 A block diagram of a computing device 3100 in which various embodiments of the present disclosure may be implemented is shown. The computing device 3100 may be implemented as or included in the source device 110 (or video encoder 114 or 200) or the destination device 120 (or video decoder 124 or 300).

[0210] It should be understood that Fig.31 The computing device 3100 shown is for illustration purposes only and does not in any way imply any limitation on the functionality and scope of the embodiments of the present disclosure.

[0211] like Fig.31As shown, computing device 3100 includes a general computing device 3100. Computing device 3100 may include at least one or more processors or processing units 3110, memory 3120, storage unit 3130, one or more communication units 3140, one or more input devices 3150, and one or more output devices 3160.

[0212] In some embodiments, the computing device 3100 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server provided by a service provider, a large computing device, etc. The user terminal can be, for example, any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet computer, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It can be expected that the computing device 3100 can support any type of interface with the user (such as a "wearable" circuit, etc.).

[0213] The processing unit 3110 may be a physical or virtual processor and may implement various processes based on a program stored in the memory 3120. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capabilities of the computing device 3100. The processing unit 3110 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0214] The computing device 3100 typically includes various computer storage media. Such media can be any media accessible to the computing device 3100, including but not limited to volatile and non-volatile media, or removable and non-removable media. The memory 3120 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), or flash memory), or any combination thereof. The storage unit 3130 can be any removable or non-removable medium, and can include a machine-readable medium, such as a memory, a flash drive, a disk, or other media, which can be used to store information and / or data and can be accessed in the computing device 3100.

[0215] The computing device 3100 may also include additional removable / non-removable, volatile / non-volatile storage media. Fig.31Not shown in the figure, but a magnetic disk drive for reading and / or writing a removable and nonvolatile magnetic disk and an optical disk drive for reading and / or writing a removable nonvolatile optical disk may be provided. In this case, each drive may be connected to a bus (not shown) via one or more data medium interfaces.

[0216] The communication unit 3140 communicates with another computing device via a communication medium. In addition, the functions of the components in the computing device 3100 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 3100 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.

[0217] Input device 3150 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 3160 may be one or more of various output devices, such as a display, speaker, printer, etc. Through communication unit 3140, computing device 3100 may also communicate with one or more external devices (not shown) such as storage devices and display devices, one or more of which enable a user to interact with computing device 3100, or any device (e.g., a network card, a modem, etc.) that enables computing device 3100 to communicate with one or more other computing devices if necessary. Such communication may be performed via an input / output (I / O) interface (not shown).

[0218] In some embodiments, some or all components of the computing device 3100 may also be arranged in a cloud computing architecture, rather than being integrated in a single device. In a cloud computing architecture, each component can be provided remotely and work together to implement the functions described in the present disclosure. In some embodiments, cloud computing provides computing, software, data access and storage services, which does not require the end user to understand the physical location or configuration of the system or hardware that provides these services. In various embodiments, cloud computing uses a suitable protocol to provide services through a wide area network (such as the Internet). For example, a cloud computing provider provides applications through a wide area network, which can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data can be stored on a server at a remote location. The computing resources in a cloud computing environment can be merged or distributed at the location of a remote data center. Cloud computing infrastructure can provide services through a shared data center, although they act as a single access point for users. Therefore, cloud computing architecture can be used to provide components and functions described herein from a service provider at a remote location. Alternatively, they can be provided from a conventional server or directly installed or otherwise installed on a client device.

[0219] The computing device 3100 may be used to implement video encoding / decoding in embodiments of the present invention. The memory 3120 may include one or more video encoding / decoding modules 3125 having one or more program instructions. These modules may be accessed and executed by the processing unit 3110 to perform the functions of various embodiments described herein.

[0220] In an example embodiment performing video encoding and decoding, the input device 3150 may receive video data as input to be encoded 3170. For example, the video data may be processed by the video codec module 3125 to generate an encoded bitstream. The encoded bitstream may be provided as output 3180 via the output device 3160.

[0221] In an example embodiment performing video decoding, input device 3150 may receive an encoded bitstream as input 3170. The encoded bitstream may be processed, for example, by video codec module 3125 to generate decoded video data. The decoded video data may be provided as output 3180 via output device 3160.

[0222] Although the present disclosure has been specifically shown and described with reference to its preferred embodiments, it will be appreciated by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present application, as defined by the appended claims. These changes are intended to be covered by the scope of the present application. Therefore, the foregoing description of the embodiments of the present application is not intended to be limiting.

Claims

1. A method for video processing, comprising: For conversion between a current video block of a video and a bitstream of the video, obtaining values ​​of a set of adjustment parameters associated with values ​​of a set of model parameters of a local illumination compensation (LIC) model used for encoding and decoding the current video block; updating the values ​​of the set of model parameters based on the values ​​of the set of tuning parameters; as well as The converting is performed based on updated values ​​of the set of model parameters.

2. The method according to claim 1, wherein the set of model parameters comprises at least one of the following: The amount of scaling of the LIC model, or The offset of the LIC model.

3. The method according to claim 2, wherein the set of adjustment parameters comprises at least one of the following: a scaling adjustment parameter for updating the value of the scaling amount, or An offset adjustment parameter is used to update the value of the offset.

4. The method of claim 3, wherein updating the values ​​of the set of model parameters comprises at least one of: updating the value of the scaling amount based on the sum of the value of the scaling amount and the value of the scaling amount adjustment parameter, or The value of the offset is updated based on a sum of the value of the offset and a value of the offset adjustment parameter.

5. The method of any one of claims 1 to 4, wherein performing the conversion based on the updated value comprises: obtaining target values ​​of the set of model parameters by shifting or clipping at least one of the updated values; as well as The converting is performed based on the target value.

6. The method according to any one of claims 1 to 5, wherein the current video block is encoded using one of: Advanced Motion Vector Prediction (AMVP) mode, Affine AMVP mode, Merge mode, or Sub-block merging mode.

7. The method according to any one of claims 1 to 5, wherein the set of adjustment parameters includes a first adjustment parameter, and obtaining the value of the set of adjustment parameters includes: A value for the first adjustment parameter is selected from a plurality of candidate values ​​for the first adjustment parameter based on at least one of: The size of the current video block, or A quantization parameter (QP) value used for encoding and decoding the current video block.

8. The method of claim 7, wherein the number of candidate values ​​in the plurality of candidate values ​​depends on at least one of the following: the size of the current video block, or The QP value used for encoding and decoding the current video block.

9. The method of any one of claims 1 to 5, wherein the values ​​of the set of adjustment parameters depend on a level of the current video block.

10. The method according to any one of claims 1 to 5, wherein the video further comprises a first video block encoded and decoded using the LIC mode before the current video block, and obtaining the values ​​of the set of adjustment parameters comprises: The values ​​of the set of adjustment parameters are determined based on statistics of the first video block and values ​​of the set of adjustment parameters used to encode and decode the first video block.

11. The method according to any one of claims 1 to 10, wherein the video further comprises a second video block different from the current video block, and the information about whether to update the values ​​of the set of model parameters for encoding and decoding the second video block depends on at least one of the following: a sequence of said second video blocks, a frame of the second video block, the size of the second video block, or A QP value for encoding and decoding the second video block.

12. The method according to any one of claims 1 to 11, wherein the video is associated with a plurality of color planes, and the values ​​of the set of model parameters for encoding and decoding a video block associated with at least one color plane of the plurality of color planes are updated.

13. The method of claim 12, wherein the values ​​of the set of adjustment parameters are different for each color plane of the plurality of color planes.

14. The method of claim 12, wherein the values ​​of the set of adjustment parameters are the same for all color planes of the plurality of color planes.

15. The method of claim 12, wherein the values ​​of the set of adjustment parameters are the same for a subset of the plurality of color planes.

16. The method according to any one of claims 1 to 15, wherein the set of adjustment parameters comprises a second adjustment parameter, a value for the second adjustment parameter or an indication of the value for the second adjustment parameter being indicated in the bitstream at one of: Picture level, Stripe level, Tile level, Codec Tree Unit (CTU) level, Codec Unit (CU) level, Prediction Unit (PU) level, or Transform Unit (TU) level.

17. The method of claim 16, wherein the value for the second tuning parameter is selected from a set of candidate values ​​and the indication comprises an index for the value for the second tuning parameter.

18. The method of claim 16, wherein the value for the second adjustment parameter is equal to a product of a predefined number and K, K being a real number and included in the indication. The method according to claim 18 , wherein the predefined number is 1 / 64 or 1 / 32.

20. The method of claim 17, wherein the set of candidate values ​​depends on codec information of the current video block.

21. The method according to claim 20, wherein the codec information comprises at least one of the following: Strip type, Image type, Time layer, QP, Block size, Codec mode, or Color components.

22. The method of claim 16, wherein the value or the indication for the second adjustment parameter is encoded in a predictive manner.

23. The method of claim 22, wherein the value for the second adjustment parameter is determined based on a value for the second adjustment parameter used to encode a neighboring video block of the current video block.

24. The method of claim 16, wherein a syntax element indicating the value for the second adjustment parameter is binarized as one of: Fixed length code, Unary code, truncate the unary code, or Exponential Columbus code.

25. The method of claim 16, wherein a syntax element indicating the value for the second adjustment parameter is encoded using at least one codec context in arithmetic coding.

26. The method of claim 16, wherein a syntax element indicating the value for the second adjustment parameter is encoded using bypass codec.

27. The method of claim 16, wherein a sign and an absolute value of the second adjustment parameter are encoded and decoded separately.

28. A method according to any one of claims 17 or 20 to 21, wherein the set of candidate values ​​is indicated in one of: Strip head, Picture header, Picture Parameter Set (PPS), Sequence Parameter Set (SPS), or Adaptive Parameter Set (APS).

29. The method of any one of claims 1 to 28, wherein if a motion vector (MV) candidate list for the current video block includes at least one candidate coded using LIC mode, at least one additional candidate is added to the MV candidate list.

30. The method of claim 29, wherein the current video block is encoded using one of: Merge mode, Merge mode with motion vector difference (MMVD) mode, Combined Inter-Intra Prediction (CIIP) mode, Affine merge mode, Sub-block merging mode, or Template matching merge mode.

31. The method according to any one of claims 29 to 30, wherein the MV candidate is one of: Merge the list, Affine merge lists, or Template matching merge list.

32. A method according to any one of claims 29 to 31, wherein the at least one additional candidate includes a first additional candidate corresponding to a first candidate among the at least one candidate, and the first additional candidate is determined based on information of the first candidate and a reference value of the set of model parameters, and the reference value is obtained by using the adjustment value of the set of model parameters for encoding and decoding the first candidate using the set of adjustment parameters.

33. The method of claim 32, wherein values ​​of the set of tuning parameters are selected and used in a manner similar to the values ​​of the set of tuning parameters used to update the values ​​of the set of model parameters.

34. The method according to any one of claims 29 to 33, wherein the at least one candidate comprises a plurality of candidates encoded and decoded using the LIC mode, and the at least one additional candidate is generated based on a portion of the plurality of candidates.

35. The method of claim 32, wherein the current video block is encoded using a merge mode, and values ​​of the set of adjustment parameters are selected based on a type of the merge mode.

36. The method of claim 32, wherein the first additional candidate is added to the MV candidate list after the first candidate.

37. The method of claim 32, wherein the first additional candidate immediately follows the first candidate in the MV candidate list.

38. The method of any one of claims 32 to 37, wherein the at least one additional candidate comprises a plurality of additional candidates determined based on the first additional candidates having different values ​​of the set of adjustment parameters.

39. The method of any one of claims 29 to 38, wherein the at least one candidate comprises a plurality of candidates encoded using LIC mode, and one or more additional candidates are determined for each of the plurality of candidates and added to the MV candidate list.

40. The method according to any one of claims 1 to 28, wherein if the motion candidate list for merge mode used for encoding the current video block includes a second candidate encoded using LIC mode, a second additional candidate is determined based on information of the second candidate and reference values ​​of the set of model parameters, the reference values ​​are obtained by adjustment values ​​of the set of model parameters used for encoding the second candidate using the set of adjustment parameters, and the values ​​of the set of adjustment parameters are indicated in the bitstream.

41. The method of claim 40, wherein the values ​​of the set of adjustment parameters are determined and used based on at least one of: The type of the merge mode, The size of the current video block, or QP.

42. The method of any one of claims 1 to 41, wherein information about at least one of the following depends on a dimension of the current video block or a QP used for encoding and decoding the current video block: whether to indicate a first syntax element in the bitstream, the first syntax element indicating first information about whether the value of the set of adjustment parameters is used for encoding and decoding the current video block, or How to indicate the first syntax element in the bitstream.

43. The method of claim 42, wherein the first syntax element is indicated in the bitstream.

44. The method according to any one of claims 42 to 43, wherein the first information depends on whether the current video block is encoded using AMVP mode or merge mode.

45. The method of any one of claims 42-43, wherein the first information depends on whether the current video block is uni-predicted or bi-predicted.

46. ​​The method of any one of claims 42 to 43, wherein the current video block is encoded using a merge prediction mode, and the first information depends on a type of the merge prediction mode.

47. The method of any one of claims 42 to 46, wherein the first syntax element is indicated in the bitstream for at least one of: Y color plane, U color plane, or V color plane.

48. The method of any one of claims 42 to 47, wherein the first syntax element is indicated in the bitstream if the current video block is larger than a predefined size.

49. The method of any one of claims 42 to 47, wherein the first syntax element is indicated in the bitstream if a sum of a width and a height of the current video block is greater than a predefined threshold.

50. The method of any one of claims 42 to 47, wherein the first syntax element is indicated in the bitstream if the current video block is smaller than a predefined size.

51. The method of any one of claims 42 to 47, wherein the first syntax element is indicated in the bitstream if a sum of a width and a height of the current video block is less than a predefined threshold.

52. The method of any one of claims 42 to 47, wherein the first syntax element is indicated in the bitstream if the current video block is greater than or equal to a predefined size.

53. The method of any one of claims 42 to 47, wherein the first syntax element is indicated in the bitstream if a sum of a width and a height of the current video block is greater than or equal to a predefined threshold.

54. The method of any one of claims 42 to 47, wherein the first syntax element is indicated in the bitstream if the current video block is less than or equal to a predefined size.

55. The method of any one of claims 42 to 47, wherein the first syntax element is indicated in the bitstream if a sum of a width and a height of the current video block is less than or equal to a predefined threshold.

56. The method of any one of claims 42 to 47, wherein the first syntax element is indicated in the bitstream if the QP is greater than a predefined threshold.

57. The method of any one of claims 42 to 47, wherein the first syntax element is indicated in the bitstream if the QP is less than a predefined threshold.

58. The method of any one of claims 42 to 47, wherein the first syntax element is indicated in the bitstream if the QP is greater than or equal to a predefined threshold.

59. The method of any one of claims 42 to 47, wherein the first syntax element is indicated in the bitstream if the QP is less than or equal to a predefined threshold.

60. The method of any one of claims 42 to 59, wherein the first syntax element is binarized as one of: Fixed length code, Unary code, truncate the unary code, or Exponential Columbus code.

61. The method according to any one of claims 42 to 59, wherein the first syntax element is coded using at least one codec context in arithmetic coding.

62. The method of any one of claims 42 to 59, wherein the first syntax element is encoded using bypass codec.

63. A method according to any one of claims 1 to 62, wherein whether and / or how to apply the method is indicated at one of: Sequence level, Picture group level, Picture level, Stripe level, or Tile group level.

64. A method according to any one of claims 1 to 62, wherein whether and / or how the method is to be applied is indicated in one of: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependency Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), Strip header, or Tile group header.

65. A method according to any one of claims 1 to 62, wherein whether and / or how to apply the method is indicated at an area containing more than one sample or pixel.

66. The method of claim 65, wherein the region comprises at least one of: Prediction Block (PB), Transform Block (TB), Codec Block (CB), Prediction Unit (PU), Transformation Unit (TU), Codec Unit (CU), Virtual Pipeline Data Unit (VPDU), Codec Tree Unit (CTU), CTU line, Strips, tiles, or Sub-picture.

67. The method according to any one of claims 1 to 62, wherein whether and / or how the method is applied depends on the encoded information.

68. The method of claim 67, wherein the encoded information comprises at least one of: Block size, Color format, Single tree partitioning, Double tree partitioning, Color components, Strip type, or Image type.

69. The method of any one of claims 1 to 68, wherein the converting comprises encoding the current video block into the bitstream.

70. The method of any one of claims 1 to 68, wherein the converting comprises decoding the current video block from the bitstream.

71. An apparatus for video processing, comprising a processor and a non-volatile memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method according to any one of claims 1 to 70.

72. A non-transitory computer readable storage medium storing instructions that cause a processor to perform the method according to any one of claims 1 to 70.

73. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by an apparatus for video processing, wherein the method comprises: Obtaining values ​​of a set of adjustment parameters associated with values ​​of a set of model parameters of a LIC model used for encoding and decoding a current video block of the video; updating the values ​​of the set of model parameters based on the values ​​of the set of tuning parameters; as well as The bitstream is generated based on updated values ​​of the set of model parameters.

74. A method for storing a bitstream of a video, comprising: Obtaining values ​​of a set of adjustment parameters associated with values ​​of a set of model parameters of a LIC model used for encoding and decoding a current video block of the video; updating the values ​​of the set of model parameters based on the values ​​of the set of tuning parameters; generating the bitstream based on updated values ​​of the set of model parameters; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

Images