Method and device for video processing and medium

Abstract

Embodiments of the present disclosure provide a solution for video processing. A method for video processing is presented. The method comprises the following steps: for conversion between a video unit of a video and a bit stream of the video unit, obtaining first motion information and second motion information of the video unit; refining the second motion information by using the first motion information during a motion refining process of the video unit; and performing a conversion based on the refined first motion information and the second motion information.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure generally relate to video processing technology, and more particularly, to improvements on template matching. BACKGROUND

[0002] Nowadays, digital video capability is being applied to various aspects of people's life. Various types of video compression techniques have been proposed for video coding / decoding, such as MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10 Advanced Video Coding (AVC), ITU-T H.265 High Efficiency Video Coding (HEVC) standard, Versatile Video Coding (VVC) standard. However, it is generally desirable to further improve the coding efficiency of video coding techniques. SUMMARY

[0003] Embodiments of the present disclosure provide a solution for video processing.

[0004] In a first aspect, a method for video processing is proposed. The method comprises: obtaining first motion information and second motion information of a video unit of a video for conversion between the video unit and a bitstream of the video unit; refining the second motion information by using the first motion information during a motion refinement process of the video unit; and performing the conversion based on the refined first motion information and the second motion information.

[0005] In a second aspect, an apparatus for video processing is proposed. The apparatus comprises a processor and a non-transitory memory having instructions thereon. The instructions, when executed by the processor, cause the processor to perform the method according to the first aspect of the present disclosure.

[0006] In a third aspect, a non-transitory computer-readable storage medium is proposed. The non-transitory computer-readable storage medium stores instructions that cause a processor to perform the method according to the first aspect of the present disclosure.

[0007] In a fourth aspect, another non-transitory computer-readable recording medium is proposed. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method executed by an apparatus for video processing. The method comprises: obtaining first motion information and second motion information of a video unit of the video; refining the second motion information by using the first motion information during a motion refinement process of the video unit; and generating the bitstream based on the refined first motion information and the second motion information.

[0008] In a fifth aspect, a method for storing a bitstream of a video is proposed. The method includes obtaining first motion information and second motion information of a video unit of the video; refining the second motion information by using the first motion information during a motion refinement process of the video unit; generating the bitstream based on the refined first motion information and the second motion information; and storing the bitstream in a non-transitory computer-readable recording medium.

[0009] This Summary is intended to introduce some of the concepts that are further described in the detailed description below. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and other objects, features and advantages of the example embodiments of the present disclosure will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which like reference characters refer to the like components throughout the drawings.

[0011] FIG. 1 A block diagram showing an example video coding system is shown in accordance with some embodiments of the present disclosure;

[0012] FIG. 2 A block diagram showing a first example video encoder is shown in accordance with some embodiments of the present disclosure;

[0013] FIG. 3 A block diagram showing an example video decoder is shown in accordance with some embodiments of the present disclosure;

[0014] FIG. 4 An example of an encoder block diagram of VVC is shown;

[0015] FIG. 5 67 intra prediction modes are shown;

[0016] FIG. 6A And FIG. 6B Reference samples for wide-angle intra prediction are shown;

[0017] FIG. 7 Discontinuity in case of direction exceeding 45° is shown;

[0018] FIG. 8A And FIG. 8B MMVD search points are shown;

[0019] FIG. 9 An illustration for symmetric MVD mode is shown;

[0020] FIG. 10 Extended CU region used in BDOF is shown;

[0021] FIG. 11 Top and left neighboring blocks used in CIIP weight derivation are shown;

[0022] FIG. 12 Control point based affine motion model is shown;

[0023] FIG. 13 Affine MVFs for each subblock are shown;

[0024] FIG. 14 Location of inherited affine motion predictor is shown;

[0025] FIG. 15 Control point motion vector inheritance is shown;

[0026] FIG. 16 Location of candidate positions for constructed affine Merge mode is shown;

[0027] FIG. 17 Illustration of motion vector usage for proposed combined method is shown;

[0028] FIG. 18 Subblock MV VSB and pixel delta v(i,j) (red arrows) are shown;

[0029] FIG. 19A And FIG. 19B SbTMVP process in VVC is shown;

[0030] FIG. 20 Local illumination compensation is shown;

[0031] FIG. 21 No downsampling for short side is shown;

[0032] FIG. 22 Decoder-side motion vector refinement is shown;

[0033] FIG. 23 Diamond-shaped region in search region is shown;

[0034] FIG. 24 Location of spatial Merge candidate is shown;

[0035] FIG. 25 Candidate pairs considered for redundancy check for spatial Merge candidate are shown;

[0036] FIG. 26 Illustration of motion vector scaling for temporal Merge candidate is shown;

[0037] FIG. 27 Candidate positions, C0 and C1, for temporal Merge candidate are shown;

[0038] FIG. 28 VVC spatial neighboring blocks of the current block are shown;

[0039] FIG. 29 An illustration of virtual blocks in the i-th round search is shown;

[0040] FIG. 30 An example of GPM partitioning grouped at the same angle is shown;

[0041] FIG. 31 Unidirectional prediction MV selection for geometric partition mode is shown;

[0042] FIG. 32 An example generation of the blending weight w0 using geometric partition mode is shown;

[0043] FIG. 33 Spatial neighboring blocks used to derive spatial Merge candidates are shown;

[0044] FIG. 34 Template matching is performed on a search region around the initial MV is shown;

[0045] FIG. 35 An illustration of sub-blocks for OBMC application is shown;

[0046] FIG. 36 SBT position, type and transform type is shown;

[0047] FIG. 37 Neighboring samples used to compute SAD are shown;

[0048] FIG. 38 Neighboring samples used to compute SAD for sub-CU level motion information are shown;

[0049] FIG. 39 An ordering process is shown;

[0050] FIG. 40 A reordering process in the encoder is shown;

[0051] FIG. 41 A reordering process in the decoder is shown;

[0052] FIG. 42 A first HPT and a second HPT are shown;

[0053] FIG. 43 Spatial neighbors used to derive affine Merge / AMVP candidates: (a) for deriving inherited candidates (b) for deriving first type of constructed candidates are shown;

[0054] FIG. 44A diagram showing non-adjacent neighbors to first type of constructed affine Merge / AMVP candidates is shown;

[0055] FIG. 45 A diagram showing different numbers of search points in the same search pattern is shown;

[0056] FIG. 46 A diagram showing different numbers of search points in different search patterns is shown;

[0057] FIG. 47 A flow diagram of a method for video processing according to embodiments of the disclosure is shown; and

[0058] FIG. 48 A block diagram of a computing device in which various embodiments of the disclosure can be implemented is shown.

[0059] Throughout the drawings, identical or similar reference numerals can designate identical or similar elements throughout the several views. DETAILED DESCRIPTION

[0060] The principles of the present disclosure will now be described with reference to some embodiments. It should be understood that the embodiments are described for illustrative purposes only and to help the person skilled in the art to understand and implement the present disclosure, without implying any limitation to the scope of the present disclosure. The disclosure described herein can be implemented in various ways in addition to those described below.

[0061] In the following description and claims, unless otherwise defined, all scientific and technical terms used in this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0062] References in the present disclosure to “one embodiment”, “an embodiment”, “example embodiments”, etc. indicate that the embodiment described can include a particular feature, structure, or characteristic, but every embodiment can not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an example embodiment, it is submitted that the feature, structure, or characteristic is within the knowledge of one of ordinary skill in the art, whether or not explicitly described, to affect such a feature, structure, or characteristic in connection with other embodiments.

[0063] It should be understood that although the terms “first” and “second” etc. can be used herein to describe various elements, the elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element without departing from the scope of the example embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the associated terms.

[0064] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes" and / or "including," when used herein, specify the presence of stated features, elements and / or components, but do not preclude the presence or addition of one or more other features, elements, components and / or combinations thereof. Example Environment

[0065] FIG. 1 is a block diagram illustrating an example video coding system 100 that can utilize the techniques of this disclosure. As shown, video coding system 100 can include a source device 110 and a destination device 120. Source device 110 can also be referred to as a video encoding device, and destination device 120 can also be referred to as a video decoding device. In operation, source device 110 can be configured to generate encoded video data, and destination device 120 can be configured to decode the encoded video data generated by source device 110. Source device 110 can include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0066] Video source 112 can include a source such as a video capture device. Examples of video capture devices include, but are not limited to, an interface to receive video data from a video content provider, a computer graphics system to generate video data, and / or a combination thereof.

[0067] Video data can include one or more pictures. Video encoder 114 encodes the video data from video source 112 to generate a bitstream. The bitstream can include a sequence of bits that form an encoded representation of the video data. The bitstream can include encoded pictures and associated data. An encoded picture is an encoded representation of a picture. The associated data can include sequence parameter sets, picture parameter sets, and other syntax structures. I / O interface 116 can include a modulator / demodulator and / or a transmitter. The encoded video data can be transmitted directly to destination device 120 via I / O interface 116 by network 130A. The encoded video data can also be stored onto a storage medium / server 130B for access by destination device 120.

[0068] Destination device 120 can include I / O interface 126, video decoder 124, and display device 122. I / O interface 126 can include a receiver and / or a modem. I / O interface 126 can acquire encoded video data from source device 110 or storage medium / server 130B. Video decoder 124 can decode encoded video data. Display device 122 can display the decoded video data to a user. Display device 122 can be integrated with destination device 120, or can be external to destination device 120, which is configured to interface with an external display device.

[0069] Video encoder 114 and video decoder 124 can operate according to a video compression standard, such as the High Efficiency Video Coding (HEVC) standard, Versatile Video Coding (VVC) standard, and other existing and / or future standards.

[0070] FIG. 2 is a block diagram illustrating an example of a video encoder 200 that can be FIG. 1 an example of video encoder 114 in system 100 shown.

[0071] Video encoder 200 can be configured to implement any or all of the techniques of this disclosure. In FIG. 2 example, video encoder 200 includes a plurality of functional components. The techniques described in this disclosure can be shared among the functional components of video encoder 200. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0072] In some embodiments, video encoder 200 can include partitioning unit 201, prediction unit 202, which can include mode select unit 203, motion estimation unit 204, motion compensation unit 205, and intra-prediction unit 206, residual generation unit 207, transform unit 208, quantization unit 209, inverse-quantization unit 210, inverse-transform unit 211, reconstruction unit 212, buffer 213, and entropy encoding unit 214.

[0073] In other examples, video encoder 200 can include more, less, or different functional components. In one example, prediction unit 202 can include an intra-block copy (IBC) unit. The IBC unit can perform prediction in an IBC mode in which at least one reference picture is the picture in which the current video block is located.

[0074] Furthermore, although some components, such as motion estimation unit 204 and motion compensation unit 205, can be integrated, for purposes of explanation, these components are shown as separate components in FIG. 2are shown separately in the example.

[0075] Partition unit 201 can partition a picture into one or more video blocks. Video encoder 200 and video decoder 300 can support various video block sizes.

[0076] Mode selection unit 203 can select one of a plurality of coding modes (intra- or inter-coding), e.g., based on the error results, and provide the resulting intra- or inter-coded block to residual generation unit 207 to generate residual block data and to reconstruction unit 212 to reconstruct the coded block for use as a reference picture. In some examples, mode selection unit 203 can select a combined intra- inter prediction (CIIP) mode in which prediction is based on both an inter prediction signal and an intra prediction signal. In the case of inter prediction, mode selection unit 203 can also select a resolution for a motion vector for the block (e.g., sub-pixel accuracy or integer pixel accuracy).

[0077] To perform inter prediction for a current video block, motion estimation unit 204 can generate motion information for the current video block by comparing one or more reference frames from cache 213 to the current video block. Motion compensation unit 205 can determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from cache 213 other than the picture associated with the current video block.

[0078] Motion estimation unit 204 and motion compensation unit 205 can perform different operations for a current video block, e.g., depending on whether the current video block is in an I slice, a P slice, or a B slice. As used herein, an “I slice” can refer to a portion of a picture composed of macroblocks all of which are based on macroblocks within the same picture. Further, as used herein, in some aspects, a “P slice” and a “B slice” can refer to portions of a picture composed of macroblocks that are independent of macroblocks in the same picture.

[0079] In some examples, motion estimation unit 204 can perform uni-directional prediction for a current video block, and motion estimation unit 204 can search a reference picture of list 0 or list 1 for a reference video block for the current video block. Motion estimation unit 204 can then generate a reference index indicating the reference picture of list 0 or list 1 containing the reference video block and a motion vector indicating a spatial displacement between the current video block and the reference video block. Motion estimation unit 204 can output the reference index, the prediction direction indicator, and the motion vector as motion information for the current video block. Motion compensation unit 205 can generate a predicted video block for the current video block based on the reference video block indicated by the motion information for the current video block.

[0080] Alternatively, in other examples, the motion estimation unit 204 can perform bi-prediction for the current video block. The motion estimation unit 204 can search the reference pictures in list 0 for one reference video block for the current video block and also search the reference pictures in list 1 for another reference video block for the current video block. The motion estimation unit 204 can then generate a plurality of reference indices that indicate the reference pictures in list 0 and list 1 that contain the plurality of reference video blocks and motion vectors that indicate a plurality of spatial displacements between the plurality of reference video blocks and the current video block. The motion estimation unit 204 can output the plurality of reference indices and the plurality of motion vectors for the current video block as motion information for the current video block. The motion compensation unit 205 can generate a predicted video block for the current video block based on the plurality of reference video blocks indicated by the motion information for the current video block.

[0081] In some examples, the motion estimation unit 204 can output a full set of motion information for use in the decoding process at the decoder. Alternatively, in some embodiments, the motion estimation unit 204 can reference the motion information of another video block to signal the motion information of the current video block. For example, the motion estimation unit 204 can determine that the motion information of the current video block is sufficiently similar to the motion information of a neighboring video block.

[0082] In one example, the motion estimation unit 204 can indicate a value in a syntax structure associated with the current video block to the video decoder 300, the value indicating that the current video block has the same motion information as another video block.

[0083] In another example, the motion estimation unit 204 can identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates a difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 can use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0084] As discussed above, the video encoder 200 can signal motion vectors in a predictive manner. Two examples of predictive signaling techniques that can be implemented by the video encoder 200 include advanced motion vector prediction (AMVP) and Merge mode signaling.

[0085] The intra prediction unit 206 can perform intra prediction for the current video block. When the intra prediction unit 206 performs intra prediction for the current video block, the intra prediction unit 206 can generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block can include a predicted video block and various syntax elements.

[0086] Residual generation unit 207 can generate residual data for the current video block by subtracting (e.g., indicated by the minus sign) the prediction video block(s) from the current video block. The residual data for the current video block can include residual video blocks corresponding to different sample components of samples in the current video block.

[0087] In other examples, such as in skip mode, there can be no residual data for the current video block for the current video block, and residual generation unit 207 can not perform the subtraction operation.

[0088] Transform processing unit 208 can generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video blocks associated with the current video block.

[0089] After transform processing unit 208 generates the transform coefficient video blocks associated with the current video block, quantization unit 209 can quantize the transform coefficient video blocks associated with the current video block based on one or more quantization parameter (QP) values associated with the current video block.

[0090] Inverse quantization unit 210 and inverse transform unit 211 can apply inverse quantization and inverse transform, respectively, to the transform coefficient video blocks to reconstruct the residual video blocks from the transform coefficient video blocks. Reconstruction unit 212 can add the reconstructed residual video blocks to corresponding samples from the prediction video block(s) generated by prediction unit 202 to produce a reconstructed video block associated with the current video block for storage in buffer 213.

[0091] After reconstruction unit 212 reconstructs the video block, loop filtering operations can be performed to reduce video block effect artifacts in the video block.

[0092] Entropy encoding unit 214 can receive data from other functional components of video encoder 200. When entropy encoding unit 214 receives data, entropy encoding unit 214 can perform one or more entropy encoding operations to generate entropy encoded data and output a bitstream that includes the entropy encoded data.

[0093] FIG. 3 FIG. 3 is a block diagram illustrating an example of a video decoder 300 that can be FIG. 1 of the video decoder 124 in the system 100 shown.

[0094] The video decoder 300 can be configured to perform any or all of the techniques of the present disclosure. In FIG. 3In examples of the video decoder 300, the video decoder 300 includes a number of functional components. The techniques described in this disclosure can be shared among the various components of the video decoder 300. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0095] In FIG. 3 In examples of the video decoder 300, the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transformation unit 305, and a reconstruction unit 306 and buffer 307. In some examples, the video decoder 300 can perform a decoding process generally reciprocal to the encoding process described with respect to the video encoder 200.

[0096] The entropy decoding unit 301 can retrieve an encoded bitstream. The encoded bitstream can include entropy encoded video data (e.g., encoded blocks of video data). The entropy decoding unit 301 can decode the entropy encoded video data and the motion compensation unit 302 can determine motion information including motion vectors, motion vector precision, reference picture list index, and other motion information from the entropy decoded video data. The motion compensation unit 302 can determine such information, for example, by performing AMVP and Merge modes. AMVP is used including deriving a number of most probable candidates based on data from neighboring PBs and reference pictures. The motion information typically includes horizontal and vertical motion vector displacement values, one or two reference picture indices, and in the case of prediction regions in B slices, an identification of which reference picture list is associated with each index. As used herein, in some aspects, “Merge mode” can refer to deriving motion information from spatially or temporally neighboring blocks.

[0097] The motion compensation unit 302 can generate a motion compensated block, possibly performing interpolation based on an interpolation filter. An identifier of an interpolation filter used at sub-pixel precision can be included in the syntax elements.

[0098] The motion compensation unit 302 can use the interpolation filter used by the video encoder 200 during encoding of the video block to calculate interpolated values for sub-integer pixels of the reference block. The motion compensation unit 302 can determine the interpolation filter used by the video encoder 200 from the received syntax information, and the motion compensation unit 302 can use the interpolation filter to generate the prediction block.

[0099] Motion compensation unit 302 can use at least some of the syntax information to determine the size of the blocks of the frame(s) and / or slice(s) used to encode the coded video sequence, partitioning information describing how each macroblock of a picture of the coded video sequence is partitioned, modes indicating how each partition is encoded, one or more reference frames (and lists of reference frames) for each inter-coded block, and other information to decode the coded video sequence. As used herein, in some aspects, a "slice" can refer to a data structure that can be decoded independently of other slices of the same picture in terms of entropy coding, signal prediction, and residual signal reconstruction. A slice can be an entire picture, or can also be a region of a picture.

[0100] Intra prediction unit 303 can use, for example, intra prediction modes received in the bitstream to form a prediction block from spatial neighboring blocks. Dequantization unit 304 dequantizes, i.e., inverse quantizes, quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 301. Inverse transform unit 305 applies an inverse transform.

[0101] Reconstruction unit 306 can obtain a decoded block, for example, by adding the residual block to the corresponding prediction block generated by motion compensation unit 302 or intra prediction unit 303. If desired, a deblocking filter can also be used to filter the decoded block in order to remove blockiness artifacts. The decoded video blocks are then stored in buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction, and which also produces decoded video for presentation on a display device.

[0102] Some example embodiments of the present disclosure will be described in detail below. It should be understood that the use of section headings in this document is for ease of understanding only and is not to be construed as limiting the embodiments disclosed in that section to only the section. Furthermore, although some embodiments are described with reference to the versatile video coding or other specific video codec, the disclosed techniques are applicable to other video coding technologies as well. Moreover, although some embodiments describe video encoding steps in detail, it should be understood that the corresponding decoding steps would be implemented by a decoder. Furthermore, the term video processing includes video encoding or compression, video decoding or decompression, and video transcoding, in which video pixels are represented from one compression format to another or at different compression bit rates. 1. BRIEF OVERVIEW The present disclosure relates to video coding techniques. In particular, it relates to template matching for inter prediction, and how to apply it with other coding tools in image / video coding. It can be applied to existing video coding standards such as HEVC or Versatile Video Coding (VVC). It can also be applicable to future video coding standards or video codecs. 2. Introduction Video coding standards have evolved mainly through the development of the well-known ITU-T and ISO / IEC standards. The ITU-T produced H.261 and H.263 standards, ISO / IEC produced MPEG-1 and MPEG-4 Visual, and the two organizations jointly produced the H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Coding (AVC) standards and H.265 / HEVC standard. Starting from H.262, the video coding standards are based on the hybrid video coding structure, where temporal prediction is combined with transform coding in the spatial domain. To explore future video coding technologies beyond HEVC, the Joint Video Exploration Team (JVET) was founded by VCEG and MPEG in 2015. Since then, the JVET has adopted many new methods and incorporated them into the reference software named Joint Exploration Model (JEM). In April 2018, the Joint Video Team (JVT) between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) was formed to work on the VVC standard with the goal of a 50% bitrate reduction compared to HEVC. 2.1. Coding process of a typical video codec FIG. 4 An example of the encoder block diagram of VVC is shown, which contains three in-loop filters: Deblocking Filter (DF), Sample Adaptive Offset (SAO), and ALF. Unlike DF, which uses a pre-defined filter, SAO and ALF utilize the original samples of the current picture, by adding an offset and by applying a Finite Impulse Response (FIR) filter, respectively, and utilize the coded side information to signal the offset and filter coefficients to reduce the mean square error between the original and the reconstructed samples. ALF is located at the last processing stage of each picture and can be considered as a tool that tries to capture and fix artifacts caused by the previous stages. 2.2. Intra mode coding with 67 intra prediction modes To capture arbitrary edge directions that are present in natural videos, the number of directional intra modes was extended from 33 used in HEVC to 65, as shown in FIG. 5 The planar and DC modes remain unchanged. These denser directional intra prediction modes are applied to all block sizes and for both luma intra prediction and chroma intra prediction. In HEVC, each intra-coded block has a square shape and each side of it has a length that is a power of 2. Therefore, no partitioning operation is needed to generate an intra prediction value using the DC mode. In VVC, a block can have a rectangular shape, which in general case requires a partitioning operation to be used for each block. To avoid the partitioning operation for DC prediction, only the longer side is used to calculate the average value for non-square blocks. 2.2.1. Wide-angle intra prediction Although 67 modes are defined in VVC, the exact prediction direction for a given intra prediction mode index also depends on the block shape. The regular angular intra prediction directions are defined as clockwise directions from 45 degrees to -135 degrees. In VVC, for non-square blocks, several regular angular intra prediction modes are adaptively replaced by wide-angle intra prediction modes. The replaced modes are signaled using the original mode index, which is remapped to the index of the wide-angle mode after parsing. The total number of intra prediction modes remains unchanged, i.e., 67, and the intra mode coding method remains unchanged. To support these prediction directions, a top reference of length 2W+1 and a left reference of length 2H+1 are defined, as shown in FIG. 6A and FIG. 6B The number of replaced modes in the wide-angle direction modes depends on the aspect ratio of the block. The replaced intra prediction modes are shown in Table 2-1. Table 2-1 - Intra prediction modes replaced by wide-angle modes As shown in FIG. 7 , in the case of wide-angle intra prediction, two vertically neighboring predicted samples can use two non-adjacent reference samples. Therefore, a low-pass reference sample filter and edge smoothing are applied to wide-angle prediction to reduce the negative impact of the increased gap α If the wide-angle mode represents a non-fractional offset. There are 8 modes in the wide-angle modes that satisfy this condition, the 8 modes are [-14, -12, -10, -6, 72, 76, 78, 80]. When a block is predicted by these modes, the samples in the reference buffer are directly copied without applying any interpolation. With this modification, the number of samples that need to be smoothed is reduced. In addition, it aligns the design of non-fractional modes in regular prediction modes with the wide-angle modes. ​In VVC, 4:2:2 and 4:4:4 chroma formats are supported in addition to 4:2:0. The chroma derivation mode (DM) derivation table for 4:2:2 chroma format is initially ported from HEVC, extending the number of entries from 35 to 67 to align with the extension of intra prediction modes. Since HEVC specification does not support prediction angles lower than -135 degrees and higher than 45 degrees, the range of luma intra prediction modes from 2 to 5 is mapped to 2. Therefore, the chroma DM derivation table for 4:2:2 chroma format is updated by replacing some values of the entries of the mapping table to convert the prediction angles more accurately for chroma blocks. 2.3. Inter prediction For each inter predicted CU, the motion parameters consist of the motion vector, the reference picture index and the reference picture list usage index, and additional information required for the inter predicted sample generation using the new coding features of VVC. The motion parameters can be signaled in an explicit or implicit manner. When a CU is coded in skip mode, the CU is associated with one PU and does not have significant residual coefficients, no coded motion vector delta or reference picture index. Merge mode is specified, where the motion parameters for the current CU are obtained from neighboring CUs, including spatial candidates and temporal candidates, and additional scheduling introduced in VVC. Merge mode can be applied to any inter predicted CU, not only for skip mode. An alternative to merge mode is the explicit transmission of motion parameters, where the motion vector for each reference picture list, the corresponding reference picture index and the reference picture list usage flag, and other required information are explicitly signaled for each CU. 2.4. Intra block copy (IBC) Intra block copy (IBC) is a tool adopted in the HEVC extension on SCC. It is well known that it significantly improves the coding efficiency of screen content material. Since IBC mode is implemented as a block-level coding mode, block matching (BM) is performed at the encoder to find the best block vector (or motion vector) for each CU. Here, the block vector is used to indicate the displacement from the current block to a reference block that has already been reconstructed inside the current picture. The luma block vector of an IBC coded CU is in integer precision. The chroma block vector is also rounded to integer precision. When combined with AMVR, the IBC mode can switch between 1-pixel motion vector precision and 4-pixel motion vector precision. An IBC coded CU is considered as a third prediction mode different from intra or inter prediction modes. IBC mode is applicable to CUs with width and height smaller than or equal to 64 luma samples. At the encoder side, hash-based motion estimation is performed for IBC. The encoder performs RD check for blocks whose width or height is not larger than 16 luma samples. For non-Merge mode, block vector search is first performed using hash-based search. If hash search does not return a valid candidate, block matching based local search will be performed. In hash-based search, the hash key match (32-bit CRC) between the current block and the reference block is extended to all allowed block sizes. The hash key computation is based on 4x4 sub-blocks for each location in the current picture. For a current block of larger size, the hash key is determined to match the hash key of a reference block when all hash keys of all 4x4 sub-blocks match the hash keys in the corresponding reference location. If hash keys of multiple reference blocks are found to match the hash key of the current block, the block vector cost of each matching reference is computed and the one with the smallest cost is selected. In block matching search, the search range is set to cover both the previous CTU and the current CTU. At CU level, IBC mode is signaled with a flag and it can be signaled as IBC AMVP mode or IBC Skip / Merge mode as follows: - IBC Skip / Merge mode: Merge candidate index is used to indicate which block vector from the list of IBC coded blocks of neighboring candidates is used to predict the current block. The Merge list consists of spatial candidates, HMVP candidates and pairwise candidates. - IBC AMVP mode: Block vector difference is coded in the same way as motion vector difference. Block vector prediction method uses two candidates as the predictor, one from left neighbor and one from above neighbor if IBC coded. When either neighbor is not available, the default block vector will be used as the predictor. A flag is signaled to indicate the block vector predictor index. 2.5. Merge mode with MVD (MMVD) In addition to Merge mode where the implicitly derived motion information is directly used for prediction sample generation of the current CU, Merge mode with motion vector difference (MMVD) is introduced in VVC. The MMVD flag is signaled right after the regular Merge flag to specify whether MMVD mode is used for the CU. In MMVD, after the selection of the Merge candidate, the Merge candidate. Further information includes a Merge candidate flag, an index specifying the magnitude of the motion, and an index indicating the direction of the motion. In MMVD mode, one of the first two candidates in the Merge list is selected to be used as the MV basis. An MMVD candidate flag is signaled to specify which of the first Merge candidate and the second Merge candidate is used. The distance index specifies the magnitude information and indicates a predefined offset from the starting point. As shown in FIG. 8A and FIG. 8B The offset is added to the horizontal component or the vertical component of the starting MV. The relationship of the distance index to the predefined offset is specified in Table 2-2. Table 2-2 - Relationship of distance index to predefined offset The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions as shown in Table 2-3. It is noted that the meaning of the MVD sign can change depending on the information of the starting MV. When the starting MV is a uni-predicted MV or a bi-predicted MV and both lists point to the same side of the current picture (i.e., both reference POCs are greater than the POC of the current picture or both are less than the POC of the current picture), the sign in Table 2-3 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bi-predicted MV and the two MVs point to different sides of the current picture (i.e., one reference POC is greater than the POC of the current picture and the other reference POC is less than the POC of the current picture), and the difference of the POC in list 0 is greater than the difference of the POC in list 1, the sign in Table 2-3 specifies the sign of the MV offset added to the list 0 MV component of the starting MV and the sign of the list 1 MV has the opposite value. Otherwise, if the difference of the POC in list 1 is greater than the difference of the POC in list 0, the sign in Table 2-3 specifies the sign of the MV offset added to the list 1 MV component of the starting MV and the sign of the list 0 MV has the opposite value. The MVD is scaled according to the difference of the POC in each direction. If the difference of the POC in both lists is the same, no scaling is needed. Otherwise, if the difference of the POC in list 0 is greater than the difference of the POC in list 1, the MVD is scaled by dividing the MVD by the difference of the POC in list 0 and multiplying the result by the difference of the POC in list 1. Otherwise, the MVD is scaled by dividing the MVD by the difference of the POC in list 1 and multiplying the result by the difference of the POC in list 0. the sign of the list 0 MV has the opposite value. Otherwise, if the difference of the POC in list 1 is greater than the difference of the POC in list 0, the sign in Table 2-3 specifies the sign of the MV offset added to the list 1 MV component of the starting MV and the sign of the list 0 MV has the opposite value. The MVD is scaled according to the difference of the POC in each direction. If the difference of the POC in both lists is the same, no scaling is needed. Otherwise, if the difference of the POC in list 0 is greater than the difference of the POC in list 1, the MVD is scaled by dividing the MVD by the difference of the POC in list 0 and multiplying the result by the difference of the POC in list 1. Otherwise, the MVD is scaled by dividing the MVD by the difference of the POC in list 1 and multiplying the result by the difference of the POC in list 0. L0's POC difference is defined as td, L1's POC difference is defined as tb, the MVD of list 1 is scaled as shown in FIG. 26 If L1's POC difference is greater than L0's POC difference, the MVD of list 0 is scaled in the same way. If the starting MV is uni-predicted, the MVD is added to the available MV. Table 2-3 - Sign of MV offset specified by direction index 2.6. Symmetric MVD coding In VVC, in addition to normal uni-prediction and bi-prediction mode MVD signaling, a symmetric MVD mode for bi-prediction MVD signaling is applied. In symmetric MVD mode, the motion information including the reference picture indices of both List 0 and List 1 and the MVD of List 1 is not signaled but derived. The decoding process of symmetric MVD mode is as follows: 1. At slice level, the variables BiDirPredFlag, RefIdxSymL0 and RefIdxSymL1 are derived as follows: - If mvd_l1_zero_flag is equal to 1, BiDirPredFlag is set equal to 0. - Otherwise, if the closest reference picture in List 0 and the closest reference picture in List 1 form a pair of forward and backward reference pictures or a pair of backward and forward reference pictures, BiDirPredFlag is set to 1 and both List 0 and List 1 reference pictures are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. 2. At CU level, if the CU is bi-prediction coded and BiDirPredFlag is equal to 1, a symmetric mode flag is explicitly signaled to indicate whether the symmetric mode is used or not. When the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag and MVD0 are explicitly signaled. The reference indices of List 0 and List 1 are set equal to the pair of reference pictures, respectively. MVD1 is set equal to (-MVD0). The final motion vector is as shown in the following equation. FIG. 9 A diagram for symmetric MVD mode is shown. In the encoder, symmetric MVD motion estimation starts from the initial MV evaluation. A set of initial MV candidates, including the MVs obtained from uni-prediction search, the MVs obtained from bi-prediction search and the MVs from AMVP list. One MV with the lowest rate-distortion cost is selected as the initial MV for symmetric MVD motion search. 2.7. Bi-directional optical flow (BDOF) The bi-directional optical flow (BDOF) tool is included in VVC. BDOF, previously known as BIO, is included in JEM. The BDOF in VVC is a simpler version compared to the JEM version, which requires much less computation, especially in terms of the number of multiplications and the multiplier size. BDOF is used to refine the bi-prediction signal of a CU at the 4x4 subblock level. BDOF is applied to a CU if the CU satisfies all the following conditions: - The CU is coded using the "true" bi-prediction mode, i.e., one of the two reference pictures is before the current picture in display order and the other is after the current picture in display order. - The distance from the two reference pictures to the current picture (i.e., the POC difference) is the same. - Both reference pictures are short-term reference pictures. - The CU is not coded using the affine mode or the SbTMVP Merge mode. - The CU has more than 64 luma samples. - Both the CU height and the CU width are greater than or equal to 8 luma samples. - The BCW weight index indicates equal weights. - WP is not enabled for the current CU. - CIIP mode is not used for the current CU. BDOF is only applied to the luma component. As the name indicates, the BDOF mode is based on the concept of optical flow, which assumes that the motion of an object is smooth. For each 4x4 subblock, a motion refinement (v x ,v y ) is computed by minimizing the difference between the L0 predicted samples and the L1 predicted samples. The motion refinement is then used to adjust the bi-predicted sample values in the 4x4 subblock. The following steps are applied in the BDOF process. First, the horizontal and vertical gradients of the two prediction signals and k=0,1 are computed by directly computing the difference between two neighboring samples, i.e., where I (k) (i,j) is the sample value at coordinate (i,j) of the prediction signal in list k, k=0,1, and shift1 is computed based on the luma bit depth bitDepth as shift1=max(6,bitDepth-6). Then, the auto- and cross-correlations of the gradients S1, S2, S3, S5 and S6 are computed as S1=∑ (i,j)∈Ω Abs(ψx (i,j)), S3 =∑ (i,j)∈Ω θ(i,j) · Sign(ψ x (i,j)) (2-3) S5 =∑ (i,j)∈Ω Abs(ψ y (i,j)), S6 =∑ (i,j)∈Ω θ(i,j) · Sign(ψ y (i,j)) where θ(i,j) = (I (1) (i,j) » n b ) - (I (0) (i,j) » n b ) where Ω is a 6x6 window around the 4x4 subblock, and n a and n b are set equal to min(l, bitDepth - 11) and min(4, bitDepth - 8), respectively. Motion refinement (v x , v v ) is then derived using the following formula by the cross-correlation term and the auto-correlation term: where th' BIO = 2 max(5,BD-7) . is a floor function, and Based on the motion refinement and the gradient, the following adjustment is computed for each sample in the 4x4 subblock: Finally, the BDOF samples of the CU are computed by adjusting the bi-predicted samples as follows: pred BDOF (x,y) = (I (0) (x,y) + I (1) (x,y) + b(x,y) + o offset ) » shift (2-7) These values are chosen such that the multipliers in the BDOF process do not exceed 15 bits, and the maximum bit-width of the intermediate parameters in the BDOF process is kept within 32 bits. To derive the gradient values, some prediction samples i in the list k (k = 0, 1) outside the current CU boundary need to be generated (l) (i,j). As shown in FIG. 10 VVC, one extended row / column around the CU boundary is used. To control the computational complexity of generating the out-of-boundary prediction samples, the prediction samples in the extended area (white positions) are generated by directly fetching the reference samples at the nearby integer positions without interpolation (using floor() operation on the coordinates), and the normal 8-tap motion compensation interpolation filter is used to generate the prediction samples inside the CU (gray positions). These extended sample values are only used in the gradient calculation. For the remaining steps in the BDOF process, if any sample and gradient values outside the CU boundary are needed, the sample and gradient values are padded (i.e., repeated) from their nearest neighbors. When the width and / or height of a CU is larger than 16 luma samples, it will be divided into sub-blocks with width and / or height equal to 16 luma samples, and the sub-block boundaries are treated as the CU boundaries in the BDOF process. The maximum unit size for the BDOF process is limited to 16x16. For each sub-block, the BDOF process can be skipped. When the SAD between the initial L0 prediction samples and the L1 prediction samples is smaller than a threshold, the BDOF process is not applied to the sub-block. The threshold is set to equal to (8*W*(H>>1), where W indicates the sub-block width and H indicates the sub-block height. To avoid the additional complexity of SAD calculation, the SAD between the initial L0 prediction samples and the L1 prediction samples calculated in the DVMR process is reused here. If BCW is enabled for the current block, i.e., the BCW weight index indicates unequal weights, bi-directional optical flow is disabled. Similarly, if WP is enabled for the current block, i.e., luma_weight_lx_flag is one for either of the two reference pictures, BDOF is also disabled. BDOF is also disabled when the CU is coded with symmetric MVD mode or CIIP mode. 2.8. Intra-Inter Joint Prediction (CIIP) In VVC, when a CU is coded with Merge mode, an additional flag is signaled to indicate whether the Intra-Inter Joint Prediction (CIIP) mode is applied to the current CU if the CU contains at least 64 luma samples (i.e., the CU width times the CU height is equal to or larger than 64), and if both the CU width and the CU height are smaller than 128 luma samples. As the name indicates, CIIP prediction combines an inter prediction signal with an intra prediction signal. The inter prediction signal P inter is derived using the same inter prediction process as the regular Merge mode; and the intra prediction signal P intraThe regular intra prediction process with the planar mode is followed. Then, the intra prediction signal and the inter prediction signal are combined using a weighted average, where the weight value depends on the coding modes of the top and left neighboring blocks (as shown in FIG. 11 The coding modes of the top and left neighboring blocks are calculated as follows: - If the top neighbor is available and is intra coded, set isIntraTop to 1, otherwise set isIntraTop to 0; - If the left neighbor is available and is intra coded, set isIntraLeft to 1, otherwise set isIntraLeft to 0; - If (isIntraLeft + isIntraTop) is equal to 2, set wt to 3; - Otherwise, if (isIntraLeft + isIntraTop) is equal to 1, set wt to 2; - Otherwise, set wt to 1. The CIIP prediction is formed as follows: P CIIP = ((4 - wt) * P inter + wt * P intra + 2) » 2 (2-8). 2.9. Affine motion compensated prediction In HEVC, only translational motion model is applied for motion compensated prediction (MCP). In the real world, there are many kinds of motions, such as zooming-in / zooming-out, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transform motion compensated prediction is applied. As shown in FIG. 12 ((a) 4-parameter affine model, (b) 6-parameter affine model), the affine motion field of a block is described by the motion information of two control points (4-parameter) or three control point motion vectors (6-parameter). For the 4-parameter affine motion model, the motion vector at sample position (x, y) in the block is derived as: For the 6-parameter affine motion model, the motion vector at sample position (x, y) in the block is derived as: where (mv 0x , mv 0y ) is the motion vector of the top-left control point, (mv 1x , mv 1y ) is the motion vector of the top-right control point, and (mv 2x , mv 2y ) is the motion vector of the bottom-left control point. To simplify motion-compensated prediction, block-based affine transform prediction is applied. To derive the motion vector of each 4x4 luma sub-block, the motion vector of the center sample of each sub-block is calculated according to the above equation (as shown in FIG. 13 Figure 1), and rounded to 1 / 16 fractional precision. Then a motion-compensated interpolation filter is applied to generate the prediction of each sub-block with the derived motion vector. The sub-block size for chroma components is also set to 4x4. The MV of a 4x4 chroma sub-block is calculated as the average of the MVs of the four corresponding 4x4 luma sub-blocks. As with translational inter prediction, there are also two affine inter prediction modes: affine Merge mode and affine AMVP mode. 2.9.1. Affine Merge Prediction The AF M ERGE mode can be applied to a CU whose width and height are both greater than or equal to 8. In this mode, the CPMVs of the current CU are generated based on the motion information of spatial neighboring CUs. There can be up to five CPMVP candidates, and one of them is indicated by signaling an index to be used for the current CU. The following three types of CPMV candidates are used to form the affine Merge candidate list: - Inherited affine Merge candidates inferred from the CPMVs of neighboring CUs. - Constructed affine Merge candidate CPMVPs derived using the translational MVs of neighboring CUs. - Zero MVs. In VVC, there are at most two inherited affine candidates, which are derived from the affine motion model of neighboring blocks, one from the left neighboring CU and one from the top neighboring CU. The candidate blocks are shown in FIG. 14 Figure 2. For the left prediction, the scan order is A0->A1, and for the top prediction, the scan order is B0->B1->B2. Only the first inherited candidate from each side is selected. No de-duplication check is performed between the two inherited candidates. When a neighboring affine CU is identified, its control point motion vectors are used to derive the CPMVP candidates in the affine Merge list of the current CU. As shown in FIG. 15 Figure 3, if a neighboring bottom-left block A is coded in affine mode, the motion vectors v2, v3, and v4 of the top-left, top-right, and bottom-left corners of the CU containing block A are obtained. When block A is coded with a 4-parameter affine model, two CPMVs of the current CU are calculated from v2 and v3. When block A is coded with a 6-parameter affine model, three CPMVs of the current CU are calculated from v2, v3, and v4. ​A constructed affine candidate means that the candidate is constructed by combining the neighboring translational motion information of each control point. The motion information for a control point is derived from the specified spatial and temporal neighbors as shown in Table 1. FIG. 16 CPMV k (k = 1, 2, 3, 4) denotes the k-th control point. For CPMV1, the B2->B3->A2 block is checked and the MV of the first available block is used. For CPMV2, the Bl->B0 block is checked and for CPMV3, the Al->A0 block is checked. If available, TMVP is used as CPMV4. After obtaining the MVs of the four control points, the affine Merge candidate is constructed based on the motion information. The following combinations of control point MVs are used to construct in order: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}. The combinations of 3 CPMVs construct a 6-parameter affine Merge candidate and the combinations of 2 CPMVs construct a 4-parameter affine Merge candidate. To avoid the motion scaling process, if the reference indices of the control points are different, the related combination of control point MVs is discarded. After the inherited affine Merge candidates and the constructed affine Merge candidates are checked, if the list is still not full, a zero MV is inserted at the end of the list. 2.9.2. Affine AMVP prediction The affine AMVP mode can be applied to a CU whose width and height are both greater than or equal to 16. An affine flag at CU level is signaled in the bitstream to indicate whether the affine AMVP mode is used or not, and then another flag is signaled to indicate whether it is 4-parameter affine or 6-parameter affine. In this mode, the difference between the CPMVs of the current CU and their prediction values CPMVPs are signaled in the bitstream. The affine AMVP candidate list size is 2 and is generated by using the following four types of CPVM candidates in order: - Inherited affine AMVP candidate inferred from the CPMVs of neighboring CUs. - Constructed affine AMVP candidate CPMVP derived using the translational MVs of neighboring CUs. - Zero MV. - Zero MV. The check order of inherited affine AMVP candidates is the same as the check order of inherited affine Merge candidates. The only difference is that for AMVP candidates, only affine CUs with the same reference picture as in the current block are considered. When inserting inherited affine motion predictor values into the candidate list, no de-duplication process is applied. Constructed AMVP candidates are derived from FIG. 16 the prescribed spatial neighbors as shown. The same check order as in affine Merge candidate construction is used. In addition, the reference picture index of the neighboring blocks is also checked. The first block in the check order that is inter coded and has the same reference picture as in the current CU is used. There is only one. When the current CU is coded with 4-parameter affine mode, and both mvO and mv1 are available, they are added as one candidate in the affine AMVP list. When the current CU is coded with 6-parameter affine mode, and all three CPMVs are available, they are added as one candidate in the affine AMVP list. Otherwise, the constructed AMVP candidate is set to unavailable. If the affine AMVP list candidates are still less than 2 after the inherited affine AMVP candidates and constructed AMVP candidates are checked, mvO, mv1 and mv2, if available, will be added in order as translational MVs to predict all control point MVs of the current CU. Finally, if the affine AMVP list is still not full, a zero MV is used to fill the affine AMVP list. 2.9.3. Affine motion information storage In VVC, the CPMVs of an affine CU are stored in a separate buffer. The stored CPMVs are only used to generate inherited CPMVPs in affine Merge mode and affine AMVP mode for the most recently coded CU. The sub-block MVs derived from the CPMVs are used for motion compensation, MV derivation for the Merge / AMVP list of translational MVs and deblocking. To avoid picture row buffering for additional CPMVs, the affine motion data inheritance from a CU above is treated differently from the inheritance from regular neighboring CUs. If the candidate CU for affine motion data inheritance is in the above CTU row, the bottom-left and bottom-right sub-block MVs in the row buffer, not the CPMVs, are used for affine MVP derivation. In this way, the CPMVs are only stored in the local buffer. If the candidate CU is 6-parameter affine coded, the affine model is downgraded to the 4-parameter model. As FIG. 17 shown, along the top CTU boundary, the bottom-left and bottom-right sub-block motion vectors of a CU are used for affine inheritance of the CU in the bottom CTU. 2.9.4. Prediction refinement with optical flow for affine mode Compared with pixel-based motion compensation, sub-block based affine motion compensation can save memory access bandwidth and reduce computational complexity at the cost of prediction accuracy loss. To achieve more fine-grained motion compensation, prediction refinement using optical flow (PROF) is used to refine the sub-block based affine motion compensation prediction without increasing the memory access bandwidth for motion compensation. In VVC, after the sub-block based affine motion compensation is performed, the luma prediction samples are refined by adding the difference derived by the optical flow equation. PROF is described as the following four steps: Step 1) Sub-block based affine motion compensation is performed to generate the sub-block prediction I(i,j). Step 2) The spatial gradient g x (i,j) of the sub-block prediction is calculated at each sample position using a 3-tap filter [-1, 0, 1]. y The gradient calculation is exactly the same as in BDOF. Step 3) The luma prediction refinement is calculated by the following optical flow equation. x (i,j) = (I(i+1,j) » shift1) - (I(i-1,j) » shift1) (2-11) (i,j) = (I(i,j+1) » shift1) - (I(i,j-1) » shift1) (2-12) Step 4) The refined luma prediction I'(i,j) is calculated as follows. y shift1 is used to control the precision of the gradient. Each side of the sub-block (i.e., 4x4) prediction is extended by one sample for gradient calculation. To avoid additional memory bandwidth and additional interpolation calculation, those extended samples on the extended boundaries are copied from the nearest integer pixel positions in the reference picture. Step 3) The luma prediction refinement is calculated by the following optical flow equation. ΔI(i,j) = g x (i,j) * Δv x (i,j) + g y (i,j) * Δv y (i,j) (2-13) where Δv(i,j) is the difference between the sample MV (denoted by v(i,j)) and the sub-block MV of the sub-block that the sample position (i,j) belongs to, as shown in FIG. 18 Δv(i,j) is quantized in 1 / 32 luma sample precision. Since the affine model parameters and the sample position relative to the center of the sub-block do not change from sub-block to sub-block, Δv(i,j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. Let dx(i,j) and dy(i,j) be the horizontal and vertical displacement from the sample position (i,j) to the center (x SB ,ySB The horizontal and vertical offsets, Δv(x, y), can be derived by the following equation: To maintain accuracy, the center of the sub-block (x SB ,y SB ) is calculated as ((W SB -1) / 2,(H SB -1) / 2), where W SB and H SB are the width and height of the sub-block, respectively. For the 4-parameter affine model, For the 6-parameter affine model, where (v 0x ,v 0y ), (v 1x ,v 1y ), (v 2x ,v 2y ) are the control point motion vectors at the top-left, top-right and bottom-left, and w and h are the width and height of the CU. Step 4) Finally, the luma prediction refinement ΔΙ(ί, j) is added to the sub-block prediction I(i, j). The final prediction I' is generated as the following equation. I'(i, j) = I(i, j) + ΔΙ(ί, j) (2-18) For the affine coded CU, PROF is not applied in two cases: 1) all control point MVs are the same, which indicates that the CU has only translational motion; 2) the affine motion parameters are larger than the specified limit, because the sub-block based affine MC is downgraded to CU based MC to avoid large memory access bandwidth requirement. A fast encoding method is applied to reduce the encoding complexity of affine motion estimation with PROF. In the following two cases, PROF is not applied in the affine motion estimation stage: a) if the CU is not the root block and its parent block does not select affine mode as its best mode, then PROF is not applied because the probability that the current CU selects affine mode as the best mode is low; b) if the amplitudes of the four affine parameters (C, D, E, F) are all smaller than a pre-defined threshold, and the current picture is not a low-delay picture, then PROF is not applied because the improvement introduced by PROF is small in this case. In this way, the affine motion estimation with PROF can be accelerated. 2.10. Sub-block based temporal motion vector prediction (SbTMVP) VVC supports a sub-block based temporal motion vector prediction (SbTMVP) method. Similar to the temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the collocated picture to improve the motion vector prediction and Merge mode for CUs in the current picture. The same collocated picture used by TMVP is used for SbTVMP. SbTMVP differs from TMVP in the following two main aspects: TMVP predicts the motion at the CU level, but SbTMVP predicts the motion at the sub-CU level; While TMVP obtains the temporal motion vector from a collocated block in the collocated picture (the collocated block is the right-bottom or center block relative to the current CU), SbTMVP applies a motion displacement before obtaining the temporal motion information from the collocated picture, where the motion displacement is obtained from the motion vector of one of the spatial neighboring blocks from the current CU. The SbTVMP process is shown as FIG. 19A and FIG. 19B The SbTMVP predicts the motion vector for the sub-CUs within the current CU in two steps. In the first step, the spatial near-neighbor A1 in FIG. 19A is examined. If A1 has a motion vector that uses the collocated picture as its reference picture, then that motion vector is selected as the motion displacement to be applied. If no such motion is identified, then the motion displacement is set to (0, 0). In the second step, the motion displacement identified in step 1 is applied (i.e., added to the coordinates of the current block) to obtain the motion information (motion vector and reference index) at the sub-CU level from the collocated picture as shown in FIG. 19B FIG. 19B The example inassumes that the motion displacement is set to the motion of block A1. Then, for each sub-CU, the motion information of its corresponding block in the collocated picture (covering the smallest motion grid of the center sample) is used to derive the motion information for the sub-CU. After the motion information of the collocated sub-CU is identified, it is converted to the motion vector and reference index for the current sub-CU in a similar way as the TMVP process of HEVC, where the temporal motion scaling is applied to align the reference picture of the temporal motion vector with the reference picture of the current CU. FIG. 19B The derivation of the sub-CU motion field by applying the motion displacement from the spatial near-neighbor and scaling the motion information from the corresponding collocated sub-CU is shown. In VVC, the combination of the combined subblock-based merge list containing both SbTMVP candidate and affine merge candidate is used for the signaling of the subblock-based merge mode. SbTMVP mode is enabled / disabled by a sequence parameter set (SPS) flag. If SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry of the list of subblock-based merge candidates, followed by the affine merge candidate. The size of the subblock-based merge list is signaled in the SPS, and the maximum allowed size of the subblock-based merge list is 5 in VVC. The sub-CU size used in SbTMVP is fixed to 8x8, and as with the affine merge mode, SbTMVP mode is only applicable to CUs with both width and height greater than or equal to 8. The coding logic of the additional SbTMVP merge candidate is the same as that of other merge candidates, i.e., for each CU in P slice or B slice, an additional RD check is performed to decide whether to use the SbTMVP candidate. 2.11. Adaptive Motion Vector Resolution (AMVR) In HEVC, when use_integer_mv_flag in slice header is equal to 0, the motion vector difference (MVD) (between the motion vector of a CU and the predicted motion vector) is signaled in quarter luma sample units. In VVC, a CU-level adaptive motion vector resolution (AMVR) scheme is introduced. AMVR allows the MVD of a CU to be coded with different precision. Depending on the mode for the current CU (normal AMVP mode or affine AMVP mode), the MVD of the current CU can be adaptively selected as follows: - Normal AMVP mode: quarter luma sample, half luma sample, integer luma sample, or double luma sample. - Affine AMVP mode: quarter luma sample, integer luma sample, or 1 / 16 luma sample. The CU-level MVD resolution indication is conditionally signaled if the current CU has at least one non-zero MVD component. If all MVD components (i.e., both horizontal MVD and vertical MVD for reference list L0 and reference list L1) are zero, quarter luma sample MVD resolution is inferred. For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter-luma-sample MVD precision is used for the CU. If the first flag is 0, no further signaling is needed and quarter-luma-sample MVD precision is used for the current CU. Otherwise, a second flag is signaled to indicate whether half-luma-sample or other MVD precision (integer or quarter-luma-sample) is used for normal AMVP CUs. In the case of half-luma-sample, a 6-tap interpolation filter is used for the half-luma-sample positions instead of the default 8-tap interpolation filter. Otherwise, a third flag is signaled to indicate whether integer-luma-sample MVD precision or quarter-luma-sample MVD precision is used for normal AMVP CUs. In the case of affine AMVP CUs, the second flag is used to indicate whether integer-luma-sample MVD precision or 1 / 16-luma-sample MVD precision is used. To ensure that the reconstructed MVs have the expected precision (quarter-luma-sample, half-luma-sample, integer-luma-sample, or quarter-luma-sample), the motion vector predictor for a CU will be rounded to the same precision as the MVD before being added with the MVD. The motion vector predictor is rounded to zero (i.e., negative motion vector predictors are rounded to positive infinity and positive motion vector predictors are rounded to negative infinity). MVD precision or 1 / 16-luma-sample MVD precision is used. To ensure that the reconstructed MVs have the expected precision (quarter-luma-sample, half-luma-sample, integer-luma-sample, or quarter-luma-sample), the motion vector predictor for a CU will be rounded to the same precision as the MVD before being added with the MVD. The motion vector predictor is rounded to zero (i.e., negative motion vector predictors are rounded to positive infinity and positive motion vector predictors are rounded to negative infinity). The encoder uses RD checks to determine the motion vector resolution for the current CU. To avoid performing CU-level RD checks four times for each MVD resolution, in VTM11, the RD checks for MVD precisions other than quarter-luma-sample are invoked conditionally. For normal AMVP mode, the RD cost of quarter-luma-sample MVD precision and integer-luma-sample MV precision are first calculated. Then, the integer-luma-sample MV precision is only invoked if the quarter-luma-sample MVD precision is not used. For affine AMVP mode, the RD cost of integer-luma-sample MVD precision and 1 / 16-luma-sample MVD precision are first calculated. Then, the 1 / 16-luma-sample MVD precision is only invoked if the integer-luma-sample MVD precision is not used. MVD resolution performs CU-level RD checks four times, in VTM11, the RD checks for MVD precisions other than quarter-luma-sample are invoked conditionally. For normal AMVP mode, the RD cost of quarter-luma-sample MVD precision and integer-luma-sample MV precision are first calculated. Then, the integer-luma-sample MV precision is only invoked if the quarter-luma-sample MVD precision is not used. For affine AMVP mode, the RD cost of integer-luma-sample MVD precision and 1 / 16-luma-sample MVD precision are first calculated. Then, the 1 / 16-luma-sample MVD precision is only invoked if the integer-luma-sample MVD precision is not used. The RD cost of the MVD precision is compared with the RD cost of the quarter luma sample MVD precision to decide whether it is necessary to further check the RD cost of the quarter luma sample MVD precision. When the RD cost of the quarter luma sample MVD precision is much smaller than the RD cost of the integer luma sample MVD precision, the RD check of the quarter luma sample MVD precision is skipped. Then, if the RD cost of the integer luma sample MVD precision is significantly larger than the best RD cost of the previously tested MVD precisions, the check of the half luma sample MVD precision is skipped. For affine AMVP mode, if no affine inter mode is selected after checking the rate-distortion cost of affine Merge / skip mode, Merge / skip mode, quarter luma sample MVD precision normal AMVP mode and quarter luma sample MVD precision affine AMVP mode, the 1 / 16 luma sample MV precision and 1-pixel MV precision affine inter mode are not checked. In addition, in 1 / 16 luma sample and quarter luma sample MV precision affine inter mode, the affine parameters obtained in the quarter luma sample MV precision affine inter mode are used as the starting search points. 2.12. Bi-prediction with CU-level weights (BCW) In HEVC, bi-prediction signal is generated by averaging two prediction signals obtained from two different reference pictures and / or using two different motion vectors. In VVC, bi-prediction mode is extended beyond simple averaging to allow weighted averaging of two prediction signals. P bi-pred = ((8 - w) * P0 + w * P1 + 4) » 3 (2-19) Five weights, w e {-2, 3, 4, 5, 10}, are allowed in weighted average bi-prediction. For each bi-predicted CU, the weight w is determined in one of two ways: 1) for non-Merge CUs, the weight index is signaled after the motion vector difference; 2) for Merge CUs, the weight index is inferred from neighboring blocks based on the Merge candidate index. BCW is only applied to CUs with 256 or more luma samples (i.e., CU width times CU height is greater than or equal to 256). For low-delay pictures, all 5 weights are used. For non-low-delay pictures, only 3 weights (w e {3, 4, 5}) are used. At the encoder, fast search algorithms are applied to find the weight index without significantly increasing the encoder complexity. These algorithms are summarized as follows. The reader can refer to the VTM software for more details. When combined with AMVR, if the current picture is a low-delay picture, the unequal weights are only conditionally checked for 1-pixel motion vector precision and 4-pixel motion vector precision. - When combined with affine combination, affine ME will be performed for unequal weights if and only if affine mode is selected as the current best mode. - When the two reference pictures in bi-prediction are the same, unequal weights are checked conditionally only. - When certain conditions are met, unequal weights are not searched, depending on the POC distance between the current picture and its reference pictures, the coding QP and the temporal level. BCW weight index is coded using one context coded bin followed by bypass coded bins. The first context coded bin indicates whether equal weights are used; if unequal weights are used, additional bins are signaled using bypass coding to indicate which unequal weights are used. Weighted prediction (WP) is a coding tool supported by H.264 / AVC and HEVC standards for efficiently coding video content with gradual changes. Support for WP is also added to the VVC standard. WP allows the signaling of weighting parameters (weights and offsets) for each reference picture in each of the reference picture lists L0 and L1. Then, during motion compensation, the weight(s) and offset(s) of the corresponding reference picture(s) are applied. WP and BCW are designed for different types of video content. To avoid the interaction between WP and BCW that would complicate the VVC decoder design, if a CU uses WP, the BCW weight index is not signaled and w is inferred to be 4 (i.e., equal weights are applied). For Merge CUs, the weight index is inferred from neighboring blocks based on the Merge candidate index. This can be applied to both normal Merge mode and inherited affine Merge mode. For constructed affine Merge mode, the affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index for a CU using constructed affine Merge mode is simply set to be equal to the first control point BCW index of MV. In VVC, CIIP and BCW cannot be jointly applied to a CU. When a CU is coded with CIIP mode, the BCW index of the current CU is set to 2, e.g., equal weights. 2.13. Local illumination compensation (LIC) Local illumination compensation (LIC) is a coding tool to address the issue of local illumination changes between the current picture and its temporal reference picture. LIC is based on a linear model where a scaling factor and an offset are applied to the reference samples to obtain the prediction samples of the current block. Specifically, LIC can be mathematically modeled by the following equation: P(x, y) = a - p r (x + v xy + v y )+ β where P(x, y) is the prediction signal of the current block at coordinates (x, y); P r (x + v x , y + v y ) is the reference block pointed by the motion vector (v x , v y ); a and b are the corresponding scaling factor and offset applied to the reference block. FIG. 20 The LIC process is illustrated. In FIG. 20 , when LIC is applied for a block, the least mean square error (LMSE) method is employed to derive the values of the LIC parameters (i.e., a and b) by minimizing the difference between the neighboring samples of the current block (i.e., the template T in FIG. 20 ) and their corresponding reference samples in the temporal reference picture (i.e., T0 or T1 in FIG. 20 ). Additionally, to reduce the computational complexity, both the template samples and the reference template samples are down-sampled (adaptive down-sampling) to derive the LIC parameters, i.e., only the shaded samples in FIG. 20 are used to derive a and b. To improve the coding performance, as shown in FIG. 21 , no down-sampling is performed for the short side. 2.14. Decoder-side motion vector refinement (DMVR) To improve the accuracy of the MVs of the Merge mode, a decoder-side motion vector refinement based on bilateral matching (BM) is applied in VVC. In the bi-prediction operation, the initial MVs around the reference picture list L0 and the reference picture list L1 are searched for refined MVs. The BM method computes the distortion between two candidate blocks in the reference picture list L0 and list L1. As shown in FIG. 22 , the SAD between the two blocks based on each MV candidate (e.g., MV0' and MV1') around the initial MV is computed. The MV candidate with the lowest SAD becomes the refined MV and is used to generate the bi-predicted signal. In VVC, the application of DMVR is restricted and is only applied to the CUs that are coded with the following modes and features: - CU-level Merge mode with bi-predicted MVs. - One reference picture is past and the other reference picture is future relative to the current picture. - The distance (i.e., POC difference) from the two reference pictures to the current picture is the same. - Both reference pictures are short-term reference pictures. - The CU has more than 64 luma samples. - CU height and CU width are both greater than or equal to 8 luma samples. - BCW weight index indicates equal weights. - WP is not enabled for the current block. - CIIP mode is not used for the current block. The refined MV derived by the DMVR process is used to generate inter- predicted samples and is also used in temporal motion vector prediction for future picture coding. While the original MV is used in the deblocking process and is also used in spatial motion vector prediction for future CU coding. Additional features of DMVR are mentioned in the following subclause. 2.14.1. Search scheme In DVMR, the search points are around the initial MV and the MV offset respects the MV difference mirroring rule. In other words, any point (denoted by the candidate MV pair (MV0, MV1)) examined by DMVR respects the following two equations: MV0' = MV0 + MV_offset (2-20) MV1' = MV1 - MV_offset (2-21) where MV_offset denotes the refinement offset between the initial MV and the refined MV in one of the reference pictures. The refinement search range is two integer luma samples away from the initial MV. The search consists of an integer sample offset search phase and a fractional sample refinement phase. A 25-point full search is applied to the integer sample offset search. The SAD of the initial MV pair is first computed. If the SAD of the initial MV pair is less than a threshold, the integer sample phase of DMVR is terminated. Otherwise, the SAD of the remaining 24 points is computed and examined in a raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search phase. To reduce the impact of the uncertainty of DMVR refinement, a bias towards the original MV is proposed during the DMVR process. The SAD between the reference blocks referred by the initial MV candidate is reduced by 1 / 4 of the SAD value. The integer sample search is followed by the fractional sample refinement. To save computational complexity, the fractional sample refinement is derived by using the parametric error surface equation instead of by additional search with SAD comparison. The fractional sample refinement is conditionally invoked based on the output of the integer sample search phase. The fractional sample refinement is further applied when the integer sample search phase terminates with the center having the smallest SAD in the first or second iteration search. In the parametric error surface based sub-pixel offset estimation, the center position cost and the costs at four neighboring positions away from the center are used to fit a two-dimensional parabolic error surface equation of the following form E(x,y) = A(x - x min ) 2 +B(y - y min ) 2 +C (2-22) where (x min ,y min ) corresponds to the fractional position with the minimum cost, and C corresponds to the minimum cost value. By solving the above equations using the cost values of the five search points, (x min ,y min ) is computed as: x min = (E(-1,0) - E(1,0)) / (2(E(-1,0) + E(1,0) - 2E(0,0))) (2-23) y min = (E(0,-1) - E(0,1)) / (2((E(0,-1) + E(0,1) - 2E(0,0))) (2-24) x min and y min values are automatically constrained between -8 and 8, since all cost values are positive and the minimum value is E(0,0). This corresponds to the half-pixel offset with 1 / 16-pixel MV precision in VVC. The computed fraction (x min ,y min ) is added to the integer distance refined MV to get the sub-pixel precise refined delta MV. 2.14.2. Bilinear interpolation and sample padding In VVC, the resolution of MV is 1 / 16 luma sample. The samples at fractional positions are interpolated using an 8-tap interpolation filter. In DMVR, the search points are around the initial fractional pixel MV with integer sample offset, so the samples at those fractional positions need to be interpolated for the DMVR search process. To reduce the computational complexity, a bilinear interpolation filter is used to generate the fractional samples for the search process in DMVR. Another important effect of using the bilinear filter is that with a 2-sample search range, DMVR does not access more reference samples compared to the normal motion compensation process. After the refined MV is obtained through the DMVR search process, the normal 8-tap interpolation filter is applied to generate the final prediction. To not access more reference samples than the normal MC process, the samples that are not needed for the interpolation process based on the original MV but needed for the interpolation process based on the refined MV will be padded from those available samples. 2.15. Multi-pass decoder-side motion vector refinement In this contribution, multi-pass decoder-side motion vector refinement is applied instead of DMVR. In the first pass, bilateral matching (BM) is applied to the coded block. In the second pass, BM is applied to each 16x16 sub-block within the coded block. In the third pass, the MV in each 8x8 sub-block is refined by applying bi-directional optical flow (BDOF). The refined MVs are stored for both spatial and temporal motion vector prediction. 2.15.1. First pass - block-based bilateral matching MV refinement In the first pass, the refined MV is derived by applying BM to the coded block. Similar to decoder-side motion vector refinement (DMVR), the refined MV is searched around two initial MVs (MV0 and MV1) in the reference picture lists L0 and L1. The refined MVs (MV0_pass1 and MV1_pass1) are derived around the initial MVs based on the minimum bilateral matching cost between the two reference blocks in L0 and L1. The BM performs a local search to derive the integer sample precision intDeltaMV and the half-pixel sample precision halfDeltaMv. The local search applies a 3x3 square search pattern to loop through the search range [-sHor, sHor] in the horizontal direction and [-sVer, sVer] in the vertical direction, where the values of sHor and sVer are determined by the block dimensions, and the maximum values of sHor and sVer are 8. The bilateral matching cost is computed as: bilCost = mvDistanceCost + sadCost. When the block size cbW*cbH is greater than 64, the MRSAD cost function is applied to remove the DC effect of the distortion between the reference blocks. The intDeltaMV or halfDeltaMV local search is terminated when the bilCost at the center point of the 3x3 search pattern has the minimum cost. Otherwise, the current minimum cost search point becomes the new center point of the 3x3 search pattern and the search for the minimum cost continues until it reaches the end of the search range. The existing fractional sample refinement is further applied to derive the final deltaMV. The refined MV after the first pass is then derived as: • MV0_pass1 = MV0 + deltaMV, • MV1_pass1 = MV1 - deltaMV. 2.15.2. Second pass - sub-block based bilateral matching MV refinement In the second pass, the refined MV is derived by applying BM to 16x16 grid sub-blocks. For each sub-block, the refined MV is searched around the two MVs (MV0_pass1 and MV1_pass1) for the reference picture lists L0 and L1 obtained from the first pass. The refined MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the two reference sub-blocks in L0 and L1. For each sub-block, the BM performs a full search to derive the integer sample precision intDeltaMV. The full search has a search range [-sHor, sHor] in the horizontal direction and [-sVer, sVer] in the vertical direction, where the values of sHor and sVer are determined by the block dimension, and the maximum value of sHor and sVer is 8. The bilateral matching cost is computed by applying a cost factor to the SATD cost between the two reference sub-blocks, as: bilCost = satdCost * costFactor. The search area (2*sHor+1)*(2*sVer+1) is divided into FIG. 23The maximum of 5 diamond search regions are shown. Each search region is assigned a costFactor, which is determined by the distance between each search point and the starting MV (intDeltaMV), and each diamond region is processed in order starting from the center of the search region. In each region, search points are processed in a raster scan order from the top-left corner to the bottom-right corner of the region. When the minimum bilCost within the current search region is less than a threshold (which is equal to sbW*sbH), the integer-pixel full search is terminated, otherwise, the integer-pixel full search continues to the next search region until all search points are checked. The BM performs a local search to derive the half-pel precision halfDeltaMv. The search pattern and cost function are the same as defined in 2.9.1. The existing VVC DMVR fractional sample refinement is further applied to derive the final deltaMV (sbldx2). The refined MV at the second pass is then derived as: • MV0_pass2 (sbldx2) = MV0_pass1 + deltaMV (sbldx2), • MV1_pass2 (sbldx2) = MV1_pass1 - deltaMV (sbldx2). 2.15.3. Third pass - subblock-based bi-directional optical flow MV refinement In the third pass, the refined MV is derived by applying BDOF to the 8x8 grid subblocks. For each 8x8 subblock, the BDOF refinement is applied to derive the scaled Vx and Vy without clipping from the refined MV of the parent block of the second pass. The derived bioMv (Vx, Vy) is rounded to 1 / 16 sample precision and clipped between -32 and 32. The refined MV at the third pass (MV0_pass3 (sbldx3) and MV1_pass3 (sbldx3)) is derived as: • MV0_pass3 (sbldx3) = MV0_pass2 (sbldx2) + bioMv, • MV1_pass3 (sbldx3) = MV0_pass2 (sbldx2) - bioMv. 2.16. Sample-based BDOF In sample-based BDOF, instead of deriving the motion refinement (Vx, Vy) on a block basis, the motion refinement is performed for each sample. A coded block is partitioned into 8x8 sub-blocks. For each sub-block, whether to apply BDOF is determined by checking the SAD between two reference sub-blocks against a threshold. If it is decided to apply BDOF to a sub-block, for each sample in the sub-block, a sliding 5x5 window is used, and the existing BDOF process is applied for each sliding window to derive Vx and Vy. The derived motion refinements (Vx, Vy) are applied to adjust the bi-predicted sample value for the center sample of the window. 2.17. Extended Merge prediction In VVC, the Merge candidate list is constructed by including the following five types of candidates in order: (1) Spatial MVP from spatial neighboring CUs. (2) Temporal MVP from collocated CUs. (3) History-based MVP from a FIFO table. (4) Pairwise average MVP. (5) Zero MV. The size of the Merge list is signaled in the sequence parameter set header, and the maximum allowed size of the Merge list is 6. For each CU coded in Merge mode, the index of the best Merge candidate is coded using truncated unary binarization (TU). The first bin of the Merge index is coded with a context, and bypass coding is used for the other bins. The derivation process of each category of Merge candidates is provided in this section. As done in HEVC, VVC also supports parallel derivation of the Merge candidate list for all CUs within a certain size of region. 2.17.1. Spatial candidate derivation The derivation of spatial Merge candidates in VVC is the same as in HEVC, except that the positions of the first two Merge candidates are swapped. In the candidates located at FIG. 24 Up to four Merge candidates are selected among the candidates located at the positions B0, A0, B1, A1, and B2. The derivation order is B0, A0, B1, A1, and B2. Position B2 is considered only when one or more than one CU at positions B0, A0, B1, A1 is not available (e.g., because it belongs to another slice or tile) or is intra coded. After adding the candidate at position A1, a redundancy check is performed for the addition of the remaining candidates, which ensures that candidates with the same motion information are not included in the list, thereby improving coding efficiency. To reduce the computational complexity, not all possible pairs of candidates are considered in the mentioned redundancy check. Instead, only the pairs linked with arrows in FIG. 25 are considered, and only if the corresponding candidates for the redundancy check do not have the same motion information, the candidate is added to the list. 2.17.2. Temporal candidate derivation In this step, only one candidate is added to the list. Specifically, in the derivation of the temporal Merge candidate, the scaled motion vector is derived based on the collocated CU belonging to the collocated reference picture. The reference picture list to be used for the derivation of the collocated CU is explicitly signaled in the slice header. As shown by the dashed line in FIG. 26 , the scaled motion vector for the temporal Merge candidate is obtained using the POC distances tb and td from the motion vector of the collocated CU, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal Merge candidate is set equal to 0. As shown in FIG. 27 , the position for the temporal candidate is selected between the positions of the candidates C0 and C1. If the CU at position C0 is unavailable, intra coded or outside the current row of the CTU, then position C1 is used. Otherwise, position C0 is used for the derivation of the temporal Merge candidate. 2.17.3. History-based Merge candidate derivation The history-based MVP (HMVP) Merge candidate is added to the Merge list, after the spatial and TMVPs. In this method, the motion information of previously coded blocks is stored in a table and used as MVPs for the current CU. During the encoding / decoding process, a table with multiple HMVP candidates is maintained. When a new CTU row is encountered, the table is reset (emptied). As long as there is a non-subblock inter coded CU, the associated motion information is added to the last entry of the table as a new HMVP candidate. The HMVP table size S is set to 6, which indicates that up to 6 history-based MVP (HMVP) candidates can be added to the table. When a new motion candidate is inserted into the table, a constrained first-in-first-out (FIFO) rule is utilized, where a redundancy check is first applied to find if there is a same HMVP in the table. If found, the same HMVP is removed from the table and all the HMVP candidates after it are shifted forward and the same HMVP is inserted to the last entry of the table. The HMVP candidates can be used in the Merge candidate list construction process. The latest few HMVP candidates in the table are checked in order and inserted into the candidate list, after the TMVP candidates. A redundancy check is applied to the HMVP candidates for spatial or temporal Merge candidates. To reduce the number of redundancy check operations, the following simplifications are introduced: The number of HMPV candidates used for the Merge list generation is set to (N<=4)?M:(8-N), where N indicates the number of existing candidates in the Merge list and M indicates the number of available HMVP candidates in the table. The construction process from the Merge candidate list of HMVP is terminated once the total number of available Merge candidates reaches the maximum allowed Merge candidates minus 1. 2.17.4. Pairwise average Merge candidate derivation Pairwise average candidates are generated by averaging predefined pairs of candidates in the existing Merge candidate list, and the predefined pairs are defined as {(0, 1), (0, 2), (1, 2), (0, 3), (1, 3), (2, 3)}, where the numbers represent the Merge indices in the Merge candidate list. The averaged motion vector is calculated separately for each reference list. If both motion vectors are available in one list, the two motion vectors are averaged even if they point to different reference pictures; if only one motion vector is available, the motion vector is used directly; if no motion vector is available, the list is kept invalid. When the Merge list is not full after adding the pairwise average Merge candidate, a zero MVP is inserted at the end until the maximum number of Merge candidates is reached. 2.17.5. Merge estimation region Merge estimation region (MER) allows independent derivation of the Merge candidate list for CUs in the same Merge estimation region (MER). Candidate blocks that are within the same MER as the current CU are not included for the generation of the Merge candidate list of the current CU. In addition, the history-based motion vector predictor candidate list is only updated when (x Log2ParMrgLevel and (yCb + cbHeight) » Log2ParMrgLevel is greater than (yCb Log2ParMrgLevel), where (x 2.18. New Merge candidate 2.18.1. Non-adjacent Merge candidate derivation In VVC, FIG. 28The five spatial neighboring blocks and one temporal neighbor shown are used to derive Merge candidates. It is proposed to use the same modes as in VVC to derive additional Merge candidates from locations that are non-adjacent to the current block. To achieve this, for each search round i, a virtual block is generated based on the current block as follows: First, the relative position of the virtual block with respect to the current block is computed by: Offsetx = -i x gridX, Offsety = -i x gridY where Offsetx and Offsety denote the offset of the top-left corner of the virtual block with respect to the top-left corner of the current block, and gridX and gridY are the width and height of the search grid. Second, the width and height of the virtual block are computed by: newWidth = i x 2 x gridX + currWidth, newHeight = i x 2 x gridY + currHeight. where currWidth and currHeight are the width and height of the current block. newWidth and newHeight are the width and height of the new virtual block. gridX and gridY are currently set to currWidth and currHeight, respectively. FIG. 29 The relationship between the virtual block and the current block is shown. After the virtual block is generated, blocks A i , B i , C i , D i , and E i can be regarded as the VVC spatial neighboring blocks of the virtual block, and their positions are obtained using the same modes as in VVC. Obviously, if the search round i is 0, the virtual block is the current block. In this case, blocks A i , B i , C i , D i , and E i are the spatial neighboring blocks used in the VVC Merge mode. When constructing the Merge candidate list, de-duplication is performed to guarantee that each element in the Merge candidate list is unique. The maximum search round is set to 1, which means that the five non-adjacent spatial neighboring blocks are utilized. The non-adjacent spatial Merge candidates are inserted into the Merge list after the temporal Merge candidates in the order of B1->A1->C1->D1->E1. 2.18.2. STMVP It is proposed to use three spatial Merge candidates and one temporal Merge candidate to derive the average candidate as the STMVP candidate. The STMVP is inserted before the top-left spatial Merge candidate. The STMVP candidate is de-duplicated together with all previous Merge candidates in the Merge list. For spatial candidates, the first three candidates in the current Merge candidate list are used. For temporal candidates, the same position as the VTM / HEVC collocated position is used. For spatial candidates, the first, second and third candidates in the current Merge candidate list, which are inserted before the STMVP, are denoted as F, S and T. The temporal candidate with the same position as the VTM / HEVC collocated position used in TMVP is denoted as Col. The motion vector of the STMVP candidate in prediction direction X (denoted as mvLX) is derived as follows: 1) If the reference indices of the four Merge candidates are all valid and equal to 0 in prediction direction X (X = 0 or 1), mvLX = (mvLX_F + mvLX_S + mvLX_T + mvLX_Col) » 2 2) If the reference indices of three of the four Merge candidates are valid and equal to 0 in prediction direction X (X = 0 or 1), mvLX = (mvLX_F x 3 + mvLX_S x 3 + mvLX_Col x 2) » 3 or mvLX = (mvLX_F x 3 + mvLX_T x 3 + mvLX_Col x 2) » 3 or mvLX = (mvLX_S x 3 + mvLX_T x 3 + mvLX_Col x 2) » 3. 3) If the reference indices of two of the four Merge candidates are valid and equal to 0 in prediction direction X (X = 0 or 1), mvLX = (mvLX_F + mvLX_Col) » 1 or mvLX = (mvLX_S + mvLX_Col) » 1 or mvLX = (mvLX_T + mvLX_Col) » 1. NOTE: If the temporal candidate is not available, the STMVP mode is turned off. 2.18.3. Merge list size If both non-adjacent Merge candidates and STMVP Merge candidates are considered, the size of the Merge list is signaled in the sequence parameter set header and the maximum allowed size of the Merge list is 8. 2.19. Geometric partition mode (GPM) In VVC, a geometric partition mode is supported for inter prediction. The geometric partition mode is signaled using a CU-level flag as a kind of Merge mode, where other Merge modes include regular Merge mode, MMVD mode, CIIP mode, and subblock Merge mode. For each possible CU size w x h = 2 m x 2 n (where m, n e {3...6} excluding 8x64 and 64x8), the geometric partition mode supports 64 partitions in total. When this mode is used, the CU is divided into two parts by a straight line that is geometrically positioned FIG. 30 ). The position of the dividing line is mathematically derived from the angle and offset parameters of the specific partition. Each part of the geometric partition in the CU is inter predicted using its own motion; only uni-prediction is allowed for each partition, i.e., each part has one motion vector and one reference index. The uni-prediction motion constraint is applied to ensure the same as regular bi-prediction, i.e., only two motion- compensated predictions are needed for each CU. The uni-prediction motion for each partition is derived using the process described in 2.19.1. If the geometric partition mode is used for the current CU, further the geometric partition index and two Merge indices (one for each partition) are signaled to indicate the partition mode (angle and offset) of the geometric partition. The number of maximum GPM candidate size is explicitly signaled in the SPS, and the syntax binarization for GPM Merge indices is specified. After predicting each part of the geometric partition, a hybrid process with adaptive weights as in 2.19.2 is used to adjust the sample values along the geometric partition edge. This is the prediction signal for the whole CU, and the transform and quantization processes will be applied to the whole CU as in other prediction modes. Finally, the motion field of the CU predicted using the geometric partition mode is stored as in 2.19.3. 2.19.1. Uni-prediction candidate list construction The uni-prediction candidate list is directly derived from the Merge candidate list constructed according to the extended Merge prediction process in 2.17. Let n denote the index of the uni-prediction motion in the geometric uni-prediction candidate list. The LX motion vector of the nth extended Merge candidate (where X equals the parity of n) is used as the nth uni-prediction motion vector for the geometric partition mode. These motion vectors are in FIG. 31The corresponding LX motion vector of the nth extended Merge candidate is marked with "x". If the corresponding LX motion vector of the nth extended Merge candidate does not exist, the L(l-X) motion vector of the same candidate is used instead as the uni-prediction motion vector for the geometric partition mode. 2.19.2. Blending along the geometric partition edge After each part of the geometric partition is predicted using its own motion prediction, blending is applied to the two prediction signals to derive the samples around the geometric partition edge. The blending weight for each position of the CU is derived based on the distance between the single position and the partition edge. The distance for a position (x, y) to the partition edge is derived as: where i, j are the indices for the angle and offset of the geometric partition, which depend on the signaled geometric partition index. p x,j and the sign of p y,j depend on the angle index i. The weight of each part of the geometric partition is derived as follows: wldxL(x, y) = partldx? 32 + d(x, y) : 32 - d(x, y) (2-29) w1(x, y) = 1 - w0(x, y) (2-31) partldx depends on the angle index i. One example of the weight w0 is shown in FIG. 32 . 2.19.3. Motion field storage for geometric partition mode Mvl from the first part of the geometric partition, Mv2 from the second part of the geometric partition, and the combined Mv of Mvl and Mv2 are stored in the motion field of the CU coded with the geometric partition mode. The motion vector type stored for each individual position in the motion field is determined as: sType = abs(motionldx) < 32? 2 : (motionldx < 0? (1 - partldx) : partldx) (2-32) where motionldx is equal to d(4x + 2, 4y + 2), which is recalculated from equation (2-18). partldx depends on the angle index i. If sType is equal to 0 or 1, Mv0 or Mvl is stored in the corresponding motion field, otherwise, if sType is equal to 2, the combined Mv from Mv0 and Mv2 is stored. The combined Mv is generated using the following process: 1) If Mv1 and Mv2 come from different reference picture lists (one from L0 and the other from L1), then Mv1 and Mv2 are simply combined to form a bi-directional predicted motion vector. Otherwise, if Mv1 and Mv2 come from the same list, then only the uni-predicted motion Mv2 is stored. 2.20. Multi-hypothesis prediction In multi-hypothesis prediction (MHP), up to two additional prediction values are signaled on top of the inter AMVP mode, regular Merge mode, affine Merge and MMVD mode. The resulting overall prediction signal is iteratively accumulated with each additional prediction signal. p n+1 = (1 - a n+1 ) p n + a n+1 h n+1 The weighting factor a is specified according to the following Table 2-4: Table 2-4 - Weighting factor for MHP add_hyp_weight_idx α 0 1 / 4 1 -1 / 8 For the inter AMVP mode, MHP is applied only when non-equal weights in BCW are selected in bi-predictive mode. The additional hypotheses can be either Merge mode or AMVP mode. In the case of Merge mode, the motion information is indicated through Merge index and the Merge candidate list is the same as in the geometric partition mode. In the case of AMVP mode, the reference index, MVP index and MVD are signaled. 2.21. Non-adjacent spatial candidates Non-adjacent spatial Merge candidates are inserted in the regular Merge candidate list after TMVP. The pattern of spatial Merge candidates is shown in FIG. 33 . The distance between the non-adjacent spatial candidate and the current coding block is based on the width and height of the current coding block. 2.22. Template matching (TM) Template matching (TM) is a decoder-side MV derivation method to refine the motion information of a current CU by finding the closest match between a template (i.e., the top and / or left neighboring blocks of the current CU) in the current picture and a block (i.e., of the same size as the template) in the reference picture. As FIG. 34 shown in FIG. 2-2, a better MV is searched around the initial motion of the current CU within a search range of [-8, +8] pixels. There are two modifications in the template matching in this contribution: the search step size is determined based on the AMVR mode, and in Merge mode TM can be cascaded with the bilateral matching process. In AMVP mode, MVP candidates are determined based on template matching error to select one that achieves the minimum difference between the current block template and the reference block template, then TM performs MV refinement only for that particular MVP candidate. TM refines the MVP candidate by using an iterative diamond search starting from full-pel MVD precision (or 4-pixel for 4-pixel AMVR mode) within a search range of [-8, +8] pixels. AMVP candidates can be further refined by using a cross search with full-pel MVD precision (or 4-pixel for 4-pixel AMVR mode) followed by half-pel and quarter-pel in turn as specified in Table 2-5 for AMVR mode. This search process ensures that the MVP candidate still maintains the same MV precision as indicated by the AMVR mode after the TM process. Table 2-5 - Search modes for AMVR and search modes for Merge mode with AMVR In Merge mode, similar search methods are applied to the Merge candidates indicated by the Merge index. As shown in Table 2-5, TM can be performed up to 1 / 8-pel MVD precision or skip those precision beyond half-pel MVD precision depending on whether the alternative interpolation filter is used according to the Merge motion information (i.e. used when AMVR is half-pel mode). In addition, when TM mode is enabled, template matching can work as a standalone process or as an additional MV refinement process between the block-based and sub-block-based bilateral matching (BM) methods depending on whether BM can be enabled according to its enabling condition check. In template matching Merge mode, the encoder can choose from uni-prediction from list 0, uni-prediction from list 1 or bi-prediction for the CU. The choice is based on the template matching cost as follows: If costBi<=factor*min(cost0, cost1) Use bi-prediction; Else if cost0<=cost1 Use uni-prediction from list 0; Else Use uni-prediction from list 1; where cost0 is the SAD of list 0 template matching, cost1 is the SAD of list 1 template matching and costBi is the SAD of bi-prediction template matching. The value of factor is equal to 1.125, which means the selection process is biased towards bi-prediction. 2.23. Overlapped Block Motion Compensation (OBMC) Overlapped Block Motion Compensation (OBMC) has been used in H.263 previously. In JEM, unlike H.263, OBMC can be turned on and off using CU-level syntax. When OBMC is used in JEM, OBMC is performed for all Motion Compensation (MC) block boundaries except the right and bottom boundaries of a CU. Furthermore, it is applied to both luma and chroma components. In JEM, an MC block corresponds to a coding block. When a CU is coded in sub-CU mode (including sub-CU Merge, affine and FRUC modes), each sub-block of the CU is an MC block. To handle CU boundaries in a uniform way, OBMC is performed at sub-block level for all MC block boundaries, where the sub-block size is set to be equal to 4x4, as shown in FIG. 35 When OBMC is applied to a current sub-block, in addition to the current motion vector, the motion vectors of four connected neighboring sub-blocks (if available and not identical to the current motion vector) are also used to derive the prediction block for the current sub-block. These multiple prediction blocks based on multiple motion vectors are combined to generate the final prediction signal of the current sub-block. The prediction block based on the motion vector of a neighboring sub-block is denoted as P N , where N denotes the index for the neighboring top, bottom, left and right sub-blocks, and the prediction block based on the motion vector of the current sub-block is denoted as P C . When P N is based on the motion information of a neighboring sub-block containing the same motion information as the current sub-block, no OBMC is performed from P N . Otherwise, each sample of P N is added to the same sample in P C , i.e. four rows / columns of P N are added to P C . The weight factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} are used for P N , and the weight factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are used for P C . The exception is for small MC blocks (i.e. when the height or width of a coding block is equal to 4 or the CU is coded in sub-CU mode), for which only two rows / columns of P N are added to P C . In this case, the weight factors {1 / 4, 1 / 8} are used for P N , and the weight factors {3 / 4, 7 / 8} are used for P C . For P N generated based on the motion vector of a vertical (horizontal) neighboring sub-block, the samples in the same row (column) of P N are added to P C with the same weight factor. ​In JEM, a CU-level flag is signaled to indicate whether OBMC is applied to the current CU for CUs with size smaller than or equal to 256 luma samples. For CUs with size larger than 256 luma samples or not coded with AMVP mode, OBMC is applied by default. At the encoder, when OBMC is applied to a CU, its impact is considered during the motion estimation stage. The prediction signal formed by OBMC using the motion information of the top and left neighboring blocks is used to compensate the top and left boundaries of the original signal of the current CU, then the normal motion estimation process is applied. 2.24. Multiple transform selection (MTS) for core transform In addition to DCT-II already employed in HEVC, a multiple transform selection (MTS) scheme is used for residual coding of both inter and intra coded blocks. It uses multiple transforms selected from DCT8 / DST7. The newly introduced transform matrices are DST-VII and DCT-VIII. Table 2-6 shows the basis functions of the selected DST / DCT. Table 2-6 - Transform basis functions of DCT-II / VIII and DST VII for N-point input To maintain the orthogonality of the transform matrices, the transform matrices are quantized more precisely than in HEVC. To keep the mid value of the transform coefficients in the 16-bit range, all coefficients have 10 bits after the horizontal transform and after the vertical transform. To control the MTS scheme, separate enabling flags are specified at SPS level for intra and inter, respectively. When MTS is enabled at SPS, a CU-level flag is signaled to indicate whether MTS is applied. Here, MTS is applied only to luma. When one of the following conditions is applied, the MTS signaling is skipped. - The position of the last significant coefficient for luma TB is smaller than 1 (i.e., DC only). - The last significant coefficient of luma TB is located in the MTS zero-out region. If the MTS CU flag is equal to 0, DCT2 is applied in both directions. However, if the MTS CU flag is equal to 1, two other flags are additionally signaled to indicate the transform type for the horizontal and vertical directions, respectively. The transform and signaling mapping table is shown in Table 2-7. The unified transform selection for ISP and implicit MTS is used by removing the intra mode and block shape dependency. If the current block is ISP mode, or if the current block is an intra block and both the intra explicit MTS and the inter explicit MTS are turned on, only DST7 is used for both the horizontal and vertical transform kernels. When it comes to transform matrix precision, the 8-bit primary transform kernel is used. Therefore, all transform kernels used in HEVC remain unchanged, including 4-point DCT-2 and DST-7, 8-point, 16-point, and 32-point DCT-2. In addition, other transform kernels including 64-point DCT-2, 4-point DCT-8, 8-point, 16-point, 32-point DST-7, and DCT-8 use the 8-bit primary transform kernel. Table 2-7 - Transform and signaling mapping table To reduce the complexity of large size DST-7 and DCT-8, for DST-7 and DCT-8 blocks with size (width or height, or both width and height) equal to 32, the high frequency transform coefficients are zeroed. Only the coefficients within the 16x16 low frequency region are kept. As in HEVC, the residual of a block can be coded in transform skip mode. To avoid the redundancy of syntax coding, the transform skip flag is not signaled when the CU level MTS CU flag is not equal to 0. Note that when LFNST or MIP is activated for the current CU, the implicit MTS transform is set to DCT2. And when MTS is enabled for an inter coded block, implicit MTS can also be enabled. 2.25. Sub-block transform (SBT) In VTM, sub-block transform is introduced for inter predicted CUs. In this transform mode, only a sub-part of the residual block is coded for a CU. When a CU has cu_cbf equal to 1, cu_sbt_flag can be signaled to indicate whether the whole residual block or a sub-part of the residual block is coded. In the former case, the inter MTS information is parsed to determine the transform type of the CU. In the latter case, a part of the residual block is coded with an inferred adaptive transform and another part of the residual block is zeroed. When SBT is used for an inter coded CU, the SBT type and SBT position information are signaled in the bitstream. As FIG. 36As shown, there are two SBT types and two SBT positions. For SBT-V (or SBT-H), the TU width (or height) can be equal to half of the CU width (or height) or ¼ of the CU width (or height), resulting in 2:2 split or 1:3 / 3:1 split. The 2:2 split is like binary tree (BT) split, while the 1:3 / 3:1 split is like asymmetric binary tree (ABT) split. In ABT split, only the small region contains non-zero residual. If one dimension of the CU is 8 in luma samples, 1:3 / 3:1 split along that dimension is not allowed. There are up to 8 SBT modes for a CU. Position-dependent transform kernel selection is applied to luma transform blocks in SBT-V and SBT-H (chroma TBs always use DCT-2). The two positions of SBT-H and SBT-V are associated with different kernel transforms. More specifically, the horizontal and vertical transforms for each SBT position are specified in FIG. 36 . For example, the horizontal and vertical transforms for SBT-V position 0 are DCT-8 and DST-7, respectively. When one side of a residual TU is larger than 32, both dimensions of the transform are set to DCT-2. Thus, the sub-block transform jointly specifies the TU tiling of a residual block, the cbf, and the horizontal and vertical kernel transform types. SBT is not applied to CUs coded in inter-intra joint mode. 2.26. Template matching based adaptive Merge candidate reordering To improve coding efficiency, after constructing the Merge candidate list, the order of each Merge candidate is adjusted according to the template matching cost. The Merge candidates are arranged in the list according to ascending order of the template matching cost. It is operated in the form of subgroups. The template matching cost is measured by the SAD (sum of absolute difference) between the neighboring samples of the current CU and their corresponding reference samples. If the Merge candidate includes motion information of bi-prediction, the corresponding reference samples are the average of the corresponding reference samples in reference list 0 and the corresponding reference samples in reference list 1, as shown in FIG. 37 If the Merge candidate contains motion information at sub-CU level, the corresponding reference samples consist of the neighboring samples of the corresponding reference sub-block, as shown in FIG. 38 As shown in FIG. 39 The ordering process is operated in the form of subgroups. The first three Merge candidates are ordered together. The last three Merge candidates are ordered together. The template size (width of the left template or height of the top template) is 1. The subgroup size is 3. 2.27. Adaptive Merge candidate list ​​We can assume there are 8 merge candidates. We take the first 5 merge candidates as the first subgroup, and the next 3... Merge candidates are assigned to the second subgroup (i.e., the last subgroup). For the encoder, after constructing the Merge candidate list, as follows FIG. 40 As shown, some Merge candidates are adaptively reordered in ascending order of Merge candidate cost. More specifically, the template matching cost of the Merge candidates in all subgroups except the last subgroup is calculated; then the Merge candidates in their own subgroups except the last subgroup are reordered; finally, the final list of Merge candidates is obtained. For the decoder, after constructing the Merge candidate list, such as FIG. 41 As shown, some / no merge candidates are adaptively reordered in ascending order at the merge candidate cost. FIG. 41 In this context, the subgroup containing the selected (transmitted via signal) Merge candidate is referred to as the selected subgroup. More specifically, if the selected Merge candidate is located in the last subgroup, the Merge candidate list construction process is terminated after deriving the selected Merge candidate, no reordering is performed, and the Merge candidate list remains unchanged; otherwise, the execution process is as follows: After deriving all Merge candidates in the selected subgroup, the Merge candidate list construction process is terminated; the template matching cost of the Merge candidates in the selected subgroup is calculated; the Merge candidates in the selected subgroup are reordered; finally, a new Merge candidate list is obtained. For both the encoder and decoder, the template matching cost is derived as a function of T and RT, where T is a set of samples in the template and RT is a set of reference samples for the template. When deriving the reference samples for the template of the Merge candidate, the motion vector of the Merge candidate is rounded to integer pixel precision. The reference samples (RT) for the template used for bidirectional prediction are derived by weighted averaging the reference samples (RT0) of the template in reference list 0 and the reference samples (RT1) of the template in reference list 1 as follows. RT=((8-w)*RT0+w*RT1+4)>>3 (2-33) The weights (8-w) of the reference templates in reference list 0 and the weights (w) of the reference templates in reference list 1 are determined by the BCW indices of the Merge candidates. The BCW indices equal to {0,1,2,3,4} correspond to w equal to {-2,3,4,5,10}, respectively. If the local illumination compensation (LIC) flag of the merge candidate is true, the reference samples of the template are derived using the LIC method. The template matching cost is calculated based on the sum of absolute difference (SAD) of T and RT. The template size is 1. This means that the width of the left template and / or the height of the top template is 1. If the coding mode is MMVD, the merge candidates used to derive the base merge candidate are not reordered. If the coding mode is GPM, the merge candidates used to derive the uni-prediction candidate list are not reordered. 2.28. Geometric prediction mode with motion vector difference In geometric prediction mode with motion vector difference (GMVD), each geometric partition in GPM can decide whether to use GMVD or not. If GMVD is selected for a geometric region, the MV of this region is calculated as the sum of the MV of the merge candidate and the MVD. All other treatments remain the same as in GPM. With GMVD, the MVD is signaled as a pair of direction and distance. Nine candidate distances (1 / 4-pixel, 1 / 2-pixel, 1-pixel, 2-pixel, 3-pixel, 4-pixel, 6-pixel, 8-pixel, 16-pixel) and eight candidate directions (four horizontal / vertical directions and four diagonal directions) are involved. In addition, when pic_fpel_mmvd_enabled_flag is equal to 1, the MVD in GMVD is also left-shifted by 2 as in MMVD. 2.29. Affine model inheritance based on history parameters and non-adjacent affine mode Affine model inheritance based on history parameters (HAMI) allows affine models to be inherited from previously affine coded blocks that can not be adjacent to the current block. Similar to the enhanced regular merge mode, a non-adjacent affine mode (NA-AFF) is introduced. A first history parameter table (HPT) is established. One entry of the first HPT stores a set of affine parameters: a, b, c, and d, each affine parameter represented by a 16-bit signed integer. The entries in the HPT are categorized by reference list and reference index. Five reference indices are supported for each reference list in the HPT. In a formal way, the category of the HPT (denoted as HPTCat) is calculated as HPTCat(RefList, RefIdx) = 5 x RefList + min(RefIdx, 4), where RefList and RefIdx denote the reference picture list (0 or 1) and the reference index, respectively. For each category, up to 7 entries can be stored, resulting in a total of 70 entries in the HPT. At the beginning of each CTU row, the number of entries for each category is initialized to 0. After decoding an affine-coded CU with reference list RefList cur and RefIdx cur , the affine parameters are utilized to update the entries in the category HPTCat(RefList cur , RefIdx cur ) in a similar way as the HMVP table update. The history affine parameter-based candidate (HAPC) is derived from one of the seven neighboring 4x4 blocks denoted as A0, A1, A2, B0, B1, B2, or B3 and a set of affine parameters stored in the corresponding entry in the first HPT. The MVs of the neighboring 4x4 blocks serve as the base MVs. In a formal way, the MV of the current block at position (x, y) is calculated as: FIG. 1 where (mv h base , mv v base ) denotes the MVs of the neighboring 4x4 blocks, (x base , y base ) denotes the center position of the neighboring 4x4 blocks. (x, y) can be the top-left corner, top-right corner, and bottom-left corner of the current block to obtain the corner position MV (CPMV) for the current block, or it can be the center of the current block to obtain the regular MV for the current block. A second history parameter table (HPT) with base MV information is also appended. There are nine entries in the second HPT, one of which includes the base MV, the reference index, and four affine parameters for each reference list, as well as the base position. The additional Merge HAPC can be generated from the second HPT with the base MV information of the corresponding affine model stored in the entry. The difference between the first HPT and the second HPT is shown in FIG. 42 . In addition, a pair-wise affine Merge candidate is generated by two history-derived or non-history-derived affine Merge candidates. The pair-wise affine Merge candidate is generated by averaging the CPMVs of the existing affine Merge candidates in the list. In response to the newly introduced HAPC, the size of the subblock-based Merge candidate list is increased from 5 to 15, all of which are involved in the ARMC process. In NA-AFF, the mode to obtain the non-adjacent spatial neighbor is as follows:​FIG. 3 As shown in FIG. 6, the non-adjacent spatial neighbors in NA-AFF are defined based on the distance between the non-adjacent spatial neighbors and the current block. The distance between the non-adjacent spatial neighbors and the current block is also defined based on the width and height of the current CU, which is the same as the existing non-adjacent regular Merge candidate [8]. FIG. 3 The motion information of the non-adjacent spatial neighbors in (a) of FIG. 6 is utilized to generate additional inherited and constructed affine Merge / AMVP candidates. Specifically, for the inherited candidates, the same derivation process of the inherited affine Merge / AMVP candidates in VVC remains unchanged except that the CPMVs are inherited from the non-adjacent spatial neighbors. The non-adjacent spatial neighbors are checked based on their distance to the current block (i.e., from near to far). At a certain distance, only the first available neighbor (coded in affine mode) from each side (e.g., left and above) of the current block is included for the inherited candidate derivation. As FIG. 43 shown by the red dashed arrows in (a) of FIG. 6, the checking order of the left and above neighbors is from bottom to top and from right to left, respectively. For the first type of constructed candidate, as shown in (b) of FIG. 6, the positions of one left and above non-adjacent spatial neighbors are first determined independently; after that, the position of the left-above neighbor can be determined accordingly, which can form a rectangular virtual block together with the left and above non-adjacent neighbors. Then, as shown in (b) of FIG. 6, the motion information of the three non-adjacent neighbors is used to form CPMVs at the top-left (A), top-right (B), and bottom-left (C) of the virtual block, which are finally mapped to the current CU to generate the corresponding constructed candidate. FIG. 43 FIG. 44 The NA-AFF candidates are inserted into the existing affine Merge candidate list and affine AMVP candidate list according to the following order: Affine Merge mode: 1. SbTMVP candidate, if available. 2. Inherited from neighboring neighbors. 3. Inherited from non-adjacent neighbors. 4. Constructed from neighboring neighbors. 5. First type of constructed affine candidate from non-adjacent neighbors. 6. Zero MV. Affine AMVP mode: 1. Inherited from neighboring neighbors. 2. Constructed from neighboring neighbors. 3. Translational MV from neighboring neighbors. 4. Translational MV from temporal neighbors. 5. Inherited from non-adjacent neighbors. ​​6. A constructed affine candidate of the first type from non-adjacent neighbors. 7. Zero MV. The size of the affine Merge candidate list is increased from 5 to 15 due to the inclusion of additional candidates generated by NA-AFF. The sub-group size of ARMC for affine Merge mode is increased from 3 to 15. FIG. 43 Spatial neighbors used for deriving affine Merge / AMVP candidates are shown: (a) for deriving inherited candidates, (b) for deriving constructed candidates of the first type. In NA-AFF: 1. The region from which non-adjacent neighbors come is restricted to be within the current CTU (i.e., no additional storage requirement for row buffer). 2. The storage granularity for affine motion information (including CPMV and reference index) is reduced from 8x8 to 16x16 (i.e., only affine motion from the top-left 8x8 block is saved). Additionally, the saved CPMV is mapped to each 16x16 block before storage, so that position and size information is not needed. 3. Only the top-left and top-right CPMVs are stored (i.e., always use 4-parameter affine model for NA-AFF). 2.30. Affine MMVD In affine MMVD, an affine Merge candidate (referred to as the base affine Merge candidate) is selected, and the MVs of the control points are further refined by MVD information signaled. The MVD information for MVs of all control points are the same in one prediction direction. When the starting MV is bi-predictive MV, and the two MVs point to different sides of the current picture (i.e., one reference has a POC greater than the POC of the current picture, and the other reference has a POC less than the POC of the current picture), the MV offset added to the list 0 MV component of the starting MV has opposite value to the MV offset of the list 1 MV; otherwise, when the starting MV is bi-predictive MV, and both lists point to the same side of the current picture (i.e., both references have POCs greater than the POC of the current picture, or both have POCs less than the POC of the current picture), the MV offset added to the list 0 MV component of the starting MV has the same value as the MV offset of the list 1 MV. 2.31. Adaptive decoder-side motion vector refinement (ADMVR) In ECM-2.0, if the selected Merge candidate satisfies the DMVR condition, a multi-pass decoder-side motion vector refinement (DMVR) method is applied in the regular Merge mode. In the first pass, bilateral matching (BM) is applied to the coded block. In the second pass, BM is applied to each 16x16 sub-block within the coded block. In the third pass, the MV in each 8x8 sub-block is refined by applying bi-directional optical flow (BDOF). The adaptive decoder-side motion vector refinement method consists of two new Merge modes, which are introduced for refining the MVs only in one direction (L0 or L1) of the bi-prediction of the Merge candidate that satisfies the DMVR condition. A multi-pass DMVR process is applied to the selected Merge candidate to refine the motion vectors, however, in the first pass (i.e., PU level) DMVR, MVD0 or MVD1 is set to zero. Similar to the regular Merge mode, the Merge candidates of the proposed Merge modes are derived from the spatial neighboring coded blocks, TMVP, non-adjacent blocks, HMVP and pair-wise candidates. The difference is that only those that satisfy the DMVR condition are added to the candidate list. The two proposed Merge modes use the same Merge candidate list (i.e. ADMVR Merge list) and the Merge index is coded as in the regular Merge mode. 3. Problem In the current design of template matching, the factor (i.e., 1.125) for the determination of whether it is using uni-prediction or bi-prediction after motion vector refinement is constant, which can limit the coding performance. The refined motion vector with lower cost (e.g., MV'0) is used to further refine another refined motion vector with larger cost (e.g., MV'1) to get a further refined motion vector (MV"1). The refinement can be performed in an iterative manner. 4. DETAILED DESCRIPTION The following specific solutions should be considered as examples to explain the general concept. The solutions should not be interpreted in a narrow way. Furthermore, the solutions can be combined in any way. Iterative refinement for TM 1. It is proposed that during a motion refinement process, a first motion information (MI A ) is used to refine a second MI (MI B ). a. In one example, the motion refinement process can refer to bilateral matching. b. In one example, the motion refinement process can refer to template matching. c. In one example, the MIs A or the MIs B may be predefined. d. In one example, the MIs A or the MIs B may be in the same reference list. i. Alternatively, the MIs A or the MIs B may be in different reference lists. e. In one example, the MIs A or the MIs B may be in the same direction, such as the picture order count (POC) of both MIs A and MIs B are less than or greater than the POC of the current picture. i. Alternatively, the MIs A or the MIs B may be in different directions, such as the POC of MIs A is less than the POC of the current picture while the POC of MIs B is greater than the current picture. ii. In another example, the POC of MIs A is greater than the POC of the current picture while the POC of MIs B is less than the current picture. f. In one example, the MIs A may be used to determine the refinement process of MIs B such as: i. search range ii. starting search point iii. mode shape g. In one example, the MIs A may be refined before being used to refine MIs B . h. In one example, the MIs A or the MIs B may be refined more than once, and the i-th refined MI representation of MIs A and MIs B is denoted as MIs A (ri) and MIs B (ri) . i. In one example, iterative refinement can be used. i. In one example, the i-th refined MI (MIs A A ) of MIs (ri) may be used to obtain MIs BThe jth A refined MI (MI) B (rj) ). 1) In one example, i can be less than, equal to, or greater than j. a) In one example, i = 1 and j = 1, or i = 2 and j = 1, or i = 1 and j = 2, or i = 2 and j = 2, or i = 0 and j = 0, i = 1 and j = 0, or i = 0 and j = 1. =1. 2) In one example, in MI A The i-th refined MI (MI A (ri) ) was used to obtain MI B The j refined MIs (MI) B (rj) After that, MI B The j-th refined MI (MI B (rj) ) can be used to obtain MI A The (i+1)th refined MI (MI A (r(i+1)) ). ii. Alternative site, MI B The j-th refined MI (MI B (rj) ) can be used to obtain MI A The i-th refined MI (MI A (ri) ). iii. In one example, the template size / shape during iterative refinement may differ from the template size / shape without iterative refinement. iv. In one example, the search range and / or pattern shape (e.g., rhombus, square, intersection) used for the search may differ from the search range and / or pattern shape without iterative refinement. 2. It is proposed that the determination of the search range and / or pattern shape used in template matching can depend on the encoding and decoding information. a. In one example, encoding / decoding information may refer to the precision of the motion vector (MV) or MV difference (MVD), or indicate the syntax elements of the MV or MVD. b. In one example, the search range for the first MV precision can be greater than the search range for the second MV precision. i. In one example, the first MV precision is an integer, and the second MV precision is a fraction. ii. In one example, the precision of the first MV is 1 / M, and the precision of the second MV is 1 / N, where M is less than N. c. In one example, the number of search points for the pattern shape at the first MV precision may be greater than the number of search points for the second MV precision. i. In one example, the pattern shapes can be the same. Example in... FIG. 42 It is shown in the middle. ii. In one example, the pattern shape can be different. Example in FIG. 43 It is shown in the middle. 3. Instead of constant factors (S) as described in Section 2.22, it is proposed to determine whether at least one adaptive factor can be used for unidirectional or bidirectional forecasting. a. In one example, when the template matching cost (costBi) of bidirectional prediction is less than or equal to the cost Uni of bidirectional prediction. When (costUnit) is used, one-way prediction can be used, where costUnit is equal to the template matching cost of one-way prediction (cost0(cost0) or cost1(cost1), or min(cost0, cost1), or max(cost0, cost1)). b. In one example, it is determined that it can be used for template matching. c. In one example, determine which video unit can be used. d. In one example, more than one factor can be predefined, transmitted via signal, or derived. e. In one example, the determination of which factor to use may depend on the encoding / decoding information. i. In one example, encoding / decoding information can refer to a POC or a POC difference. The POCs of the current image and the two reference images are represented as poc0, poc1, and poc2, respectively. 1) In one example, the first factor can be used when the POC difference (d) is less than or equal to T. (S1), and when the POC difference (d) is greater than T, a second factor (S2) can be used, where T is an integer greater than 1. a) In one example, d = abs(poc0 – poc1) + abs(poc0 – poc2). b) In one example, T = 2, or 3, or 4, or 5, or 6, or 7, or 8, or 10, or 16. c) In one example, S1 is less than S, and S2 is equal to or greater than S, for example, S = 1.125. i. S1 = 1.115 and S2 = 1.125. d) In one example, S1 is equal to or smaller than S, and S2 is larger than S, e.g., S = 1.125. i. S1 = 1.125 and S2 = 1.135 ii. In one example, the coding information can refer to whether two reference pictures are in the same direction. 1) In one example, when one reference picture is in the forward direction and the other reference picture is in the backward direction, a first factor (S3) can be used, and when both reference pictures are in the forward direction or both are in the backward direction, a second factor (S4) can be used, where a reference picture in the forward direction means its POC is smaller than the POC of the current picture, and a reference picture in the backward direction means its POC is larger than the POC of the current picture and. iii. In one example, the coding information can be: 1) whether a specific coding tool is allowed. 2) the block dimension and / or the block size. 3) the depth of the block. 4) the slice / picture type and / or the partition tree type (single tree, or dual tree, or local dual tree). 5) the block position. 6) the color component. 4. It is proposed that the first motion refinement is used as part of the second motion refinement. a. In one example, the first motion refinement can refer to template matching, and the second motion refinement can refer to bilateral matching (e.g., DMVR / multi-pass DMVR / adaptive DMVR). i. In one example, the first motion refinement can refer to template matching for bi-prediction, and the second motion refinement can refer to bilateral matching (e.g., DMVR / multi-pass DMVR / adaptive DMVR). b. In one example, the coding information from the first motion refinement can be used in the second motion refinement. i. In one example, the coding information can refer to the cost calculated in the first motion refinement. General aspects 5. In the above examples, the video unit can refer to a color component / sub-picture / slice / tile / coding tree unit (CTU) / CTU row / CTU group / coding unit (CU) / prediction unit (PU) / transform unit (TU) / coding tree block (CTB) / coding block (CB) / prediction block (PB) / transform block (TB) / block / sub-block of a block / sub-region within a block / any other region containing more than one sample or pixel. 6. Whether and / or how the methods disclosed above can be applied at the sequence level / picture group level / picture level / band level / At the slice level, signals are transmitted, such as in the sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / strip header / slice header. 7. Whether and / or how the methods disclosed above can be applied to transmit signals at PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU lines / strips / films / sub-images / other types of areas containing more than one sample point or pixel. 8. Whether and / or how to apply the methods disclosed above may depend on the encoded / decoded information, such as block size, color format, single / dual tree segmentation, color components, and stripe / image type.

[0103] As used herein, the term "video unit" or "video block" can refer to a sequence, picture, strip, slice, brick, sub-picture, codec tree unit (CTU) / codec tree block (CTB), CTU / CTB line, one or more codec units (CU) / codec blocks (CB), one or more CTU / CTB, one or more Virtual Pipeline Data Units (VPDU), or a sub-region within a picture / strip / slice / brick. In the following discussion, IntraTMP can be replaced by other codec tools that rely on encoded / decoded / reconstructed information within the same region, such as palettes or intra-block copy (IBC).

[0104] FIG. 47 A flowchart of a method 4700 for video processing according to an embodiment of the present disclosure is shown. Method 4700 is implemented during the conversion between video units of a video and a bitstream of a video.

[0105] At box 4710, for the conversion between video units and video unit bitstreams, first motion information and second motion information of the video units are obtained. In some embodiments, the first motion information and second motion information are in the same reference list. In some other embodiments, the first motion information and second motion information are in different reference lists.

[0106] At frame 4720, during the motion refinement process of the video unit, the second motion information is refined using the first motion information. In some embodiments, the motion refinement process is bilateral matching. In some other embodiments, the motion refinement process is template matching.

[0107] At block 4730, a conversion is performed based on the refined first motion information and the second motion information. In some embodiments, the conversion can include encoding the video unit into a bitstream. Alternatively or additionally, the conversion can include decoding the video unit from a bitstream. In this way, it improves the coding efficiency and coding performance.

[0108] In some embodiments, the first motion information or the second motion information is in the same direction. For example, the picture order count (POC) of both the first motion information and the second motion information is less than or greater than the POC of the current picture.

[0109] In some embodiments, the first motion information or the second motion information is in different directions. For example, the POC of the first motion information is less than the POC of the current picture, and the POC of the second motion information is greater than the POC of the current picture. In some other embodiments, the POC of the first motion information is greater than the POC of the current picture, while the POC of the second motion information is less than the POC of the current picture.

[0110] In some embodiments, the first motion information is refined before being used to refine the second motion information. In some other embodiments, the first motion information or the second motion information is refined more than once.

[0111] In some embodiments, iterative refinement is used. For example, the i-th refined motion information of the first motion information is used to obtain the j-th refined motion information of the second motion information, where i and j are integers. In some embodiments, i is less than j. In some other embodiments, i is equal to j. In some further embodiments, i is greater than j. In some embodiments, i = 1 and j = 1, or i = 2 and j = 1, or i = 1 and j = 2, or i = 2 and j = 2, or i = 0 and j = 0, or i = 1 and j = 0, or i = 0 and j = 1.

[0112] In some embodiments, after the i-th refined motion information of the first motion information is used to obtain the j-th refined motion information of the second motion information, the j-th refined motion information of the second motion information is used to obtain the (i+1)-th refined motion information of the first motion information. In some other embodiments, the j-th refined motion information of the second motion information is used to obtain the i-th refined motion information of the first motion information. In some embodiments, the template size or the template shape during the iterative refinement is different from the template size or the template shape without the iterative refinement.

[0113] In some embodiments, at least one of the search range or the pattern shape used for searching in the motion refinement process is different from at least one of the search range or the pattern shape used for searching without the motion refinement. For example, the pattern shape comprises one of the following: a diamond, a square, or a cross.

[0114] In some embodiments, the first motion information or the second motion information is predefined. In some other embodiments, the first motion information is used for a refinement process to determine the second motion information. In some further embodiments, the first motion information is used for determining at least one of the following in the refinement process: a search range, a starting search point, or a pattern shape.

[0115] In some embodiments, the determination of at least one of the search range or the pattern shape used for searching in the motion refinement process depends on the coding information, and wherein the motion refinement process is template matching. For example, the coding information comprises at least one of the following: a precision of a motion vector (MV), a precision of a MV difference (MVD), or a syntax element indicating the MV or the MVD.

[0116] In some embodiments, the search range for a first MV precision is larger than the search range for a second MV precision. In some embodiments, the first MV precision is integer and the second MV precision is fractional. In some other embodiments, the first MV precision is 1 / M and the second MV precision is 1 / N, where M is smaller than N.

[0117] In some embodiments, the number of search points of a pattern shape for a first MV precision is more than the number of search points of a pattern shape for a second MV precision. For example, the pattern shape is the same. In some other embodiments, the pattern shape is different.

[0118] In some embodiments, at least one adaptive factor is used for the determination of whether uni-prediction or bi-prediction is used. In some embodiments, the determination of which factor to use depends on the coding information. For example, the coding information comprises a POC or a POC difference.

[0119] In some embodiments, a first factor is used if the POC difference is smaller than or equal to a threshold, and a second factor is used if the POC difference is larger than the threshold, wherein the threshold is an integer larger than 1. For example, d = abs(poc0 - poc1) + abs(poc0 - poc2), d represents the POC difference, poc0 represents the POC of the current picture, poc1 represents the POC of the first reference picture, and poc2 represents the POC of the second reference picture. In some embodiments, the threshold is equal to 2, or 3, or 4, or 5, or 6, or 7, or 8, or 10, or 16.

[0120] In some embodiments, the first factor is less than the factor value, and the second factor is equal to or greater than the factor value. In some other embodiments, the first factor is equal to or less than the factor value, and the second factor is greater than the factor value. For example, the factor value is 1.125.

[0121] In some embodiments, the first factor is equal to 1.115 and the second factor is equal to 1.125. In some other embodiments, the first factor is equal to 1.125, and the second factor is equal to 1.135.

[0122] In some embodiments, the coding information includes whether the two reference pictures are in the same direction. For example, if one of the two reference pictures is in forward direction and the other is in backward direction, a third factor is used, and if both of the two reference pictures are in forward direction or both are in backward direction, a fourth factor is used, and if the reference picture is in forward direction, the POC of the reference picture is less than the POC of the current picture, and if the reference picture is in backward direction, the POC of the reference picture is greater than the POC of the current picture. In some other embodiments, the coding information includes at least one of the following: whether a coding tool is allowed, a block dimension, a block size, a depth of the block, a slice type, a picture type, a partition tree type, a block position, or a color component.

[0123] In some embodiments, if a template matching cost of bi-prediction is less than or equal to a constant factor multiplied by a template matching cost of uni-prediction, the uni-prediction is used. For example, the template matching cost of uni-prediction is equal to one of the following: costO (costO is a sum of absolute differences (SAD) of listO template matching), costl (costl is a SAD of listl template matching), a minimum value between costO and costl, or a maximum value between costO and costl.

[0124] In some embodiments, the determination is used for template matching. In some other embodiments, the determination is used for the current video unit.

[0125] In some embodiments, more than one factor is predefined. In some other embodiments, more than one factor is signaled. In some further embodiments, more than one factor is derived.

[0126] In some embodiments, the first motion refinement is used as part of the second motion refinement. For example, the first motion refinement is template matching, and the second motion refinement includes bilateral matching. In some other embodiments, the first motion refinement is template matching for bi-prediction, and the second motion refinement is bilateral matching. In some embodiments, the bilateral matching includes one of the following: decoder-side motion vector refinement (DMVR), multi-pass DMVR, or adaptive DMVR.

[0127] In some embodiments, the coded information from the first motion refinement is used in the second motion refinement. For example, the coded information includes the cost calculated in the first motion refinement.

[0128] In some embodiments, the video unit comprises at least one of a color component, a prediction block (PB), a transform block (TB), a coding block (CB), a prediction unit (PU), a transform unit (TU), a coding tree block (CTB), a coding unit (CU), a coding tree unit (CTU), a CTU row, a CTU group, a slice, a tile, a subpicture, a block, a subblock of a block, a subregion within a block, or a region comprising more than one sample or pixel.

[0129] In some embodiments, the indication of whether and / or how the second motion information is refined during the motion refinement process of the video unit by using the first motion information is indicated at one of a sequence level, a picture group level, a picture level, a slice level, or a tile group level. In some other embodiments, the indication of whether and / or how the second motion information is refined during the motion refinement process of the video unit by using the first motion information is indicated in one of a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependent parameter set (DPS), a decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header. In some embodiments, the indication of whether and / or how the second motion information is refined during the motion refinement process of the video unit by using the first motion information is in one of a prediction block (PB), a transform block (TB), a coding block (CB), a prediction unit (PU), a transform unit (TU), a coding unit (CU), a virtual pipeline data unit (VPDU), a coding tree unit (CTU), a CTU row, a slice, a tile, a subpicture, or a region comprising more than one sample or pixel.

[0130] In some embodiments, the method further comprises determining, based on coded information of the video unit, whether and / or how the second motion information is refined during the motion refinement process of the video unit by using the first motion information. The coded information can comprise at least one of a block size, a color format, a single and / or dual tree partitioning, a color component, a slice type, or a picture type.

[0131] According to further embodiments of the disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream generated by a method for a video processing. The method includes obtaining first motion information and second motion information of a video unit of a video; refining the second motion information by using the first motion information during a motion refinement process of the video unit; and generating the bitstream based on the refined first motion information and the second motion information.

[0132] According to still further embodiments of the disclosure, a method for storing a bitstream of a video is provided. The method includes obtaining first motion information and second motion information of a video unit of a video; refining the second motion information by using the first motion information during a motion refinement process of the video unit; generating the bitstream based on the refined first motion information and the second motion information; and storing the bitstream in a non-transitory computer-readable recording medium.

[0133] Embodiments of the disclosure can be described in view of the following clauses, which features can be combined in any reasonable manner.

[0134] Clause 1. A method of video processing, comprising: for a conversion between a video unit of a video and a bitstream of the video unit, obtaining first motion information and second motion information of the video unit; refining the second motion information by using the first motion information during a motion refinement process of the video unit; and performing the conversion based on the refined first motion information and the second motion information.

[0135] Clause 2. The method of clause 1, wherein the motion refinement process is bilateral matching.

[0136] Clause 3. The method of clause 1, wherein the motion refinement process is template matching.

[0137] Clause 4. The method of clause 1, wherein the first motion information and the second motion information are in a same reference list, or wherein the first motion information and the second motion information are in different reference lists.

[0138] Clause 5. The method of clause 1, wherein the first motion information or the second motion information is in a same direction.

[0139] Clause 6. The method of clause 5, wherein a picture order count (POC) of both the first motion information and the second motion information is less than or greater than a POC of a current picture.

[0140] Item 7. The method of item 1, wherein the first motion information or the second motion information is in a different direction.

[0141] Item 8. The method of item 7, wherein a POC of the first motion information is less than a POC of a current picture, and a POC of the second motion information is greater than the POC of the current picture.

[0142] Item 9. The method of item 7, wherein a POC of the first motion information is greater than a POC of a current picture, and a POC of the second motion information is less than the POC of the current picture.

[0143] Item 10. The method of item 1, wherein the first motion information is refined before being used to refine the second motion information.

[0144] Item 11. The method of item 1, wherein the first motion information or the second motion information is refined more than once.

[0145] Item 12. The method of item 1, wherein iterative refinement is used.

[0146] Item 13. The method of item 12, wherein an i-th refined motion information of the first motion information is used to obtain a j-th refined motion information of the second motion information, where i and j are integers.

[0147] Item 14. The method of item 13, wherein i is less than j, or wherein i is equal to j, or wherein i is greater than j.

[0148] Item 15. The method of item 14, wherein i=l and j=l, or i=2 and j=l, or i=l and j=2, or i=2 and j=2, or i=0 and j=0, or i=l and j=0, or i=0 and j=l.

[0149] Item 16. The method of item 13, wherein after the i-th refined motion information of the first motion information is used to obtain the j-th refined motion information of the second motion information, the j-th refined motion information of the second motion information is used to obtain an (i+1)-th refined motion information of the first motion information.

[0150] Item 17. The method of item 13, wherein the j-th refined motion information of the second motion information is used to obtain the i-th refined motion information of the first motion information.

[0151] Item 18. The method according to Item 12, wherein the template size or template shape during the iterative refinement is different from the template size or template shape without the iterative refinement.

[0152] Item 19. The method according to Item 12, wherein at least one of the search range or pattern shape used for the search is different from at least one of the search range or pattern shape used for the search without iterative refinement.

[0153] Item 20. The method according to Item 19, wherein the pattern shape includes one of the following: rhombus, square, or intersection.

[0154] Item 21. The method according to Item 1, wherein the first motion information or the second motion information is predefined.

[0155] Item 22. The method according to Item 1, wherein the first motion information is used to determine the refinement process of the second motion information.

[0156] Item 23. The method according to Item 1, wherein the first motion information is used to determine at least one of the following in the refinement process: search range, starting search point, or pattern shape.

[0157] Item 24. The method according to Item 1, wherein the determination of at least one of the search range or pattern shape used for searching in the motion refinement process depends on the encoding / decoding information, and wherein the motion refinement process is template matching.

[0158] Item 25. The method according to Item 24, wherein the encoding / decoding information includes at least one of the following: the precision of the motion vector (MV), the precision of the MV difference (MVD), or a syntax element indicating the MV or MVD.

[0159] Item 26. The method according to Item 24, wherein the search range for the first MV precision is greater than the search range for the second MV precision.

[0160] Item 27. The method according to Item 26, wherein the first MV precision is an integer and the second MV precision is a fraction.

[0161] Item 28. The method according to Item 26, wherein the first MV precision is 1 / M, and the second The precision of MV is 1 / N, where M is less than N.

[0162] Item 29. The method according to Item 24, wherein the number of search points for the pattern shape with respect to the first MV precision is greater than the number of search points for the pattern shape with respect to the second MV precision.

[0163] Item 30. The method of item 29, wherein the pattern shapes are the same, or wherein the pattern shapes are different.

[0164] Item 31. The method of item 1, wherein at least one adaptation factor is used for a determination of whether uni-prediction or bi-prediction is used.

[0165] Item 32. The method of item 31, wherein the determination of which factor to use depends on coding information.

[0166] Item 33. The method of item 32, wherein the coding information comprises a POC or a POC difference.

[0167] Item 34. The method of item 32, wherein a first factor is used if a POC difference is less than or equal to a threshold, and a second factor is used if the POC difference is greater than the threshold, wherein the threshold is an integer greater than 1.

[0168] Item 35. The method of item 34, wherein d = abs(poc0 - poc1) + abs(poc0 - poc2), d represents the POC difference, poc0 represents a POC of a current picture, poc1 represents a POC of a first reference picture, and poc2 represents a POC of a second reference picture.

[0169] Item 36. The method of item 34, wherein the threshold is equal to 2, or 3, or 4, or 5, or 6, or 7, or 8, or 10, or 16.

[0170] Item 37. The method of item 34, wherein the first factor is less than a factor value, and the second factor is equal to or greater than the factor value, or wherein the first factor is equal to or less than the factor value, and the second factor is greater than the factor value.

[0171] Item 38. The method of item 37, wherein the factor value is 1.125.

[0172] Item 39. The method of item 38, wherein the first factor is equal to 1.115 and the second factor is equal to 1.125, or wherein the first factor is equal to 1.125 and the second factor is equal to 1.135.

[0173] Item 40. The method of item 32, wherein the coding information comprises whether two reference pictures are in the same direction.

[0174] Item 41. The method of item 40, wherein if one of the two reference pictures is forward and the other is backward, a third factor is used, and wherein if both of the two reference pictures are in the forward or both are in the backward, a fourth factor is used, and wherein if a reference picture is in the forward, a POC of the reference picture is less than a POC of a current picture, and if the reference picture is in the backward, the POC of the reference picture is greater than the POC of the current picture.

[0175] Item 42. The method of item 32, wherein the coding information comprises at least one of: whether a coding tool is allowed, a block dimension, a block size, a depth of a block, a slice type, a picture type, a partition tree type, a block position, or a color component.

[0176] Item 43. The method of item 31, wherein if a template matching cost of bi-prediction is less than or equal to a constant factor multiplied by a template matching cost of uni-prediction, the uni-prediction is used.

[0177] Item 44. The method of item 43, wherein the template matching cost of the uni-prediction is equal to one of: a cost 0, which is a sum of absolute differences (SAD) of list 0 template matching, a cost 1, which is a SAD of list 1 template matching, a minimum between the cost 0 and the cost 1, or a maximum between the cost 0 and the cost 1.

[0178] Item 45. The method of item 31, wherein the determination is used for template matching.

[0179] Item 46. The method of item 31, wherein the determination is used for a current video unit.

[0180] Item 47. The method of item 31, wherein more than one factor is predefined, or wherein more than one factor is signaled, or wherein more than one factor is derived.

[0181] Item 48. The method of item 1, wherein a first motion refinement is used as part of a second motion refinement.

[0182] Item 49. The method of item 48, wherein the first motion refinement is template matching and the second motion refinement comprises bilateral matching.

[0183] Item 50. The method of item 48, wherein the first motion refinement is template matching for bi-prediction and the second motion refinement is bilateral matching.

[0184] Item 51. The method of item 48 or 49, wherein the bi-prediction matching comprises one of: decoder-side motion vector refinement (DMVR), multi-pass DMVR, or adaptive DMVR.

[0185] Item 52. The method of item 48, wherein coding information from the first motion refinement is used in the second motion refinement.

[0186] Item 53. The method of item 52, wherein the coding information comprises a cost computed in the first motion refinement.

[0187] Item 54. The method of any of items 1 to 53, wherein the video unit comprises at least one of: a color component, a prediction block (PB), a transform block (TB), a coding block (CB), a prediction unit (PU), a transform unit (TU), a coding tree block (CTB), a coding unit (CU), a coding tree unit (CTU), a CTU row, a CTU group, a slice, a tile, a subpicture, a block, a subblock of a block, a subregion within a block, or a region comprising more than one sample or pixel.

[0188] Item 55. The method of any of items 1 to 53, wherein an indication of whether and / or how the second motion information is refined during the motion refinement process of the video unit by using the first motion information is indicated at one of: a sequence level, a picture group level, a picture level, a slice level, or a tile group level.

[0189] Item 56. The method of any of items 1 to 53, wherein an indication of whether and / or how the second motion information is refined during the motion refinement process of the video unit by using the first motion information is indicated in one of: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependent parameter set (DPS), a decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.

[0190] Item 57. The method of any of items 1 to 53, wherein an indication of whether and / or how the second motion information is refined during the motion refinement process of the video unit by using the first motion information is in one of: a prediction block (PB), a transform block (TB), a coding block (CB), a prediction unit (PU), a transform unit (TU), a coding unit (CU), a virtual pipeline data unit (VPDU), a coding tree unit (CTU), a CTU row, a slice, a tile, a subpicture, or a region comprising more than one sample or pixel.

[0191] Item 58. The method of any of items 1 to 53, further comprising determining, based on coded information of the video unit, whether and / or how the second motion information is refined during the motion refinement process of the video unit by using the first motion information, the coded information comprising at least one of: a block size, a color format, a single and / or dual tree partitioning, a color component, a slice type, or a picture type.

[0192] Item 59. The method of any of items 1 to 58, wherein the converting comprises encoding the video unit into the bitstream.

[0193] Item 60. The method of any of items 1 to 58, wherein the converting comprises decoding the video unit from the bitstream.

[0194] Item 61. An apparatus for video processing comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any of items 1 to 60.

[0195] Item 62. A non-transitory computer-readable storage medium storing instructions causing a processor to perform the method of any of items 1 to 60.

[0196] Item 63. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by an apparatus for video processing, wherein the method comprises: obtaining first motion information and second motion information of a video unit of the video; refining, during a motion refinement process of the video unit, the second motion information by using the first motion information; and generating the bitstream based on the refined first motion information and the second motion information.

[0197] Item 64. A method for storing a bitstream of a video, comprising: obtaining first motion information and second motion information of a video unit of the video; refining the second motion information by using the first motion information during a motion refinement process of the video unit; generating the bitstream based on the refined first motion information and the second motion information; and storing the bitstream in a non-transitory computer-readable recording medium. Example device

[0198] FIG. 48 A block diagram of a computing device 4800 in which various embodiments of the present disclosure can be implemented is shown. The computing device 4800 can be implemented as or included in the source device 110 (or the video encoder 114 or 200) or the destination device 120 (or the video decoder 124 or 300).

[0199] It should be understood that FIG. 48 The computing device 4800 shown in FIG. 48 is for purposes of illustration and explanation only and is not intended to imply any limitation on the functionality and scope of embodiments of the present disclosure.

[0200] As FIG. 48 shown, the computing device 4800 includes a general-purpose computing device 4800. The computing device 4800 can include at least one or more processors or processing units 4810, a memory 4820, a storage unit 4830, one or more communication units 4840, one or more input devices 4850, and one or more output devices 4860.

[0201] In some embodiments, the computing device 4800 can be implemented as any user terminal or server terminal having computing capability. The server terminal can be a server provided by a service provider, a mainframe computing device, or the like. The user terminal may, for example, be any type of mobile terminal, fixed terminal, or portable terminal including a mobile telephone, a station, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a game device, or any combination thereof, and includes accessories and peripherals of these devices or any combination thereof. It is contemplated that the computing device 4800 can support any type of interface to the user (such as "wearable" circuitry, etc.).

[0202] The processing unit(s) 4810 can be a physical or virtual processor and can implement various processes based on programs stored in the memory 4820. In a multi-processing system, multiple processing units can execute computer-executable instructions in parallel to improve the processing capacity of the computing device 4800. The processing unit(s) 4810 can also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0203] The computing device 4800 typically includes a variety of computer storage media. Such media can be any media that is accessible by the computing device 4800 and can include, without limitation, both volatile and non-volatile media, or removable and non-removable media. The memory 4820 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (such as read only memory (ROM), electrically erasable programmable read only memory (EEPROM) or flash memory), or any combination thereof. The storage unit 4830 can be any media that is removable or non-removable and can include machine- readable media, such as a memory, flash drive, diskette, or other media that can be used to store information and / or data and that can be accessed by the computing device 4800.

[0204] The computing device 4800 can also include additional removable / non-removable, volatile / non-volatile storage media. For example, computer storage media can include, but is not limited to, Blu-ray discs, DVDs, CD-ROMs, memory sticks, flash drives, magnetic tapes, magnetic disks, memory cards such as SD cards, and / or optical disks, among others. Although the FIG. 48 disk drives and optical disc drives are not shown in FIG. 4, these devices can be provided to read from or write to a removable, non-removable, volatile or non-volatile media. In such cases, each can be connected to the bus (not shown) by one or more data media interfaces.

[0205] The communication unit 4840 communicates with another computing device via a communication medium. Additionally, the functionality of the components in the computing device 4800 can be implemented by a single computing cluster or multiple computer machines in communication via a communication connection. Thus, the computing device 4800 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general network nodes.

[0206] Input device 4850 can be one or more of various input devices such as a mouse, keyboard, trackball, voice input device, or the like. Output device 4860 can be one or more of various output devices such as a display, a speaker, a printer, and the like. Via communication unit 4840, computing device 4800 can also communicate with one or more external devices (not shown) such as a storage device or a display device, or the like, which can be employed, for example, in carrying out the functions of the computing device 4800 as described herein. Such communication can be enabled, for example, via an input / output (I / O) interface (not shown).

[0207] In some embodiments, some or all of the components of computing device 4800 can also be arranged in a cloud computing architecture, rather than being integrated in a single device. In a cloud computing architecture, components can be provided remotely and can work together to implement the functions described in this disclosure. In some embodiments, cloud computing provides computation, software, data access, and storage services that do not require end-user knowledge of the physical location or configuration of the system or of the hardware and / or software resources being accessed. In various embodiments, cloud computing uses suitable protocols and / or formats to provide services via a wide area network, such as the Internet. For example, a cloud computing provider provides applications through a wide area network, which can be accessed through a web browser or any other computing component. Software or components of a cloud computing architecture, and corresponding data, can be stored on servers at remote locations. Computing resources in a cloud computing environment can be consolidated or distributed at locations in remote data centers. Cloud computing infrastructure can provide services through shared data centers, although they appear as a single point of access for users. Thus, a cloud computing architecture can be used to provide components and functions described herein from a service provider at a remote location. Alternatively, components and functions described herein can be provided from a conventional server, or installed directly or otherwise on a client device.

[0208] In embodiments of the present disclosure, computing device 4800 can be used to implement video encoding / decoding. Memory 4820 can include one or more video coding modules 4825 having one or more program instructions. These modules are accessible and executable by processing unit 4810 to perform the functions of various embodiments described herein.

[0209] In example embodiments that perform video encoding, input device 4850 can receive video data as input 4870 to be encoded. The video data can be processed by, for example, video coding module 4825 to generate an encoded bitstream. The encoded bitstream can be provided as output 4880 via output device 4860.

[0210] In an example embodiment in which video decoding is performed, input device 4850 can receive an encoded bitstream as input 4870. The encoded bitstream can be processed by, for example, video codec module 4825 to generate decoded video data. The decoded video data can be provided as output 4880 via output device 4860.

[0211] While the present disclosure has been particularly shown and described with reference to the preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details can be made therein without departing from the spirit and scope of the application as defined by the appended claims. Such variations are intended to be encompassed by the scope of the present application. Therefore, the foregoing description of embodiments of the application is not intended to be limiting.

Claims

1. A method for video processing, comprising: By converting between video units and bitstreams of the video units, first motion information and second motion information of the video units are obtained. During the motion refinement process of the video unit, the second motion information is refined by using the first motion information; as well as The transformation is performed based on the refined first motion information and the second motion information.

2. The method according to claim 1, wherein the motion refinement process is bilateral matching.

3. The method according to claim 1, wherein the motion refinement process is template matching.

4. The method according to claim 1, wherein the first motion information and the second motion information are in the same reference list, or The first motion information and the second motion information are in different reference lists.

5. The method according to claim 1, wherein the first motion information or the second motion information is in the same direction.

6. The method according to claim 5, wherein the image sequence count (POC) of the first motion information and the second motion information is less than or greater than the POC of the current image.

7. The method according to claim 1, wherein the first motion information or the second motion information is in different directions.

8. The method according to claim 7, wherein the POC of the first motion information is less than the POC of the current image, and the POC of the second motion information is greater than the POC of the current image.

9. The method according to claim 7, wherein the POC of the first motion information is greater than the POC of the current image, and the POC of the second motion information is less than the POC of the current image.

10. The method of claim 1, wherein the first motion information is refined before being used to refine the second motion information.

11. The method of claim 1, wherein the first motion information or the second motion information is refined more than once.

12. The method of claim 1, wherein iterative refinement is used.

13. The method according to claim 12, wherein the i-th refined motion information of the first motion information is used to obtain the j-th refined motion information of the second motion information, where i and j are integers.

14. The method of claim 13, wherein i is less than j, or Where i equals j, or Where i is greater than j.

15. The method of claim 14, wherein i = 1 and j = 1, or i = 2 and j = 1, or i = 1 and j = 2, or i = 2 and j = 2, or i = 0 and j = 0, or i = 1 and j = 0, or i = 0 and j = 1.

16. The method of claim 13, wherein after the i-th refined motion information of the first motion information is used to obtain the j-th refined motion information of the second motion information, the j-th refined motion information of the second motion information is used to obtain the (i+1)-th refined motion information of the first motion information.

17. The method of claim 13, wherein the j-th refined motion information of the second motion information is used to obtain the i-th refined motion information of the first motion information.

18. The method of claim 12, wherein the template size or template shape during the iterative refinement is different from the template size or template shape without iterative refinement.

19. The method of claim 12, wherein at least one of the search range or pattern shape used for the search is different from at least one of the search range or pattern shape used for the search without iterative refinement.

20. The method of claim 19, wherein the pattern shape includes one of the following: rhombus, square, or intersection.

21. The method according to claim 1, wherein the first motion information or the second motion information is predefined.

22. The method of claim 1, wherein the first motion information is used to determine the refinement process of the second motion information.

23. The method of claim 1, wherein the first motion information is used to determine at least one of the following in the refinement process: Search scope, Starting search point or Pattern shape.

24. The method of claim 1, wherein the determination of at least one of the search range or pattern shape used for searching during the motion refinement process depends on the encoding / decoding information, and wherein the motion refinement process is template matching.

25. The method of claim 24, wherein the encoding / decoding information comprises at least one of the following: The accuracy of motion vector (MV) The accuracy of MVD (MVD) Syntax elements that indicate MV or MVD.

26. The method of claim 24, wherein the search range for the first MV precision is greater than the search range for the second MV precision.

27. The method of claim 26, wherein the first MV precision is an integer and the second MV precision is a fraction.

28. The method of claim 26, wherein the first MV precision is 1 / M and the second MV precision is 1 / N, wherein M is less than N.

29. The method of claim 24, wherein the number of search points for the pattern shape with respect to the first MV precision is greater than the number of search points for the pattern shape with respect to the second MV precision.

30. The method of claim 29, wherein the patterns have the same shape, or The patterns described therein have different shapes.

31. The method of claim 1, wherein at least one adaptive factor is used to determine whether one-way or two-way prediction is used.

32. The method of claim 31, wherein the determination of which factor to use depends on the encoding / decoding information.

33. The method of claim 32, wherein the encoding / decoding information includes POC or POC difference.

34. The method of claim 32, wherein a first factor is used if the POC difference is less than or equal to a threshold, and a second factor is used if the POC difference is greater than the threshold, wherein the threshold is an integer greater than 1.

35. The method according to claim 34, wherein d = abs(poc0 – poc1) + abs(poc0 – poc2), d represents the POC difference, poc0 represents the POC of the current image, poc1 represents the POC of the first reference image, and poc2 represents the POC of the second reference image.

36. The method of claim 34, wherein the threshold is equal to 2, or 3, or 4, or 5, or 6, or 7, or 8, or 10, or 16.

37. The method of claim 34, wherein the first factor is less than the factor value, and the second factor is equal to or greater than the factor value, or The first factor is equal to or less than the factor value, and the second factor is greater than the factor value.

38. The method of claim 37, wherein the factor value is 1.

125.

39. The method of claim 38, wherein the first factor is equal to 1.115 and the second factor is equal to 1.125, or The first factor is equal to 1.125 and the second factor is equal to 1.

135.

40. The method of claim 32, wherein the encoding / decoding information includes whether the two reference images are in the same orientation.

41. The method of claim 40, wherein if one of the two reference images is forward and the other is backward, the third factor is used, and Where the two reference images are both in the forward direction or both in the backward direction, the fourth factor is used, and Wherein, if the reference image is in the forward direction, the POC of the reference image is less than the POC of the current image, and if the reference image is in the backward direction, the POC of the reference image is greater than the POC of the current image.

42. The method of claim 32, wherein the encoding / decoding information comprises at least one of the following: Are the codec tools allowed? Block dimension Block size, Block depth, Strip type, Image type Segmentation tree type, Block position, or Color components.

43. The method of claim 31, wherein the unidirectional prediction is used if the template matching cost of bidirectional prediction is less than or equal to a constant factor multiplied by the template matching cost of unidirectional prediction.

44. The method of claim 43, wherein the template matching cost of the one-way prediction is equal to one of the following: Cost 0, where cost 0 is the sum of absolute differences (SAD) of the template matches in list 0. Cost 1, where Cost 1 is the SAD of the template matching in List 1. The minimum value between the cost 0 and the cost 1, or The maximum value between cost 0 and cost 1.

45. The method of claim 31, wherein the determination is used for template matching.

46. ​​The method of claim 31, wherein the determination is used for the current video unit.

47. The method of claim 31, wherein more than one factor is predefined, or More than one of the factors is transmitted via a signal, or More than one of the factors was derived.

48. The method of claim 1, wherein the first motion refinement is used as part of the second motion refinement.

49. The method of claim 48, wherein the first motion refinement is template matching, and the second motion refinement includes bilateral matching.

50. The method of claim 48, wherein the first motion refinement is template matching for bidirectional prediction, and the second motion refinement is bilateral matching.

51. The method of claim 48 or 49, wherein the bilateral matching comprises one of the following: Decoder-side motion vector refinement (DMVR) Multiple DMVR cycles, or Adaptive DMVR.

52. The method of claim 48, wherein the encoding / decoding information from the first motion refinement is used in the second motion refinement.

53. The method of claim 52, wherein the encoding / decoding information includes the cost calculated in the first motion refinement.

54. The method according to any one of claims 1 to 53, wherein the video unit comprises at least one of the following: Color components, Predicted blocks (PB), Transform Block (TB) Code block (CB) Prediction Unit (PU) Transformer Unit (TU) Code-decoder tree block (CTB), Codec Unit (CU) Code-decode tree unit (CTU) CTU line, CTU group strips, piece, Sub-images piece, sub-blocks of a block Sub-regions within a block, or This includes regions containing more than one sample point or pixel.

55. The method according to any one of claims 1 to 53, wherein an indication of whether to refine the second motion information by using the first motion information during the motion refinement process of the video unit and / or how to refine the second motion information by using the first motion information during the motion refinement process of the video unit is indicated at one of the following: sequence level, Image group level, Image level, strip level, or Film series level.

56. The method according to any one of claims 1 to 53, wherein an indication of whether to refine the second motion information by using the first motion information during the motion refinement process of the video unit and / or how to refine the second motion information by using the first motion information during the motion refinement process of the video unit is indicated in one of the following: Sequence header, Image header, Sequence Parameter Set (SPS) Video Parameter Set (VPS) Dependency Parameter Set (DPS) Decoding Capability Information (DCI) Image Parameter Set (PPS) Adaptive Parameter Set (APS) strip head, or The beginning of the film.

57. The method according to any one of claims 1 to 53, wherein the indication of whether the second motion information is refined by using the first motion information during the motion refinement process of the video unit and / or how the second motion information is refined by using the first motion information during the motion refinement process of the video unit is in one of the following: Predicted blocks (PB), Transform Block (TB) Code block (CB) Prediction Unit (PU) Transformer Unit (TU) Codec Unit (CU) Virtual Pipeline Data Unit (VPDU) Code-decode tree unit (CTU) CTU line, strips, piece, Sub-images, or This includes regions containing more than one sample point or pixel.

58. The method according to any one of claims 1 to 53, further comprising: Based on the encoded and decoded information of the video unit, it is determined whether the second motion information is refined by using the first motion information during the motion refinement process of the video unit and / or how the second motion information is refined by using the first motion information during the motion refinement process of the video unit, wherein the encoded and decoded information includes at least one of the following: Block size, Color format, Single and / or dual tree partitioning Color components, strip type, or Image type.

59. The method according to any one of claims 1 to 58, wherein the conversion comprises encoding the video unit into the bitstream.

60. The method according to any one of claims 1 to 58, wherein the conversion comprises decoding the video unit from the bitstream.

61. An apparatus for video processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 60.

62. A non-transitory computer-readable storage medium storing instructions that cause a processor to execute the method according to any one of claims 1 to 60.

63. A non-transitory computer-readable recording medium storing a bitstream of video generated by a method performed by means of a video processing apparatus, wherein the method includes: Obtain the first motion information and the second motion information of the video unit of the video; During the motion refinement process of the video unit, the second motion information is refined by using the first motion information; as well as The bit stream is generated based on the refined first motion information and the second motion information.

64. A method for storing a bitstream of video, comprising: Obtain the first motion information and the second motion information of the video unit of the video; During the motion refinement process of the video unit, the second motion information is refined by using the first motion information; The bit stream is generated based on the refined first motion information and second motion information; as well as The bitstream is stored in a non-transitory computer-readable recording medium.

Images