Method and device for video processing and medium

Abstract

Embodiments of the present disclosure provide a solution for video processing. A method for video processing is presented. The method comprises: applying a processing program to at least one motion candidate in a motion candidate list for a transition between a video unit of a video and a bitstream of the video unit; reordering at least a portion of the motion candidates in the motion candidate list by applying a first reordering program to the motion candidate list after the processing program; and performing the conversion based on the reordered motion candidates.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to video processing techniques, and more particularly, to motion candidate list reordering with refined motion information. Background Art

[0002] Nowadays, digital video capabilities are being applied to all aspects of people's lives. Various types of video compression technologies, such as MPEG-2, MPEG-4, ITU-TH.263, ITU-TH.264 / MPEG-4 Part 10 Advanced Video Codec (AVC), ITU-TH.265 High Efficiency Video Codec (HEVC) standard, and Versatile Video Codec (VVC) standard have been proposed for video encoding / decoding. However, the encoding and decoding efficiency of video encoding and decoding technologies is generally expected to be further improved. Summary of the invention

[0003] The embodiments of the present disclosure provide a solution for video processing.

[0004] In a first aspect, a method for video processing is provided. The method comprises: for conversion between a video unit of a video and a bitstream of the video unit, applying a processing procedure to at least one motion candidate in a motion candidate list; reordering at least part of the motion candidates in the motion candidate list by applying a first reordering procedure to the motion candidate list after the processing procedure; and performing conversion based on the reordered motion candidates. Compared with conventional techniques, higher codec gain can be achieved. In addition, codec efficiency can be improved.

[0005] In a second aspect, a device for video processing is provided. The device includes a processor and a non-volatile memory having instructions thereon. The instructions, when executed by the processor, cause the processor to perform the method according to the first aspect of the present disclosure.

[0006] In a third aspect, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions for causing a processor to execute the method according to the first aspect of the present disclosure.

[0007] In a fourth aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video, the bitstream being generated by a method performed by a video processing device. The method includes: applying a processing program to at least one motion candidate in a motion candidate list for conversion between a video unit of the video and a bitstream of the video unit; reordering at least part of the motion candidates in the motion candidate list by applying a first reordering program to the motion candidate list after the processing program; and generating a bitstream based on the reordered motion candidates.

[0008] In a fifth aspect, a method for storing a bitstream of a video is provided. The method includes: for conversion between a video unit of the video and a bitstream of the video unit, applying a processing procedure to at least one motion candidate in a motion candidate list; reordering at least part of the motion candidates in the motion candidate list by applying a first reordering procedure to the motion candidate list after the processing procedure; generating a bitstream based on the reordered motion candidates; and storing the bitstream in a non-transitory computer-readable recording medium.

[0009] This disclosure is provided to introduce a selection of concepts in a simplified form, which are further described below in the Detailed Description. This disclosure is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become more apparent through the following detailed description with reference to the accompanying drawings.In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0011] Figure 1 A block diagram showing an example video encoding and decoding system according to some embodiments of the present disclosure is shown;

[0012] Figure 2 A block diagram illustrating a first example video encoder according to some embodiments of the present disclosure is shown;

[0013] Figure 3 A block diagram of an example video decoder according to some embodiments of the present disclosure is shown;

[0014] Figure 4 A schematic diagram showing an example of an encoder block diagram;

[0015] Figure 5 A schematic diagram showing 67 intra prediction modes;

[0016] Figure 6 A schematic diagram showing reference samples for wide-angle intra prediction is shown;

[0017] Figure 7 A schematic diagram showing the problem of discontinuity in the case of an orientation exceeding 45°;

[0018] Figure 8 A schematic diagram of the MMVD search point is shown;

[0019] Fig. 9 shows a schematic diagram for an illustration of a symmetric MVD mode;

[0020] Fig.10 shows the expanded CU area used in BDOF;

[0021] Fig.11 An affine motion model based on control points is shown;

[0022] Fig.12 The affine MVF of each sub-block is shown;

[0023] Fig.13 The position of the inherited affine motion predictor is shown;

[0024] Fig.14 Control point motion vector inheritance is shown;

[0025] Fig.15 The positions of candidate positions for the constructed affine merge pattern are shown;

[0026] Fig.16 A diagram showing the use of motion vectors for the proposed combined method;

[0027] Fig.17 Sub-block MV VSB and pixel Δv(i,j) are shown;

[0028] Fig.18a and Fig.18b The SbTMVP process in VVC is shown, where Fig.18a shows the spatially adjacent blocks used by ATVMP, and Fig.18b shows the derivation of a sub-CU motion field by applying a motion shift from a spatial neighbor and scaling the motion information from the corresponding co-located sub-CU;

[0029] Fig.19 Local illumination compensation is shown;

[0030] Fig. 20 It is shown that there is no subsampling for the short edges;

[0031] Fig.21 Decoding side motion vector refinement is shown;

[0032] Fig. 22 A diamond-shaped area in the search area is shown;

[0033] Fig.23 The positions of the spatial merging candidates are shown;

[0034] Fig.24 shows candidate pairs considered for redundancy check of spatial merging candidates;

[0035] Fig.25 A diagram showing motion vector scaling for temporal merging candidates is shown;

[0036] Fig.26 The candidate positions of the time domain merging candidates C0 and C1 are shown;

[0037] Fig. 27 The VVC spatial domain neighboring blocks of the current block are shown;

[0038] Fig.28 shows a diagram of a virtual block in the i-th search round;

[0039] Fig.29 An example of GPM segmentation grouped at the same angle is shown;

[0040] Fig.30 Unidirectional prediction MV selection for geometric partitioning mode is shown;

[0041] Fig.31 The use of geometric partitioning mode to generate the warp weight w_0 is shown;

[0042] Fig.32 shows spatial neighboring blocks used to derive spatial merge candidates;

[0043] Fig.33 It shows that template matching is performed in the search area around the initial MV;

[0044] Fig.34 A diagram showing sub-blocks of an OBMC application is shown;

[0045] Fig.35 The SBT position, type and transformation type are shown;

[0046] Fig.36 shows the neighboring samples used to calculate the SAD;

[0047] Fig.37 shows neighboring samples used to calculate SAD for sub-CU level motion information;

[0048] Fig.38 The classification process is shown;

[0049] Fig.39 shows the reordering process in the encoder;

[0050] Fig.40 shows the reordering process in the decoder;

[0051] Fig.41 A flowchart showing a method for video processing according to an embodiment of the present disclosure; and

[0052] Fig.42 A block diagram of a computing device is shown in which various embodiments of the present disclosure may be implemented.

[0053] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. DETAILED DESCRIPTION

[0054] The principles of the present disclosure are now described in conjunction with some embodiments. It should be understood that these embodiments are described only for the purpose of illustrating and helping those skilled in the art to understand and implement the present disclosure, and do not imply any limitation on the scope of the present disclosure. In addition to the methods described below, the disclosure described herein can also be implemented in various ways.

[0055] In the following description and claims, unless defined otherwise, all scientific and technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs.

[0056] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment must include the particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example embodiment, whether or not explicitly described, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in relation to other embodiments.

[0057] It should be understood that although the terms "first" and "second" etc. can be used to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish one element from another element. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0058] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms "including", "comprising", and / or "having" when used herein indicate the presence of the features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof. Example Environment

[0059] Figure 11 is a block diagram illustrating an example video codec system 100 that may utilize the techniques of the present disclosure. As shown, the video codec system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a video encoding device, and the destination device 120 may also be referred to as a video decoding device. In operation, the source device 110 may be configured to generate encoded video data, and the destination device 120 may be configured to decode the encoded video data generated by the source device 110. The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0060] Video source 112 may include a source such as a video capture device. Examples of a video capture device include, but are not limited to, an interface that receives video data from a video content provider, a computer graphics system for generating video data, and / or combinations thereof.

[0061] The video data may include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bit stream. The bit stream may include a bit sequence that forms a coded representation of the video data. The bit stream may include coded pictures and associated data. The coded pictures are coded representations of pictures. The associated data may include sequence parameter sets, picture parameter sets, and other grammatical structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The coded video data may be directly transmitted to the destination device 120 via the network 130A via the I / O interface 116. The coded video data may also be stored on a storage medium / server 130B for access by the destination device 120.

[0062] The destination device 120 may include an I / O interface 126, a video decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain encoded video data from the source device 110 or the storage medium / server 130B. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to the user. The display device 122 may be integrated with the destination device 120, or may be outside the destination device 120, which is configured to be connected to an external display device interface.

[0063] The video encoder 114 and the video decoder 124 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other existing and / or further standards.

[0064] Figure 2is a block diagram showing an example of a video encoder 200 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video encoder 114 in the system 100 is shown.

[0065] Video encoder 200 may be configured to implement any or all of the techniques of this disclosure. Figure 2 In the example of , video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video encoder 200. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0066] In some embodiments, the video encoder 200 may include a partitioning unit 201, a prediction unit 202, a residual generation unit 207, a transformation unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transformation unit 211, a reconstruction unit 212, a buffer 213 and an entropy coding unit 214, and the prediction unit 202 may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205 and an intra-frame prediction unit 206.

[0067] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, where at least one reference picture is a picture where the current video block is located.

[0068] Furthermore, although some components (such as the motion estimation unit 204 and the motion compensation unit 205) may be integrated, for the purpose of explanation, these components are described in detail below. Figure 2 are shown separately in the example.

[0069] The partitioning unit 201 may partition a picture into one or more video blocks. The video encoder 200 and the video decoder 300 may support various video block sizes.

[0070] The mode selection unit 203 may select one of a plurality of codec modes (intra-frame coding or inter-frame coding), for example, based on the error result, and provide the generated intra-frame codec block or inter-frame codec block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the codec block for use as a reference picture. In some examples, the mode selection unit 203 may select a combination of intra-frame and inter-frame prediction (CIIP) modes, where the prediction is based on an inter-frame prediction signal and an intra-frame prediction signal. In the case of inter-frame prediction, the mode selection unit 203 may also select a resolution for the motion vector (e.g., sub-pixel precision or integer pixel precision) for the block.

[0071] To perform inter-frame prediction on the current video block, the motion estimation unit 204 may generate motion information for the current video block by comparing the current video block with one or more reference frames from the buffer 213. The motion compensation unit 205 may determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from the buffer 213 other than the picture associated with the current video block.

[0072] The motion estimation unit 204 and the motion compensation unit 205 may perform different operations on the current video block, for example, depending on whether the current video block is in an I slice, a P slice, or a B slice. As used herein, an "I slice" may refer to a portion of a picture consisting of macroblocks, all of which are based on macroblocks within the same picture. Furthermore, as used herein, in some aspects, a "P slice" and a "B slice" may refer to a portion of a picture consisting of macroblocks that are independent of macroblocks in the same picture.

[0073] In some examples, the motion estimation unit 204 may perform unidirectional prediction on the current video block, and the motion estimation unit 204 may search the reference pictures of list 0 or list 1 to find the reference video block for the current video block. The motion estimation unit 204 may then generate a reference index and a motion vector, the reference index indicating the reference picture in list 0 or list 1 containing the reference video block, and the motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 204 may output the reference index, the prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0074] Alternatively, in other examples, the motion estimation unit 204 may perform bidirectional prediction on the current video block. The motion estimation unit 204 may search the reference pictures in list 0 for a reference video block for the current video block, and may also search the reference pictures in list 1 for another reference video block for the current video block. The motion estimation unit 204 may then generate a plurality of reference indexes and a plurality of motion vectors, the plurality of reference indexes indicating a plurality of reference pictures in list 0 and list 1 containing a plurality of reference video blocks, and the plurality of motion vectors indicating a plurality of spatial displacements between the plurality of reference video blocks and the current video block. The motion estimation unit 204 may output the plurality of reference indexes and the plurality of motion vectors of the current video block as motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the plurality of reference video blocks indicated by the motion information of the current video block.

[0075] In some examples, motion estimation unit 204 may output a complete set of motion information for use in a decoding process by a decoder. Alternatively, in some embodiments, motion estimation unit 204 may signal motion information of a current video block with reference to motion information of another video block. For example, motion estimation unit 204 may determine that motion information of a current video block is sufficiently similar to motion information of a neighboring video block.

[0076] In one example, motion estimation unit 204 may indicate to video decoder 300 a value in a syntax structure associated with the current video block that indicates that the current video block has the same motion information as another video block.

[0077] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0078] As discussed above, the video encoder 200 may signal motion vectors in a predictive manner.Two examples of prediction signaling techniques that may be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and merge mode signaling.

[0079] The intra prediction unit 206 may perform intra prediction on the current video block. When the intra prediction unit 206 performs intra prediction on the current video block, the intra prediction unit 206 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a prediction video block and various syntax elements.

[0080] The residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) the prediction video block(s) of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample portions of samples in the current video block.

[0081] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 207 may not perform a subtraction operation.

[0082] Transform processing unit 208 may generate one or more transform coefficient video blocks for a current video block by applying one or more transforms to the residual video block associated with the current video block.

[0083] After transform processing unit 208 generates a transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0084] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current video block for storage in the buffer 213.

[0085] After reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[0086] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the entropy coding unit 214 receives data, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy-coded data and output a bitstream including the entropy-coded data.

[0087] Figure 3 is a block diagram showing an example of a video decoder 300 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video decoder 124 in the system 100 is shown.

[0088] Video decoder 300 may be configured to perform any or all of the techniques of this disclosure. Figure 3 In the example of , video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video decoder 300. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0089] exist Figure 3 In the example of , the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, and a reconstruction unit 306 and a buffer 307. In some examples, the video decoder 300 can perform a decoding process that is generally opposite to the encoding process described with respect to the video encoder 200.

[0090] The entropy decoding unit 301 can retrieve the encoded bitstream. The encoded bitstream may include entropy encoded video data (e.g., encoded video data blocks). The entropy decoding unit 301 can decode the entropy encoded and decoded video data, and the motion compensation unit 302 can determine motion information from the entropy decoded video data, the motion information including motion vectors, motion vector precision, reference picture list indexes, and other motion information. The motion compensation unit 302 can determine the information, for example, by performing AMVP and merge mode. AMVP is used, including deriving several most likely candidates based on data and reference pictures of adjacent PBs. The motion information typically includes horizontal and vertical motion vector displacement values, one or two reference picture indexes, and in the case of prediction areas in B strips, also includes an identification of which reference picture list is associated with each index. As used herein, in some aspects, "merge mode" may refer to deriving motion information from blocks adjacent in the spatial or temporal domain.

[0091] The motion compensation unit 302 may generate motion compensated blocks, possibly performing interpolation based on interpolation filters.Identifiers for the interpolation filters used with sub-pixel precision may be included in the syntax elements.

[0092] The motion compensation unit 302 may calculate interpolated values ​​for sub-integer pixels of a reference block using interpolation filters used by the video encoder 200 during encoding of the video block. The motion compensation unit 302 may determine the interpolation filters used by the video encoder 200 based on received syntax information, and the motion compensation unit 302 may use the interpolation filters to generate a prediction block.

[0093] The motion compensation unit 302 may use at least part of the syntax information to determine the size of blocks used to encode (multiple) frames and / or (multiple) slices of the encoded video sequence, partition information describing how each macroblock of a picture of the encoded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-frame coding block, and other information for decoding the encoded video sequence. As used herein, in some aspects, a "slice" may refer to a data structure that can be decoded independently of other slices of the same picture in terms of entropy coding, signal prediction, and residual signal reconstruction. A slice may be an entire picture, or it may be a region of a picture.

[0094] The intra prediction unit 303 may use, for example, an intra prediction mode received in the bitstream to form a prediction block from spatially neighboring blocks. The inverse quantization unit 304 inverse quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 305 applies an inverse transform.

[0095] The reconstruction unit 306 may obtain the decoded block, for example, by adding the residual block to the corresponding prediction block generated by the motion compensation unit 302 or the intra prediction unit 303. If necessary, a deblocking filter may also be applied to filter the decoded block in order to remove blocking artifacts. The decoded video block is then stored in a buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction, and the buffer 307 also generates the decoded video for presentation on a display device.

[0096] Some example embodiments of the present disclosure will be described in detail below. It should be noted that the section titles used in this document are for ease of understanding, and the embodiments disclosed in the section are not limited to that section. In addition, although some embodiments are described with reference to general video codecs or other specific video codecs, the disclosed technology is also applicable to other video codec technologies. In addition, although some embodiments describe the video coding and decoding steps in detail, it should be understood that the corresponding decoding steps of the uncoding will be implemented by the decoder. In addition, the term video processing includes video coding or compression, video decoding or decompression, and video transcoding, in which video pixels are represented from one compression format to another compression format or at different compression bit rates. 1. Brief Overview The present disclosure relates to video coding techniques. Specifically, it relates to motion candidate lists using refined motion information for inter-frame coding, how and / or whether to apply a reordering process of motion candidate lists, and other coding tools in image / video coding. It can be applied to existing video coding standards such as HEVC or Versatile Video Codec (VVC). It can also be applied to future video coding standards or video codecs. 2. Introduction Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, ISO / IEC developed MPEG-1 and MPEG-4 Visual, and the two organizations worked together to develop H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Codec (AVC) and H.265 / HEVC standards. Starting with H.262, video codec standards are based on a hybrid video codec structure that uses temporal prediction plus transform codec. To explore future video codec technologies beyond HEVC, the Joint Video Exploration Team (JVET) was jointly established by VCEG and MPEG in 2015. Since then, JVET has adopted many new methods and put them into reference software called the Joint Exploration Model (JEM). In April 2018, the Joint Video Experts Group (JVET) was created between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) to work on the VVC standard, which aims to reduce bitrate by 50% compared to HEVC. The latest version of the VVC draft, Generic Video Codec (Draft 10), can be found at: http: / / phenix.it-sudparis.eu / jvet / doc_end_user / documents / 20_Teleconference / wg11 / JVET-T2001-v1.zip. The latest reference software for VVC is called VTM and can be found at: https: / / vcgit.hhi.fraunhofer.de / jvet / VVCSoftware_VTM / - / tags / VTM-11.0. 2.1. Encoding and decoding process of typical video codecs Figure 4 An example of an encoder block diagram for VVC is shown, which contains three loop filter blocks: Deblocking Filter (DF), Sample Adaptive Offset (SAO), and ALF. Unlike DF, which uses predefined filters, SAO and ALF use the original samples of the current picture to reduce the mean square error between the original and reconstructed samples by adding an offset and applying a finite impulse response (FIR) filter, respectively, using the coded side information that transmits the offset and filter coefficients through signaling. ALF is located at the last processing stage of each picture and can be seen as a tool that tries to capture and repair artifacts created by the previous stages. 2.2. Intra-mode codec with 67 intra-prediction modes To capture arbitrary edge directions present in natural videos, the number of directional intra modes is expanded from 33 used in HEVC to 65, as shown in Figure 5 As shown, planar and DC modes remain unchanged. These more dense directional intra prediction modes apply to all block sizes and to intra prediction for both luma and chroma. In HEVC, each intra-coded block has a square shape, and the length of each side of each square is a power of 2. Therefore, no division operation is required to generate the intra predictor using DC mode. In VVC, blocks can have a rectangular shape, which in general requires the use of a division operation for each block. To avoid division operations for DC prediction, only the longer sides are used to calculate the average for non-square blocks. 2.2.1. Wide-angle intra prediction Although 67 modes are defined in VVC, the exact prediction direction for a given intra prediction mode index further depends on the block shape. Conventional angular intra prediction directions are defined as 45 degrees to -135 degrees clockwise. In VVC, for non-square blocks, several conventional angular intra prediction modes are adaptively replaced with wide-angle intra prediction modes. The replaced mode is transmitted via a signal using the original mode index, which is remapped to the index of the wide-angle mode after parsing. The total number of intra prediction modes remains unchanged, i.e. 67, and the intra mode encoding and decoding method remains unchanged. To support these prediction directions, a top reference of length 2W+1 and a left reference of length 2H+1 are provided as Figure 6 is defined as shown. The number of alternative modes in the wide-angle direction mode depends on the aspect ratio of the block. The alternative intra-frame prediction modes are shown in Table 2-1. Table 2-1 Intra-frame prediction modes replaced by wide-angle mode Figure 7 A block diagram showing the discontinuity when the orientation exceeds 45 degrees. Figure 7 As shown in Figure 700, in the case of wide-angle intra prediction, two vertically adjacent prediction samples can use two non-adjacent reference samples. Therefore, a low-pass reference sample filter and side smoothing are applied to wide-angle prediction to reduce the negative impact of the increased gap Δpα. If the wide-angle mode represents a non-fractional offset. There are 8 modes in the wide-angle mode that meet this condition, and the 8 modes are [-14, -12, -10, -6, 72, 76, 78, 80]. When a block is predicted by these modes, the samples in the reference buffer are directly copied without applying any interpolation. With this modification, the number of samples that need to be smoothed is reduced. In addition, it aligns the design of non-fractional modes in traditional prediction mode and wide-angle mode. In VVC, 4:2:2 and 4:4:4 as well as 4:2:0 chroma formats are supported. The chroma derivation mode (DM) derivation table for the 4:2:2 chroma format was originally ported from HEVC, which expanded the number of entries from 35 to 67 to be consistent with the expansion of intra prediction modes. Since the HEVC specification does not support prediction angles below -135 degrees and above 45 degrees, the luma intra prediction modes ranging from 2 to 5 are mapped to 2. Therefore, the chroma DM derivation table for the 4:2:2: chroma format is updated by replacing the values ​​of the entries of the mapping table to more accurately convert the prediction angles of the chroma blocks. 2.3. Inter-frame prediction For each inter-predicted CU, motion parameters consisting of motion vector, reference picture index and reference picture list usage index, as well as additional information required by the new codec features of VVC for sample generation for inter-prediction. Motion parameters can be signaled explicitly or implicitly. When a CU is coded in skip mode, the CU is associated with one PU and has no significant residual coefficients, no coded motion vector increments or reference picture indices. Merge mode is specified, whereby the motion parameters of the current CU are obtained from neighboring CUs, which include spatial and temporal candidates, as well as additional arrangements introduced in VVC. Merge mode can be applied to any inter-predicted CU, not just skip mode. An alternative to merge mode is explicit transmission of motion parameters, where the motion vector, the corresponding reference picture index and reference picture list usage flag for each reference picture list, as well as other required information, are explicitly signaled for each CU. 2.4. Intra-block copy (IBC) Intra-block copying (IBC) is a tool adopted in the HEVC extension on SCC. It is well known that it significantly improves the coding efficiency of screen content materials. Since the IBC mode is implemented in a block-level coding mode, block matching (BM) is performed at the encoder to find the best block vector (or motion vector) for each CU. Here, the block vector is used to indicate the displacement from the current block to the reference block, which has been reconstructed in the current picture. The luminance block vector of the IBC-coded CU is integer precision. The chrominance block vector is also rounded to integer precision. When combined with AMVR, the IBC mode can switch between 1-pixel l and 4-pixel motion vector precision. The IBC-coded CU is regarded as a third prediction mode in addition to the intra or inter prediction mode. The IBC mode is applicable to CUs with width and height less than or equal to 64 luminance samples. At the encoder side, hash-based motion estimation is performed for IBC. The encoder performs RD checks on blocks with width or height no greater than 16 luma samples. For non-merge mode, a block vector search is first performed using a hash-based search. If the hash search does not return a valid candidate, a local search based on block matching is performed. In hash-based search, the hash key match (32-bit CRC) between the current block and the reference block is extended to all allowed blocks. The hash key calculation for each position in the current picture is based on a 4×4 sub-block. For a current block of larger size, the hash key is determined to match the hash key of the reference block when all hash keys of all 4×4 sub-blocks match the hash key of the corresponding reference position. If the hash keys of multiple reference blocks are found to match the hash key of the current block, the block vector cost of each matching reference is calculated and the one with the smallest cost is selected. In block matching search, the search range is set to cover previous and current CTUs. At the CU level, the IBC mode is signaled using a flag. The IBC mode can be signaled as IBC AMVP mode or IBC skip / merge mode as shown below: – IBC skip / merge mode: The merge candidate index is used to indicate which block vector from the list of neighboring candidate IBC codec blocks is used to predict the current block. The merge list consists of spatial candidates, HMVP candidates and pairwise candidates. – IBC AMVP mode: Block vector differences are encoded in the same way as motion vector differences. The block vector prediction method uses two candidates as predictors, one from the left neighbor and one from the upper neighbor (if IBC is encoded). When either neighbor is not available, the default block vector will be used as the predictor. A flag is signaled to indicate the block vector predictor index. 2.5. Merge Mode with MVD (MMVD) In addition to the merge mode that uses implicitly derived motion information directly for prediction sample generation of the current CU, a merge mode with motion vector difference (MMVD) is introduced in VVC. The MMVD flag is signaled immediately after the regular merge flag is sent to specify whether the MMVD mode is used for the CU. In MMVD, after a merge candidate is selected, it is further refined by MVD information transmitted by signal. Further information includes a merge candidate flag, an index specifying the magnitude of motion, and an index indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the MV basis. The MMVD candidate flag is transmitted by signal to specify which one between the first and second merge candidates is used. The distance index specifies the motion magnitude information and indicates a predefined offset from the starting point. Figure 8The MMVD search points are shown. Figure 8 As shown, the offset is added to the horizontal component or the vertical component of the starting MV. The relationship between the distance index and the predefined offset is specified in Table 2-2. Table 2-2 – Relationship between distance index and predefined offsets Distance Index 0 1 2 3 4 5 6 7 Bias (in luma samples) 1 / 4 1 / 2 1 2 4 8 16 32 The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions, as shown in Table 2-3. Note that the meaning of the MVD symbol may vary depending on the information of the starting MV. When the starting MV is a non-predicted MV or a bidirectionally predicted MV, where both lists point to the same side of the current picture (i.e., both referenced POCs are greater than the POC of the current picture, or both are less than the POC of the current picture), the symbol in Table 2-3 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectionally predicted MV, where the two MVs point to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), and the difference in POC in list0 is greater than the difference in POC in list1, the symbol in Table 2-3 specifies the sign of the MV offset added to the MV component of the starting MV of column list0, and the sign for list1 MV has the opposite value. Otherwise, if the difference in POC in list1 is greater than list0, the symbol in Table 2-3 specifies the sign of the MV offset added to the list1MV component of the starting MV and the sign for list0 MV has the opposite value. The MVD is scaled according to the difference in POC in each direction. If the POC difference in both lists is the same, no scaling is required. Otherwise, if the POC difference in list0 is greater than the POC difference in list1, the MVD for list1 is scaled by defining the POC difference of L0 as td and the POC difference of L1 as tb, as Fig. 9 If the POC difference of L1 is greater than that of L0, the MVD for list 0 is scaled in the same way. If the starting MV is unidirectionally predicted, the MVD is added to the available MVs. Table 2-3 – MV offset symbols specified by direction index Direction Index 00 01 10 11 x-axis + - N / A N / A y-axis N / A N / A + - 2.6. Symmetric MVD encoding and decoding In VVC, in addition to normal uni-prediction and bi-prediction mode MVD signaling, symmetric MVD mode for bi-prediction MVD signaling is applied. In symmetric MVD mode, the motion information including reference picture indexes of both list-0 and list-1 and MVD of list-1 is not signaled but derived. The decoding process of the symmetric MVD mode is as follows: 1. At the stripe level, the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 are derived as follows: – If mvd_l1_zero_flag is 1, BiDirPredFlag is set equal to 0. – Otherwise, if the nearest reference picture in list-0 and the nearest reference picture in list-1 form a previous-next pair of reference pictures or a previous-next pair of reference pictures, BiDirPredFlag is set to 1, and both the list-0 reference picture and the list-1 reference picture are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. 2. At CU level, if the CU is bi-directionally predicted and BiDirPredFlag is equal to 1, the symmetry mode flag indicating whether the symmetry mode is used is explicitly signaled. When the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag and MVD0 are explicitly signaled. The reference index for list-0 and list-1 are set equal to the reference picture pair, respectively. MVD1 is set equal to (-MVD0). The final motion vector is shown below. Fig. 9 is an illustration for the symmetric MVD mode. In the encoder, symmetric MVD motion estimation starts with an initial MV evaluation. A set of initial MV candidates includes MVs obtained from unidirectional prediction search, MVs obtained from bidirectional prediction search, and MVs from the AMVP list. The one with the lowest rate-distortion cost is selected as the initial MV for the symmetric MVD motion search. 2.7. Bidirectional Optical Flow (BDOF) The Bidirectional Optical Flow (BDOF) tool is included in VVC. BDOF, formerly known as BIO, is included in JEM. Compared to the JEM version, BDOF in VVC is a simpler version that requires much less computation, especially in terms of the number of multiplications and the size of the multipliers. BDOF is used to refine the bidirectional prediction signal of a CU at the 4×4 sub-block level. BDOF is applied to a CU if all of the following conditions are met: The CU is encoded using "true" bi-prediction mode, ie, one of the two reference pictures precedes the current picture in display order, and the other of the two reference pictures follows the current picture in display order. The distances from the two reference pictures to the current picture (ie, the POC difference) are the same. Both reference images are short-term reference images. The CU is not encoded or decoded using affine mode or SbTMVP merge mode. The CU has more than 64 luma samples. Both CU height and CU width are greater than or equal to 8 luma samples. The BCW weight index indicates equal weight. WP is not enabled for the current CU. CIIP mode is not used for the current CU. BDOF is applied only to the luma component. As its name suggests, BDOF mode is based on the concept of optical flow, which assumes that the motion of objects is smooth. For each 4×4 sub-block, motion refinement (v x ,v y ) is calculated by minimizing the difference between L0 prediction samples and L1 prediction samples. Then motion refinement is used to adjust the bi-prediction sample values ​​in the 4x4 sub-block. The following steps are applied in the BDOF process. First, by directly calculating the difference between two adjacent samples, the horizontal and vertical gradients of the two prediction signals, and k=0,1,is calculated, that is, Among them I (k) (i, j) is the sample value at the prediction signal coordinate (i, j) in list k, k=0,1, and shift1 is calculated based on the luma bit depth bitDepth as shift1=max(6, bitDepth-6). Then, the autocorrelations and cross-correlations of the gradients S1, S2, S3, S5, and S6 are calculated as follows: in, where Ω is a 6×6 window around a 4×4 sub-block, and n a and n b The values ​​are set to min(1, bitDepth-11) and min(4, bitDepth-8) respectively. Then the cross-correlation and autocorrelation terms are used to refine the motion (v x , v y ) is exported using: in th′ BIO =2 max(5,BD-7) . is the round-down (floor) function, and Based on the motion refinement and gradients, the following adjustments are computed for each sample in the 4×4 sub-block: Finally, the BDOF samples of the CU are calculated by adjusting the bidirectional prediction samples as shown below: pred BDOF (x, y) = (I (0) (x,y)+I (1) (x, y) + b(x, y) + o offset )>>shift (2-7) These values ​​are chosen so that the multipliers in the BDOF process do not exceed 15 bits and the maximum bit width of the intermediate parameters in the BDOF process remains within 32 bits. In order to derive the gradient value, some prediction samples I in the list k (k = 0, 1) outside the current CU boundary are (k) (i,j) needs to be generated. Fig.10 FIG. 4 is a schematic diagram showing an extended CU region used in BDOF. Fig.10 As shown in the schematic diagram 1000 of FIG. 1 , BDOF in VVC uses an extended row / column around the boundary of the CU. In order to control the computational complexity of generating prediction samples outside the boundary, the extended area (in Fig.10 The prediction samples in 1010 are generated by directly taking the reference samples at the nearby integer positions (using the floor() operation on the coordinates) without interpolation, and the conventional 8-tip motion compensation interpolation filter is used to generate the CU (in Fig.10 The predicted samples within the CU are represented as 1020 in FIG. 1020 . These extended sample values ​​are only used for gradient calculations. For the remaining steps in the BDOF process, if any sample values ​​and gradient values ​​outside the CU boundary are needed, these sample values ​​and gradient values ​​are filled (i.e., repeated) from their nearest neighbors. When the width and / or height of a CU is greater than 16 luma samples, it will be divided into sub-blocks with width and / or height equal to 16 luma samples, and the sub-block boundaries are regarded as CU boundaries in the BDOF process. The maximum unit size of the BDOF process is limited to 16x16. For each sub-block, the BDOF process can be skipped. When the SAD between the initial L0 prediction sample and the L1 prediction sample is less than a threshold, the BDOF process is not applied to the sub-block. The threshold is set equal to (8*W*(H>>1), where W represents the sub-block width and H represents the sub-block height. In order to avoid the additional complexity of SAD calculation, the SAD between the initial L0 prediction sample and the L1 prediction sample calculated in the DVMR process is reused here. If BCW is enabled for the current block, that is, the BCW weight index indicates unequal weights, bidirectional optical flow is disabled. Similarly, if WP is enabled for the current block, that is, luma_weight_lx_flag of any of the two reference pictures is 1, BDOF is also disabled; BDOF is also disabled when the CU is encoded and decoded using symmetric MVD mode or CIIP mode. 2.8. Combined Inter and Intra Prediction (CIIP) 2.9. Affine Motion Compensated Prediction In HEVC, motion compensated prediction (MCP) only applies the translational motion model. In the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensated prediction is applied. Fig.11 As shown, the affine motion field of a block is described by motion information of two control points (4 parameters) or three control point motion vectors (6 parameters). against Fig.11 Using the 4-parameter affine motion model 1110 in , the motion vector at the sample position (x, y) in the block is derived as: against Fig.11 Using the 6-parameter affine motion model 1120 in , the motion vector at the sample position (x, y) in the block is derived as: Where (mv0x,mv0y) is the motion vector of the upper left control point, (mv1x,mv1y) is the motion vector of the upper right control point, and (mv2x,mv2y) is the motion vector of the lower left control point. To simplify motion compensated prediction, block-based affine transform prediction is applied. Fig.12 A schematic diagram 1200 of the affine MVF for each sub-block is shown. To derive the motion vector for each 4×4 luminance sub-block, as Fig.12 As shown, the motion vector of the center sample of each subblock is calculated according to the above equation and rounded to 1 / 16 fractional precision. Then the motion compensated interpolation filter is applied to generate a prediction of each subblock with a derived motion vector. The subblock size of the chroma component is also set to 4×4. The MV of a 4×4 chroma subblock is calculated as the average of the MVs of the four corresponding 4×4 luminance subblocks. Like translational motion inter prediction, there are two affine motion inter prediction modes: affine merge mode and affine AMVP mode. 2.9.1. Affine Merge Prediction AF_MERGE mode can be applied to CUs with width and height greater than or equal to 8. In this mode, the CPMV of the current CU is generated based on the motion information of the spatially adjacent CUs. There can be up to five CPMVP candidates, and the index is transmitted by signal to indicate the CPMVP used for the current CU. The following three types of CPVM candidates are used to form the affine merge candidate list: - Inherited affine merge candidates inferred from the CPMV of neighboring CUs -Affine merge candidate CPMVP constructed using the translation MV of the neighboring CU -Zero MV In VVC, there are at most two inherited affine candidates, which are derived from the affine motion models of neighboring blocks, one from the left neighboring CU and one from the upper neighboring CU. Fig.13 A schematic diagram 1300 is shown showing the location of the inherited affine motion predictor. The candidate blocks are as follows Fig.13 As shown. For the left predictor, the scan order is A0->A1, and for the upper predictor, the scan order is B0->B1->B2. Only the first inheritance candidate from each side is selected. Pruning check is not performed between two inheritance candidates. When a neighboring affine CU is identified, its control point motion vectors are used to derive the CPMVP candidate in the affine merge list of the current CU. Fig.14 Schematic diagram 1400 of control point motion vector inheritance is shown. Fig.14 As shown, if the neighbor lower left block A1410 is encoded and decoded in affine mode, the motion vectors v2, v3 and v4 of the upper left corner, upper right corner and lower left corner of CU1420 containing block A1410 are obtained. When block A1410 is encoded and decoded with a 4-parameter affine model, two CPMVs of the current CU are calculated based on v2, v3 and v4. When block A is encoded and decoded with a 6-parameter affine model, three CPMVs of the current CU are calculated based on v2, v3 and v4. Constructing affine candidates means that candidates are constructed by combining the neighboring translation motion information of each control point. Fig.15 The specified spatial and temporal neighbors are derived as shown. Fig.15 A schematic diagram 1500 of constructing the position of the affine merge mode candidate position is shown. CPMVk (k=1, 2, 3, 4) represents the kth control point. For CPMV1, the B2->B3->A2 block is checked, and the MV of the first available block is used. For CPMV2, the B1->B0 block is checked, and for CPMV3, the A1->A0 block is checked. For TMVP, if available, TMVP is used as CPMV4. After the MVs of the four control points are obtained, an affine merge candidate is constructed based on the motion information. The following combinations of control point MVs are used to construct in order: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3} A combination of 3 CPMVs constructs a 6-parameter affine merge candidate, and a combination of 2 CPMVs constructs a 4-parameter affine merge candidate. To avoid the motion scaling process, the relevant combination of control point MVs is discarded if the reference indices of the control points are different. After the inherited affine merge candidates and the constructed affine merge candidates are checked, if the list is still not full, a zero MV is inserted at the end of the list. 2.9.2. Affine AMVP Prediction Affine AMVP mode can be applied to CUs with width and height greater than or equal to 16. A CU-level affine flag is signaled in the bitstream to indicate whether affine AMVP mode is used, and then another flag is signaled to indicate whether it is 4-parameter affine or 6-parameter affine. In this mode, the difference between the CPMVP of the current CU and their predictor CPMVP is signaled in the bitstream. The affine AVMP candidate list size is 2, which is generated in order using the following four types of CPVM candidates: - Inherited affine AMVP candidates inferred from the CPMV of neighboring CUs -Affine AMVP candidate CPMVP constructed using the translation MV of the neighboring CU -Translated MV from neighboring CU -Zero MV The order in which inherited affine AMVP candidates are checked is the same as that of inherited affine merge candidates. The only difference is that for AVMP candidates, only affine CUs with the same reference picture as in the current block are considered. No pruning process is applied when inserting the inherited affine motion predictor into the candidate list. The constructed AMVP candidates are Fig.13The specified spatial neighbors shown are derived. The same check order as in the affine merge candidate construction is used. In addition, the reference picture index of the neighboring blocks is also checked. The first block in the check order is used, which is inter-coded and has the same reference picture as in the current CU. There is only one. When the current CU is coded with a 4-parameter affine mode and both mv0 and mv1 are available, they are added as a candidate in the affine AMVP list. When the current CU is coded with a 6-parameter affine mode and all three CPMVs are available, they are added as a candidate in the affine AMVP list. Otherwise, the constructed AMVP candidate is set to unavailable. If the affine AMVP list candidate is still less than 2 after the inherited affine AMVP candidate and the constructed AMVP candidate are checked, mv0, mv1 and mv2 are added as translation MVs in sequence to predict all control point MVs of the current CU when available. Finally, if the affine AMVP list is still not full, the zero MV is used to fill the affine AMVP list. 2.9.3. Affine motion information storage In VVC, the CPMV of the affine CU is stored in a separate buffer. The stored CPMV is only used to generate the inherited CPMV in the affine merge mode and the inherited CPMV in the affine AMVP mode for the most recently coded CU. The sub-block MV derived from the CPMV is used for motion compensation, merging of translation MVs / MV derivation of AMVP lists, and deblocking. To avoid picture line buffering for additional CPMV, the affine motion data inheritance from the CU of the upper CTU is handled differently from the inheritance from the normal neighboring CU. If the candidate CU for affine motion data inheritance is in the upper CTU row, the lower left and lower right sub-block MVs in the line buffer are used for affine MVP derivation instead of CPMV. In this way, CPMV is only stored in the local buffer. If the candidate CU is 6-parameter affine coded, the affine model is downgraded to a 4-parameter model. Fig.16 As shown, along the top CTU boundary, the bottom left and bottom right sub-block motion vectors of the CU are used for affine inheritance of the CU in the bottom CTU. 2.9.4. Prediction refinement using optical flow for affine modality Compared with pixel-based motion compensation, sub-block-based affine motion compensation can save memory access bandwidth and reduce computational complexity, but at the expense of prediction accuracy. In order to achieve finer motion compensation granularity, prediction refinement using optical flow (PROF) is used to refine sub-block-based affine motion compensation predictions without increasing the memory access bandwidth of motion compensation. In VVC, after sub-block-based affine motion compensation is performed, the brightness prediction samples are refined by adding the differences derived from the optical flow equation. PROF is described as the following four steps: Step 1) Sub-block based affine motion compensation is performed to generate sub-block predictions. Step 2) Use a 3-tap filter [-1, 0, 1], the spatial gradient g of the sub-block prediction x (i, j) and g y (i, j) is calculated at each sample position. The gradient calculation is exactly the same as the gradient calculation in BDOF. g x (i,j)=(I(i+1,j)>>shift1)-(I(i-1,j)>>shift1) (2-10) g y (i,j)=(I(i,j+1)>>shift1)-(I(i,j-1)>>shift1) (2-11) shift1 is used to control the precision of the gradient. The sub-block (i.e. 4x4) prediction is extended by one sample on each side of the gradient calculation. To avoid extra memory bandwidth and extra interpolation calculations, these extended samples on the extended boundary are copied from the nearest integer pixel position in the reference picture. Step 3) The brightness prediction refinement is calculated by following the optical flow equation. ΔI(i, j) = g x (i, j)*Δv x (i, j)+g y (i, j)*Δv y (i, j) (2-12) where Δv(i, j) is the difference between the sample MV calculated for the sample position (i, j), denoted as v(i, j), and the sub-block MV of the sub-block to which the sample (i, j) belongs, as Fig.17 As shown in the figure, Δv(i, j) is quantized in units of 1 / 32 brightness sample accuracy. Since the affine model parameters and the sample positions relative to the subblock center do not change from subblock to subblock, Δv(i, j) can be calculated for the first subblock and reused for other subblocks in the same CU. Let dx(i, j) and dy(i, j) be the distance from the sample position (i, j) to the subblock center (x SB ,y SB )’s horizontal and vertical offsets, Δv(x, y) can be derived by the following equations: To maintain accuracy, the sub-block (x SB ,y SB ) is calculated as ((WSB-1) / 2, (HSB-1) / 2), where WSB and HSB are the width and height of the sub-block, respectively. For the 4-parameter affine model, For the 6-parameter affine model, Where (v 0x , v 0y ), (v 1x , v 1y ), (v 2x , v 2y ) are the control point motion vectors of the upper left, upper right and lower left, and w and h are the width and height of the CU. Step 4) Finally, the luma prediction refinement ΔI(i, j) is added to the sub-block prediction I(i, j) and the final prediction I' is generated as follows. I′(i,j)=I(i,j)+ΔI(i,j) (2-17) PROF is not applicable to affine codec CUs in two cases: 1) all control point MVs are the same, which indicates that the CU has only translational motion; 2) the affine motion parameters are larger than the specified limit, because the sub-block based affine MC is downgraded to CU based MC to avoid large memory access bandwidth requirements. A fast codec method is applied to reduce the codec complexity of affine motion estimation using PROF. PROF is not applied to the affine motion estimation stage in the following two cases: a) If the CU is not a root block and the parent block of the CU does not select the affine mode as its best mode, PROF is not applied because the possibility that the current CU selects the affine mode as the best mode is low; b) If the magnitudes of the four affine parameters (C, D, E, F) are all less than a predefined threshold and the current picture is not a low-latency picture, PROF is not applied. In this way, affine motion estimation using PROF can be accelerated. 2.10. Sub-block based temporal motion vector prediction (SbTMVP) VVC supports a sub-block-based temporal motion vector prediction (SbTMVP) method. Similar to the temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the co-located picture to improve the motion vector prediction and merge mode for the CU in the current picture. The same co-located picture used by TMVP is also used for SbTMVP. SbTMVP differs from TMVP in the following two main aspects: TMVP predicts motion at the CU level, but SbTMVP predicts motion at the sub-CU level; TMVP obtains the temporal motion vector from the co-located block in the co-located picture (the co-located block is the lower right or center block relative to the current CU), while SbTMVP applies motion shifting before obtaining the temporal motion information from the co-located picture, where the motion shifting is obtained from the motion vector of one of the spatial neighboring blocks from the current CU. SbTMVP process Fig.18a and Fig.18b shown. Fig.18a Schematic diagram 1810 of the spatial neighboring blocks used by SbTMVP is shown. SbTMVP predicts the motion vector of the sub-CU within the current CU in two steps. In the first step, Fig.18a The spatial neighbors A1 in are checked. If A1 has a motion vector that uses the co-located picture as its reference picture, then that motion vector is selected as the motion shift to be applied. If no such motion is identified, then the motion offset is set to (0,0). Fig.18b A schematic diagram of deriving a sub-CU motion region by applying a motion shift from a spatial neighbor and scaling the motion information from the corresponding co-located sub-CU is illustrated. In a second step, the motion shift identified in step 1 is applied (i.e., added to the coordinates of the current block in the current picture 1820) to obtain sub-CU level motion information (motion vector and reference index) from the co-located picture 1822 as shown in FIG. 18 . Fig.18b The example in assumes that the motion shift is set to the motion of block A1. Then, for each sub-CU, the motion information of the corresponding block (the minimum motion grid covering the center sample) in the co-located picture 1822 is used to derive the motion information for the sub-CU. After the motion information of the co-located sub-CU is identified, it is converted into a motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference picture of the temporal motion vector with the reference picture of the current CU. In VVC, a sub-block based merge list containing a combination of both SbTMVP candidates and affine merge candidates is used for signaling of sub-block based merge mode. The SbTMVP mode is enabled / disabled by a sequence parameter set (SPS) flag. If the SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry in the list of sub-block based merge candidates and is followed by the affine merge candidate. The size of the sub-block based merge list is signaled in the SPS, and the maximum allowed size of the sub-block based merge list is 5 in VVC. The sub-CU size used in SbTMVP is fixed to 8x8. Like the affine merge mode, the SbTMVP mode is only applicable to CUs whose width and height are greater than or equal to 8. The encoding logic of the additional SbTMVP merge candidate is the same as that for other merge candidates, ie, for each CU in a P slice or a B slice, an additional RD check is performed to decide whether to use the SbTMVP candidate. 2.11. Adaptive Motion Vector Resolution (AMVR) In HEVC, when use_integer_mv_flag in the slice header is equal to 0, the motion vector difference (MVD) (between the motion vector of the CU and the predicted motion vector) is transmitted by signal in units of quarter luma samples. In VVC, the CU-level adaptive motion vector resolution (AMVR) scheme is introduced. AMVR allows the MVD of a CU to be encoded and decoded with different precisions. Depending on the mode for the current CU (normal AMVP mode or affine AVMP mode), the MVD of the current CU can be adaptively selected as follows: - Normal AMVP mode: quarter-brightness-sample, half-brightness-sample, whole-brightness-sample or quad-brightness-sample. - Affine AMVP modes: quarter-luminance-sample, whole-luminance-sample, or 1 / 16-luminance-sample. The CU-level MVD resolution indication is conditionally signaled if the current CU has at least one non-zero MVD component.If all MVD components (ie, both horizontal and vertical MVD for reference list L0 and reference list L1) are zero, a quarter-luminance-sample MVD resolution is inferred. For a CU with at least one non-zero MVD component, a first flag is transmitted by a signal to indicate whether quarter-luminance-sample MVD precision is used for the CU. If the first flag is 0, no further signaling is required, and quarter-luminance-sample MVD precision is used for the current CU. Otherwise, a second flag is transmitted by a signal to indicate that half-luminance-sample or other MVD precision (whole or four-luminance samples) is used for a normal AMVP CU. In the case of half-luminance-samples, a 6-tap interpolation filter is used for half-luminance sample positions instead of the default 8-tap interpolation filter. Otherwise, a third flag is transmitted by a signal to indicate whether whole-luminance-sample MVD precision or four-luminance-sample MVD precision is used for a normal AMVP CU. In the case of an affine AMVP CU, a second flag is used to indicate whether whole-luminance-sample MVD precision or 1 / 16 luminance-sample MVD precision is used. To ensure that the reconstructed MV has the expected precision (quarter-luma-sample, half-luma-sample, whole-luma-sample, or quad-luma-sample), the motion vector predictor for the CU is rounded to the same precision as the MVD before the MVD is added. The motion vector predictor is rounded towards zero (i.e., a negative motion vector predictor is rounded towards positive infinity, and a positive motion vector predictor is rounded towards negative infinity). The encoder uses RD check to determine the motion vector resolution for the current CU. In order to avoid always performing four CU-level RD checks for each MVD resolution, in VTM11, only the RD check of MVD precision other than a quarter-brightness-sample is conditionally called. For the normal AVMP mode, the RD cost of a quarter-brightness-sample MVD precision and a whole-brightness-sample MV precision is calculated first. Then, the RD cost of the whole-brightness-sample MVD precision is compared with the RD cost of the quarter-brightness-sample MVD precision to determine whether it is necessary to further check the RD cost of the four-brightness-sample MVD precision. When the RD cost for the quarter-brightness-sample MVD precision is much less than the whole-brightness-sample MVD precision, the RD check of the four-brightness-sample MVD precision is skipped. Then, if the RD cost of the whole-brightness-sample MVD precision is significantly greater than the best RD cost of the previously tested MVD precision, the check of the half-brightness-sample MVD precision is skipped. For the affine AMVP mode, if the affine inter mode is not selected after checking the rate-distortion cost of the affine merge / skip mode, merge / skip mode, quarter-luma-sample MVD accuracy normal AMVP mode, and quarter-luma-sample MVD accuracy affine AMVP mode, the 1 / 16 luma-sample MV accuracy and 1-pixel MV accuracy affine inter modes are not checked. In addition, the affine parameters obtained in the quarter-luma-sample MV accuracy affine inter mode are used as the starting search points for the 1 / 16 luma-sample and quarter-luma-sample MV accuracy affine inter modes. 2.12. Bidirectional Prediction with CU-Level Weights (BCW) In HEVC, a bidirectional prediction signal is generated by averaging two prediction signals obtained from two different reference pictures and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging to allow weighted averaging of the two prediction signals. P bi-pred =((8-w)*P0+w*P1+4)>>3 (2-18) Five weights, w∈{-2, 3, 4, 5, 10}, are allowed in weighted average bidirectional prediction. For each bidirectional prediction CU, the weight w is determined in one of two ways: 1) For non-merged CUs, the weight index is signaled after the motion vector difference; 2) For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applied to CUs with 256 or more luma samples (i.e., the CU width multiplied by the CU height is greater than or equal to 256). For low-latency pictures, all 5 weights will be used. For non-low-latency pictures, only 3 weights (w∈{3, 4, 5}) are used. -At the encoder, a fast search algorithm is applied to find the weight index without significantly increasing the encoder complexity. These algorithms are summarized below. Reference can be made to the VTM software and document JVET-L0646 for further details. When combined with AMVR, the unequal weights of 1-pixel and 4-pixel motion vector accuracy are only conditionally checked if the current picture is a low-latency picture. - When combined with affine, affine ME will be performed for unequal weights if and only if the affine mode is selected as the current best mode. - Unequal weights are only conditionally checked when the two reference pictures in bidirectional prediction are the same. - Unequal weights are not searched when certain conditions are met, which depend on the POC distance between the current picture and its reference pictures, the codec QP, and the temporal level. The BCW weight index is encoded using one context encoded binary followed by a bypass encoded binary. The first context encoded binary indicates whether equal weights are used; and if unequal weights are used, an additional binary is signaled using the bypass codec to indicate which unequal weights are used. Weighted prediction (WP) is a codec tool supported by the H.264 / AVC and HEVC standards for efficient encoding and decoding of video content in fading conditions. Support for WP has also been added to the VVC standard. WP allows weighting parameters (weights and offsets) to be transmitted by signal for each reference picture list L0 and each reference picture in list L1. Then, during motion compensation, the weights and offsets of the corresponding reference pictures are applied. WP and BCW are designed for different types of video content. In order to avoid interaction between WP and BCW (which will complicate the VVC decoder design), if the CU uses WP, the BCW weight index is not transmitted by signal, and w is inferred to be 4 (ie, equal weights are applied). For the merged CU, the weight index is inferred from the neighboring blocks based on the merge candidate index. This can be applied to the normal merge mode and the inherited affine merge mode. For the constructed affine merge mode, affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index of the CU using the constructed affine merge mode is simply set equal to the BCW index of the first control point MV. In VVC, CIIP and BCW cannot be applied jointly to a CU. When a CU is encoded and decoded using CIIP mode, the BCW index of the current CU is set to 2, for example, equal weight. 2.13. Local Illumination Compensation (LIC) Local illumination compensation (LIC) is a codec tool that addresses the problem of local illumination variations between a current picture and its temporal reference picture. LIC is based on a linear model where a scaling factor and an offset are applied to a reference sample to obtain a predicted sample for the current block. Specifically, LIC can be mathematically modeled by the following equation: P(x, y) = α·P r (x+v x ,y+v y )+β Where P(x, y) is the prediction signal of the current block at the coordinate (x, y); P r (x+v x ,y+v y ) is the motion vector (v x , v y ) points to the reference block; and α and β are the corresponding scaling factors and offsets applied to the reference block. Fig.19 The LIC process is shown. Fig.19 In , when applying LIC to a block, the minimum mean square error (LMSE) method is used to minimize the neighboring samples of the current block (i.e. Fig.19 The template T in the temporal reference picture) and its corresponding reference sample in the temporal reference picture (ie, Fig.19 In addition, in order to reduce the computational complexity, both the template samples and the reference template samples are subsampled (adaptive subsampling) to derive the LIC parameters, that is, only Fig.19 The shadow samples in are used to derive α and β. To improve the codec performance, the short edge subsampling is not performed, such as Fig. 20 shown. 2.14. Decoder-side Motion Vector Refinement (DMVR) To improve the accuracy of merge mode MV, decoder-side motion vector refinement based on bilateral matching (BM) is applied in VVC. In bidirectional prediction operation, the refined MV is searched around the initial MV in reference picture list L0 and reference picture list L1. The BM method calculates the distortion between two candidate blocks in reference picture list L0 and list L1. Fig.21 FIG. 4 shows a schematic diagram of the refinement of the motion vector at the decoding side. Fig.21 As shown, based on each MV candidate around the initial MV, the SAD between block 2110 and block 2112 is calculated, where for the current picture 2102, block 2110 is in the reference picture 2101 in list L0, and block 2112 is in the reference picture 2103 in list L1. The MV candidate with the lowest SAD becomes the refined MV and is used to generate a bidirectional prediction signal. In VVC, the application of DMVR is limited and only applies to CUs with the following modes and capabilities: - CU level merge mode with bi-predictive MV. -Relative to the current picture, one reference picture is in the past and the other reference picture is in the future. - The distances from the two reference pictures to the current picture (ie, the POC difference) are the same. - Both reference images are short-term reference images. -CU has more than 64 luma samples. - CU height and CU width are both greater than or equal to 8 luma samples. -BCW weight index indicates equal weight. -Currently the block is not WP enabled. -CIIP mode is not used for the current block. The refined MV derived from the DMVR process is used to generate inter-frame prediction samples and is also used for temporal motion vector prediction for future picture encoding and decoding. The original MV is used in the deblocking process and is also used for spatial motion vector prediction for future CU encoding and decoding. Additional features of DMVR are mentioned in the following sub-clauses. 2.14.1. Search solution In DVMR, the search point is around the initial MV, and the MV offset obeys the MV difference mirror rule. In other words, any point examined by DMVR represented by the candidate MV pair (MV0, MV1) follows the following two equations: MV0′=MV0+MV_offset (2-19) MV1′=MV1-MV_offset (2-20) Wherein, MV_offset represents the refinement offset between the initial MV and the refinement MV in one of the reference pictures. The refinement search range is two integer luma samples starting from the initial MV. The search includes an integer sample offset search phase and a fractional sample refinement phase. The integer sample offset search uses a 25-point full search. First, the SAD of the initial MV pair is calculated. If the SAD of the initial MV pair is less than the threshold, the integer sampling stage of DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search stage. In order to reduce the impact of DMVR refinement uncertainty, it is proposed to support the original MV in the DMVR process. The SAD between the reference blocks referenced by the initial MV candidates is reduced by 1 / 4 of the SAD value. The integer sample search is followed by fractional sample refinement. To save computational complexity, fractional sample refinement is derived using the parametric error surface equations instead of doing an additional search using SAD comparisons. Fractional sample refinement is conditionally called based on the output of the integer sample search stage. When the integer sample search stage ends with a center with minimum SAD in either the first or second iteration search, fractional sample refinement is further applied. In the sub-pixel offset estimation based on the parameter error surface, the cost of the center position and the costs of the four neighboring positions from the center are used to fit a two-dimensional parabolic error surface equation of the following form E(x, y) = A(x min ) 2 +B(yy min ) 2 +C (2-21) Where (x min ,y min ) corresponds to the fractional position with the minimum cost, and C corresponds to the minimum cost value. By solving the above equation using the cost values ​​of the five search points, (x min ,y min ) is calculated as: x min =(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0))) (2-22) y min =(E(0,-1)-E(0,1)) / (2((E(0,-1)+E(0,1)-2E(0,0))) (2-23). x min and min The value of is automatically clamped between -8 and 8, since all cost values ​​are positive, with a minimum value of E(0,0). This corresponds to a half-pixel offset with 1 / 16-pixel MV precision in VVC. The calculated fraction (x min ,y min ) is added to the integer distance refinement MV to obtain a sub-pixel accurate refinement delta MV. 2.14.2. Linear interpolation and sample padding In VVC, the resolution of MV is 1 / 16 luma sample. Samples at fractional positions are interpolated using an 8-tap interpolation filter. In DMVR, the search points are around the initial fractional pixel MV with integer sample offsets, so the samples at these fractional positions need to be interpolated for the DMVR search process. In order to reduce the computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in DMVR. Another important effect is that by using a bilinear filter, DVMR does not access more reference samples than the normal motion compensation process within the 2-sample search range. After the refined MV is obtained through the DMVR search process, an ordinary 8-tap interpolation filter is applied to generate the final prediction. In order not to access more reference samples for the normal MC process, samples will be filled from those available samples, which are not needed for the interpolation process based on the original MV, but are needed for the interpolation process based on the refined MV. 2.14.3. Maximum DMVR processing unit When the width and / or height of a CU is greater than 16 luma samples, it will be further split into sub-blocks with width and / or height equal to 16 luma samples. The maximum unit size for the DMVR search process is limited to 16x16. 2.15. Multi-pass motion vector refinement In this contribution, multi-pass decoder-side motion vector refinement is applied instead of DMVR. In the first pass, bilateral matching (BM) is applied to one codec block. In the second pass, BM is applied to each 16x16 sub-block within the codec block. In the third pass, MVs in each 8x8 sub-block are refined by applying bidirectional optical flow (BDOF). The refined MVs are stored for spatial and temporal motion vector prediction. 2.15.1. First pass – Block-based bilateral matching MV refinement In the first pass, refined MVs are derived by applying BM to the codec blocks. Similar to decoder-side motion vector refinement (DMVR), refined MVs are searched around two initial MVs (MV0 and MV1) in reference picture lists L0 and L1. Refined MVs (MV0_pass1 and MV1_pass1) are derived around the initiating MVs based on the minimum bilateral matching cost between the two reference blocks in L0 and L1. BM performs a local search to derive integer sample precision intDeltaMV and half-pel sample precision halfDeltaMV. The local search applies a 3×3 square search pattern, looping in the horizontal search range [–sHor, sHor] and the vertical search range [–sVer, sVer], where the values ​​of sHor and sVer are determined by the block scale, and the maximum values ​​of sHor and sVer are 8. The bilateral matching cost is calculated as: bilCost = mvDistanceCost + sadCost. When the block size cbW*cbH is greater than 64, the MRSAD cost function is applied to remove the DC effect of the distortion between the reference blocks. When the bilCost of the center point of the 3×3 search pattern has the minimum cost, the intDeltaMV or halfDeltaMV local search terminates. Otherwise, the current minimum cost search point becomes the new center point of the 3×3 search pattern and continues to search for the minimum cost until it reaches the end of the search range. The existing fractional sample refinement is further applied to derive the final deltaMV. Then, the refined MV after the first pass is exported as: MV0_pass1=MV0+deltaMV, MV1_pass1=MV1–deltaMV. 2.15.2. Second pass - sub-block based bilateral matching MV refinement In the second pass, refined MVs are derived by applying BM to 16×16 grid sub-blocks. For each sub-block, the refined MVs are searched around the two MVs (MV0_pass1 and MV1_pass1) obtained in the first pass in the reference picture lists L0 and L1. The refined MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the two reference sub-blocks in L0 and L1. For each subblock, BM performs a full search to derive integer sample precision intDeltaMV. The full search has a search range of [–sHor, sHor] in the horizontal direction and a search range of [–sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block scale and the maximum values ​​of sHor and sVert are 8. The bilateral matching cost is calculated by applying the cost factor to the SATD cost between two reference sub-blocks, such as: bilCost = satdCost * costFactor. The search area (2*sHor+1)*(2*sVer+1) is divided into 5 diamond search areas, such as Fig. 222200. Each search region is assigned a costFactor determined by the distance (intDeltaMV) between each search point and the starting MV, and each diamond region is processed in order starting from the center of the search region. In each region, the search points are processed in raster scan order, starting from the upper left corner of the region to the lower right corner. When the minimum bilCost in the current search region is less than or equal to the threshold of sbW*sbH, the integer pixel (int-pel) full search is terminated, otherwise, the int-pel full search continues to the next search region until all search points are checked. BM performs a local search to derive the half-sample accuracy halfDeltaMv. The search pattern and cost function are the same as defined in 2.9.1. The existing VVC DMVR fractional sample refinement is further applied to derive the final deltaMV (sbIdx2). Then, the refined MV of the second pass is derived as: ·MV0_pass2(sbIdx2)=MV0_pass1+deltaMV(sbIdx2), ·MV1_pass2(sbIdx2)=MV1_pass1-deltaMV(sbIdx2). 2.15.3. The third pass - sub-block-based bidirectional optical flow MV refinement In the third pass, refined MVs are derived by applying BDOF to 8×8 grid sub-blocks. For each 8×8 sub-block, BDOF refinement is applied to derive scaled Vx and Vy without clipping from the refined MV of the parent sub-block in the second pass. The derived bioMv(Vx, Vy) is rounded to 1 / 16 sample precision and clipped between -32 and 32. The refined MVs of the third pass (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) are derived as: ·MV0_pass3(sbIdx3)=MV0_pass2(sbIdx2)+bioMv, ·MV1_pass3(sbIdx3)=MV0_pass2(sbIdx2)–BioMv. 2.16. Sample-based BDOF In sample-based BDOF, instead of deriving motion refinement (Vx, Vy) on a block basis, it is performed per sample. The codec block is divided into 8×8 sub-blocks. For each sub-block, whether to apply BDOF is determined by checking the SAD between the two reference sub-blocks against a threshold. If it is decided to apply BDOF to a sub-block, for each sample in the sub-block, Vx and Vy are derived using a sliding 5×5 window and applying the existing BDOF process for each sliding window. The derived motion refinement (Vx, Vy) is applied to adjust the bi-predicted sample value of the window center sample. 2.17. Extended Merge Prediction In VVC, the merge candidate list is constructed by including the following five types of candidates in order: (1) Spatial MVP from spatially adjacent CUs. (2) Temporal MVP from the co-located CU. (3) History-based MVP from FIFO table. (4) Paired average MVP. (5) Zero MV. The size of the merge list is signaled in the sequence parameter set header, and the maximum allowed size of the merge list is 6. For each CU code in merge mode, the index of the best merge candidate is encoded and decoded using truncated unary binarization (TU). The first bin of the merge index is encoded and decoded using context, while bypass encoding and decoding is used for other bins. The derivation process of merge candidates for each category is provided in this section. As in HEVC, VVC also supports parallel derivation of merge candidate lists for all CUs within a certain size region. 2.17.1. Spatial selection derivation The derivation of spatial merge candidates in VVC is the same as that in HEVC, except that the positions of the first two merge candidates are swapped. Fig.23 is a schematic diagram 2300 showing the locations of spatial merging candidates. Fig.23 Among the candidates at the positions shown, up to four merge candidates are selected. The derivation order is B0, A0, B1, A1 and B2. Position B2 is considered only when one or more CUs at positions B0, A0, B1 and A1 are not available (for example, because it belongs to another slice or tile) or is intra-coded. After the candidate at position A1 is added, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving the coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Fig.24 is a diagram 2400 showing candidate pairs considered for redundancy check of spatial merging candidates. Fig.24The pairs linked with arrows in , and candidates are added to the list only if the corresponding candidates for redundancy check do not have the same motion information. 2.17.2. Time domain selection derivation In this step, only one candidate is added to the list. In particular, in the derivation of this temporal merge candidate, the scaled motion vector is derived based on the co-located CU belonging to the co-located reference picture. The reference picture list to be used for deriving the co-located CU is explicitly signaled in the slice header. Fig.25 As shown by the dashed line in diagram 2500 of , a scaled motion vector for the temporal merge candidate is obtained, which is scaled from the motion vector of the co-located CU using the POC distances, tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero. Fig.26 2600 is a schematic diagram showing candidate positions for time domain merging candidates C0 and C1. Fig.26 As shown, the position of the temporal candidate is selected between candidates C0 and C1. If the CU at position C0 is not available and is intra-coded, or the CU at position C0 is outside the current row of the CTU, position C1 is used. Otherwise, position C0 is used in the derivation of the temporal merge candidate. 2.17.3. History-based Merge Candidate Derivation History-based MVP (HMVP) merge candidates are added to the merge list after spatial MVP and TMVP. In this method, the motion information of the previous codec block is stored in a table and used as the MVP of the current CU. A table with multiple HMVP candidates is maintained during the encoding / decoding process. The table is reset (cleared) when a new CTU row is encountered. Whenever there is a non-sub-block inter-coded CU, the associated motion information is added as a new HMVP candidate to the last entry of the table. The HMVP table size S is set to 6, which indicates that up to 6 history-based MVP (HMVP) candidates can be added to the table. When a new motion candidate is inserted into the table, a constrained first-in, first-out (FIFO) rule is used, where a redundancy check is first applied to find if there is an identical HMVP in the table. If found, the identical HMVP is removed from the table, and then all HMVP candidates are moved forward, and the identical HMVP is inserted into the last entry of the table. HMVP candidates can be used in the merge candidate list construction process. The nearest HMVP candidates in the table are checked in order and inserted into the candidate list after the TMVP candidate. Redundancy check is applied to HMVP candidates to spatial or temporal merge candidates. To reduce the number of redundant checking operations, the following simplifications are introduced: The number of HMPV candidates for merge list generation is set to (N<=4) × M:(8-N), where N indicates the number of existing candidates in the merge list and M indicates the number of available HMVP candidates in the table. Once the total number of available merge candidates reaches the maximum allowed merge candidates minus 1, the merge candidate list construction process from HMVP is terminated. 2.17.4. Pairwise Average Merge Candidate Derivation The pairwise average candidate is generated by averaging the predefined candidate pairs in the existing merge candidate list, and the predefined pairs are defined as {(0, 1), (0, 2), (1, 2), (0, 3), (1, 3), (2, 3)}, where the number represents the merge index of the merge candidate list. The average motion vector is calculated separately for each reference list. If two motion vectors are available in one list, they are averaged even if they point to different reference pictures; if only one motion vector is available, it is used directly; if no motion vector is available, this list is kept invalid. When the merge list is not full after adding pairwise average merge candidates, zero MVPs will be inserted at the end until the maximum number of merge candidates is encountered. 2.17.5. Merging Estimated Regions The merge estimation region (MER) allows the merge candidate list to be derived independently for a CU in the same merge estimation region (MER). For generating the merge candidate list for the current CU, candidate blocks in the same MER as the current CU are not included. In addition, the update process for the history-based motion vector predictor candidate list is updated only when (xCb+cbWidth)>>Log2ParMrgLevel is greater than xCb>>Log2Par MrgLevel and (yCb+cbHeight)>>Log2parMrglevel is greater than (yCb>>Log2ParMrgLevel), and where (xCb, yCb) is the top left luma sample position of the current CU in the picture and (cbWidth, cbHeight) is the CU size. The MER size is selected at the encoder side and signaled in the sequence parameter set as log2_parallel_merge_level_minus2. 2.18. New Merge Candidates 2.18.1. Non-adjacent merge candidate derivation Fig. 27 A schematic diagram 2700 of VVC spatial neighboring blocks of a current block is shown. In VVC, Fig. 27The five spatial neighboring blocks and one temporal neighboring point shown are used to derive the merge candidate. It is proposed to derive additional merge candidates from non-adjacent locations to the current block using the same pattern as VVC. To this end, for each search round i, a virtual block is generated based on the current block as follows: First, the relative position of the virtual block to the current block is calculated as follows: Offsetx=-i×gridX, Offsety=-i×gridY Where Offsetx and Offsety represent the offset of the upper left corner of the virtual block relative to the upper left corner of the current block, and gridX and gridY are the width and height of the search grid. Second, the width and height of the virtual block are calculated in the following way: newWidth=i×2×gridX+currWidth newHeight=i×2×gridY+currHeight. Where currWidth and currHeight are the width and height of the current block. newWidth and newHeight are the width and height of the new virtual block. gridX and gridY are currently set to currWidth and currHeight respectively. Fig.28 A schematic diagram of a virtual block in the i-th round of search is shown, which shows the relationship between the virtual block and the current block. After generating the virtual block, block A i , B i , C i , D i and E i can be considered as VVC spatial neighbors of the virtual block, and their positions are obtained in the same way as VVC. Obviously, if the search round i is 0, the virtual block is the current block. In this case, block A i , B i , C i , D i and E i It is the spatial neighboring block used in VVC merge mode. When constructing the merge candidate list, pruning is performed to ensure the uniqueness of each element in the merge candidate list. The maximum search round is set to 1, which means that five non-adjacent spatial neighboring blocks are utilized. Non-adjacent spatial merging candidates are inserted into the merging list after the temporal merging candidates in the order of B1->A1->C1->D1->E1. 2.18.2.STMVP It is proposed to use three spatial merging candidates and one temporal merging candidate to derive an average candidate as the STMVP candidate. The STMVP is inserted before the upper left spatial merge candidate. The STMVP candidate is pruned along with all previous merge candidates in the merge list. For spatial candidates, the first three candidates in the current merged candidate list are used. For the temporal candidates, the same positions as the VTM / HEVC co-location positions are used. For spatial candidates, the first, second, and third candidates inserted into the current merge candidate list before STMVP are denoted as F, S, and T. The temporal candidate having the same position as the VTM / HEVC co-location position used in TMVP is denoted Col. The motion vector of the STMVP candidate in prediction direction X (denoted as mvLX) is derived as follows: 1) If the reference indices of the four merge candidates are all valid and equal to zero in the prediction direction X (X=0 or 1), mvLX=(mvLX_F+mvLX_S+mvLX_T+mvLX_Col)>>2. 2) If the reference indexes of three of the four merge candidates are valid and equal to zero in the prediction direction X (X=0 or 1), mvLX=(mvLX_F×3+mvLX_S×3+mvLX_Col×2)>>3 or mvLX=(mvLX_F×3+mvLX_T×3+mvLX_Col×2)>>3 or mvLX=(mvLX_S×3+mvLX_T×3+mvLX_Col×2)>>3. 3) If the reference indexes of two of the four merge candidates are valid and equal to zero in the prediction direction X (X=0 or 1), mvLX=(mvLX_F+mvLX_Col)>>1 or mvLX=(mvLX_S+mvLX_Col)>>1 or mvLX=(mvLX_T+mvLX_Col)>>1. NOTE: If time domain candidates are not available, STMVP mode is turned off. 2.18.3 Merge List Size If both non-adjacent merge candidates and STMVP merge candidates are considered, the size of the merge list is signaled in a set of sequence parameter headers, and the maximum allowed size of the merge list is 8. 2.19. Geometric Partitioning Model (GPM) In VVC, geometric partitioning mode is supported for inter prediction. The geometric partitioning mode is signaled using a CU level flag as a merge mode. Other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. For each possible CU size w×h=2 m ×2 n , where m,n∈{3…6} and excluding 8x64 and 64x8, the geometric partitioning mode supports a total of 64 partitions. When this mode is used, the CU is split into two parts by a geometrically positioned straight line (e.g. Fig. 20 ). The position of the partition line is mathematically derived from the angle and offset parameters of the specific partition. Each part of the geometric partition in the CU uses its own motion for inter-frame prediction; each partition only allows unidirectional prediction, that is, each part has a motion vector and a reference index. Unidirectional prediction motion constraints are applied to ensure that, as with traditional bidirectional prediction, only two motion compensated predictions are required for each CU. The unidirectional prediction motion for each partition is derived using the process described in 2.19.1. If a geometric partitioning mode is used for the current CU, a geometric partitioning index indicating the partitioning mode (angle and offset) of the geometric partitioning and two merge indices (one for each partition) are further transmitted through a signal. The number of maximum GPM candidate sizes is explicitly transmitted through a signal in the SPS, and the syntax binarization for the GPM merge indices is specified. After predicting each part of the geometric partitioning, a hybrid process with adaptive weights is used as in 2.19.2 to adjust the sample values ​​along the edges of the geometric partitioning. This is the prediction signal for the entire CU, and the transformation and quantization processes will be applied to the entire CU as in other prediction modes. Finally, the motion field of the CU predicted using the geometric partitioning mode is stored as in 2.19.3. 2.19.1. One-way prediction candidate list construction The unidirectional prediction candidate list is directly derived from the merge candidate list constructed according to the extended merge prediction process in 2.17. Denote n as the index of the unidirectional prediction motion in the geometric unidirectional prediction candidate list. The LX motion vector of the nth extended merge candidate is used as the nth unidirectional prediction motion vector of the geometric partitioning mode, with X equal to the parity of n. Fig.30 The unidirectional prediction MV selection for the geometric partitioning mode is shown. These motion vectors are Fig.30 If the corresponding LX motion vector of the n-th extended merge candidate does not exist, the L(1-X) motion vector of the same candidate is used instead as the unidirectional prediction motion vector of the geometric partitioning mode. 2.19.2. Blending Along Geometric Partition Edges After predicting each part of the geometric partition using its own motion, blending is applied to the two prediction signals to derive samples around the geometric partition edges. The blending weights for each position of the CU are derived based on the distance between the individual position and the partition edge. The distance from position (x, y) to the partition edge is derived as: where i, j are the angle and offset indices of the geometric partition, which depend on the geometric partition index transmitted by the signal. x，j and ρ y，j The sign of depends on the angle index i. The weight of each part of the geometric partition is derived as follows: wIdxL(x,y)=partIdx? 32+d(x,y):32-d(x,y) (2-28) w1(x,y)=1-w o (x, y) (2-30) partIdx depends on the angle index i. Fig.31 Schematic generation of warp weights W0 using a geometric partitioning pattern is shown. An example of weight w0 is Fig.31 is shown in the figure. 2.19.3. Motion Field Storage for Geometry Partitioning Patterns Mv1 from the first part of the geometric partitioning, Mv2 from the second part of the geometric partitioning, and the combination Mv of Mv1 and Mv2 are stored in the motion field of the CU encoded in the geometric partitioning mode. The type of motion vector stored for each individual position in the motion field is determined as: sType=abs(motionIdx)<32?2: (motionIdx≤0?(1-partIdx):partIdx) (2-31) Where motionIdx is equal to d(4x+2, 4y+2), which is recalculated according to equation (2-18). partIdx depends on the angle index i. If sType is equal to 0 or 1, then Mv0 or Mv1 is stored in the corresponding motion field, otherwise, if sTType is equal to 2, the combined Mv from Mv0 and Mv2 is stored. The combined Mv is generated using the following process: 1) If Mv1 and Mv2 come from different reference picture lists (one from L0 and the other from L1), Then Mv1 and Mv2 are simply combined to form a bidirectional prediction motion vector. Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored. 2.20. Multi-hypothesis prediction In Multiple Hypothesis Prediction (MHP), up to two additional predictors are signaled on top of Inter AMVP mode, Normal Merge mode, Affine Merge mode and MMVD mode. The resulting overall prediction signal is iteratively accumulated with each additional prediction signal. p n+1 =(1-α n+1 ) n +α n+1 h n+1 The weight factor α is specified according to the following Table 2-4: Table 2-4 - Weighting factors for MHP add_hyp_weight_idx α 0 1 / 4 1 -1 / 8 For inter-AMVP mode, MHP is applied only when unequal weights in BCW are selected in bi-prediction mode. Additional assumptions may be merge mode or AMVP mode. In the case of merge mode, motion information is indicated by a merge index, and the merge candidate list is the same as in the geometry partition mode. In the case of AMVP mode, reference index, MVP index, and MVD are transmitted by signal. 2.21. Non-adjacent spatial domain candidates Non-adjacent spatial merging candidates are inserted after TMVP in the regular merging candidate list. Fig.32 The spatial neighboring blocks used to derive the spatial merge candidate are shown in FIG. Fig.32 As shown in . The distance between the non-adjacent spatial candidates and the current codec block is based on the width and height of the current codec block. 2.22. Template Matching (TM) Template Matching (TM) is a decoder-side MV derivation method used to refine the motion information of the current CU by finding the closest match between a template in the current picture (i.e., the top and / or left neighboring blocks of the current CU) and a block in a reference picture (i.e., the same size as the template). Fig.33 It is shown that template matching is performed on the search area around the initial MV. Fig.33 As shown, in the [-8, +8] pixel search range, a better MV is searched around the initial motion of the current CU. This contribution adopts the template matching previously proposed in JVET-J0021 and makes two modifications: the search step size is determined based on the AMVR mode, and the TM can cascade the bilateral matching process in the merge mode. In AMVP mode, MVP candidates are determined based on template matching error to select the one that achieves the minimum difference between the current block template and the reference block template, and then TM performs MV refinement only on this specific MVP candidate. TM refines this MVP candidate by using an iterative diamond search, starting with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode) within the [-8, +8] pixel search range. The AMVP candidate can be further refined by using a cross search with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode), and then using half-pixels and quarter-pixels in turn according to the AMVR mode specified in Table 2-5. This search process ensures that the MVP candidate still maintains the same MV accuracy as indicated by the AMVR mode after the TM process. Table 2-5 – Search modes of AMVR and merge modes with AMVR In merge mode, a similar search method is applied to the merge candidates indicated by the merge index. As shown in Table 2-5, TM can be performed all the way down to 1 / 8 pixel MVD accuracy, or skip accuracy beyond half-pixel MVD accuracy, depending on whether an alternative interpolation filter is used based on the merged motion information (used when AMVR is half-pixel mode). In addition, when TM mode is enabled, template matching can work as an independent process between block-based and sub-block-based bilateral matching (BM) methods or as an additional MV refinement process, depending on whether BM can be enabled based on its enabling condition check. 2.23. Overlapped Block Motion Compensation (OBMC) Overlapped Block Motion Compensation (OBMC) has been used previously in H.263. In JEM, unlike H.263, OBMC can be turned on and off using CU-level syntax. When OBMC is used in JEM, OBMC is performed on all motion compensated (MC) block boundaries except the right and bottom boundaries of the CU. In addition, it also applies to luminance and chrominance components. In JEM, an MC block corresponds to a codec block. When a CU is encoded and decoded with sub-CU modes (including sub-CU merging, affine, and FRUC modes), each sub-block of the CU is an MC block. Fig.34 is an illustration of the sub-blocks to which OBMC is applied. In order to handle CU boundaries in a unified manner, OBMC is performed on all MC block boundaries at the sub-block level, where the sub-block size is set equal to 4×4, as shown in Fig.34 shown. When OBMC is applied to the current sub-block, in addition to the current motion vector, the motion vectors of the four connected neighboring sub-blocks, if available and different from the current motion vector, are also used to derive the prediction block of the current sub-block. These multiple prediction blocks based on multiple motion vectors are combined to generate the final prediction signal of the current sub-block. The prediction block based on the motion vector of the neighboring sub-block is represented as P N , where N indicates the index of the neighboring upper, lower, left, and right sub-blocks, and the prediction block based on the motion vector of the current sub-block is represented as P C When P N When based on the motion information of the neighboring sub-block containing the same motion information as the current sub-block, OBMC is not N Otherwise, P N Each sample of is added to P C The same sample in P N Four rows / columns are added to P C The weighting factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} are used for P N , and weighting factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are used for P C . Except for small MC blocks (i.e., when the height or width of the codec block is equal to 4, or when the CU is encoded and decoded in sub-CU mode), there are only two rows / columns of P N Added to P C In this case, weighting factors {1 / 4, 1 / 8} are used for P N , and the weighting factors {3 / 4, 7 / 8} are used for P C For P generated based on the motion vectors of vertically (horizontally) adjacent sub-blocks N , P N Samples in the same row (column) of are added to P with the same weighting factor C . In JEM, for CUs with a size less than or equal to 256 luma samples, a CU-level flag is signaled to indicate whether OBMC is applied to the current CU. For CUs with a size greater than 256 luma samples or not encoded and decoded using AMVP mode, OBMC is applied by default. At the encoder, when OBMC is applied to a CU, its impact is taken into account in the motion estimation stage. The prediction signal formed by OBMC using the motion information of the top neighboring blocks and the left neighboring blocks is used to compensate the top and left boundaries of the original signal of the current CU, and then the normal motion estimation process is applied. 2.24. Multiple Transform Selection (MTS) for Core Transforms In addition to DCT-II already adopted in HEVC, the Multiple Transform Selection (MTS) scheme is also used for residual coding of inter-frame and intra-frame coded blocks. It uses multiple selected transforms in DCT8 / DST7. The newly introduced transform matrices are DST-VII and DCT-VIII. Table 2-6 shows the basic functions of the selected DST / DCT. Table 2-6 – Transform basis functions of DCT-II / VIII and DSTVII for N-point input To maintain the orthogonality of the transform matrix, the quantization of the transform matrix is ​​more accurate than that in HEVC. To keep the intermediate values ​​of the transform coefficients within 16 bits, all coefficients are 10 bits after horizontal and vertical transforms. To control the MTS scheme, separate enable flags are specified at the SPS level for intra and inter frames respectively. When MTS is enabled at SPS, a CU level flag signal is issued to indicate whether MTS is applied. Here, MTS is only applied to luma. When one of the following conditions applies, MTS signaling will be skipped. - The position of the last significant coefficient of the luma TB is less than 1 (ie DC only). -The last significant coefficient of the luminance TB is located within the MTS zeroing region. If the MTS CU flag is equal to 0, DCT2 is applied in both directions. However, if the MTS CU flag is equal to 1, two other flags are additionally signaled to indicate the transform types in the horizontal and vertical directions, respectively. The transform and signaling mapping table is shown in Table 2-7. The transform selection for ISP and implicit MTS is unified by eliminating intra-frame mode and block shape dependencies. If the current block is in ISP mode or the current block is an intra-frame block and both intra-frame and inter-frame explicit MTS are turned on, only DST7 is used for horizontal and vertical transform cores. In terms of transform matrix accuracy, an 8-bit main transform core is used. Therefore, all transform cores used in HEVC remain the same, including 4-point DCT-2 and DST-7, 8-point, 16-point and 32-point DCT-2. In addition, other transform cores (including 64-point DCT-2, 4-point DCT-8, 8-point, 16-point, 32-point DST-7 and DCT-8) all use 8-bit main transform cores. Table 2-7 – Transformation and signaling mapping table To reduce the complexity of large-size DST-7 and DCT-8, high-frequency transform coefficients are cleared to zero for DST-7 and DCT-8 blocks with size (width or height, or width and height) equal to 32. Only coefficients in the 16×16 low-frequency region are retained. As in HEVC, the residual of a block can be coded using transform skip mode. To avoid syntax coding redundancy, the transform skip flag is not signaled when the CU level MTS_CU_flag is not equal to 0. Note that when LFNST or MIP is activated for the current CU, the implicit MTS transform is set to DCT2. In addition, when MTS is enabled for an inter-coded block, implicit MTS can still be enabled. 2.25. Sub-Block Transform (SBT) In VTM, sub-block transform is introduced for inter-predicted CU. In this transform mode, only a sub-part of the residual block is encoded and decoded for the CU. When cu_cbf of an inter-predicted CU is equal to 1, cu_sbt_flag can be signaled to indicate whether the entire residual block or a sub-part of the residual block is encoded and decoded. For the former case, the inter-MTS information is further parsed to determine the transform type of the CU. In the latter case, a part of the residual block is encoded and decoded by inferring the adaptive transform, while the other part of the residual block is cleared. When SBT is used for an inter-coded CU, the SBT type and SBT position information are signaled in the bitstream. There are two SBT types and two SBT positions, such as Fig.35 As shown. For SBT-V (or SBT-H), the TU width (or height) can be equal to half of the CU width (or height) or 1 / 4 of the CU width (or height), resulting in 2:2 partitioning or 1:3 / 3:1 partitioning. 2:2 partitioning is similar to binary tree (BT) partitioning, while 1:3 / 3:1 partitioning is similar to asymmetric binary tree (ABT) partitioning. In ABT splitting, only small areas contain non-zero residuals. If one dimension of the CU is 8 (in units of luminance samples), 1:3 / 3:1 partitioning along that dimension is not allowed. A CU has a maximum of 8 SBT modes. Position-dependent transform kernel selection is applied to luma transform blocks in SBT-V and SBT-H (chroma TBs always use DCT-2). The two positions, SBT-H and SBT-V, are associated with different kernel transforms. Fig.35 The SBT position, type and transform type are shown. More specifically, the horizontal and vertical transforms of each SBT position are shown in Fig.35 For example, the horizontal and vertical transforms for SBT-V position 0 are DCT-8 and DST-7, respectively. When one side of the residual TU is larger than 32, the transforms for both dimensions are set to DCT-2. Therefore, the sub-block transforms jointly specify the TU tiling, cbf, and the horizontal and vertical core transform types for the residual block. SBT is not applied to CUs encoded in inter-intra combination mode. 2.26. Adaptive Merge Candidate Reordering Based on Template Matching In order to improve the encoding and decoding efficiency, after building the merge candidate list, the order of each merge candidate is adjusted according to the template matching cost. The merge candidates are arranged in the list according to the template matching cost in ascending order. It operates in the form of subgroups. The template matching cost is measured by the SAD (sum of absolute differences) between the neighboring samples of the current CU and its corresponding reference samples. Fig.36 The neighboring samples used to calculate the SAD are shown. If the merge candidate includes bidirectional prediction motion information, the corresponding reference sample is the average of the corresponding reference sample in reference list 0 and the corresponding reference sample in reference list 1, such as Fig.36 37 shows the neighboring samples used to calculate the SAD for the sub-CU level motion information. If the merge candidate includes the sub-CU level motion information, the corresponding reference sample is composed of the neighboring samples of the corresponding reference sub-block, such as Fig.37 shown. The sorting process is operated in the form of subgroups, such as Fig.38 As shown in . The first three merge candidates are sorted together. The following three merge candidates are sorted together. Fig.38 As shown, the original merge candidate list 3810 is sorted to obtain the updated merge candidate list 380. In this example, the template size (the width of the left template or the height of the upper template) is 1, and the subgroup size is 3. 2.27. Adaptive Merge Candidate List Assume that the number of merging candidates is 8. The first 5 merging candidates are used as the first subgroup, and the following 3 merging candidates are used as the second subgroup (ie, the last subgroup). Fig.39 3900 is a flowchart of a reordering process in an encoder. For the encoder, after the merge candidate list is constructed at block 3902, some merge candidates are adaptively reordered in ascending order of merge candidate cost, such as Fig.39 shown. More specifically, in box 3904, the template matching costs of the merge candidates in all subgroups except the last subgroup are calculated; then in box 3906, the merge candidates in its own subgroup except the last subgroup are reordered; finally, in box 3908, a final merge candidate list is obtained. For the decoder, after constructing the merge candidate list, some / no merge candidates are adaptively reordered in ascending order of merge candidate cost, e.g. Fig.40 As shown, Fig.40 4000 is a flowchart of a reordering process in a decoder. Fig.40 In , the subgroup where the selected (transmitted by signal) merging candidate belongs is called the selected subgroup. More specifically, at block 4002, it is determined whether the selected merge candidate is located in the last subgroup. If the selected merge candidate is located in the last subgroup, then at block 4004, the merge candidate list construction process is terminated after the selected merge candidate is derived, and at block 4006, no reordering is performed and the merge candidate list is not changed; otherwise, the execution process is as follows: In box 4008, after all merge candidates in the selected subgroup are exported, the merge candidate list construction process is terminated; in box 4010, the template matching cost of the merge candidates in the selected subgroup is calculated; in box 4012, the merge candidates in the selected subgroup are reordered; finally, in box 4014, a new merge candidate list is obtained. For both the encoder and the decoder, the template matching cost is derived as a function of T and RT, where T is the set of samples in the template and RT is the set of reference samples for the template. When deriving the reference samples of the template of the merge candidate, the motion vector of the merge candidate is rounded to integer pixel precision. The reference samples of the template (RT) for bidirectional prediction are derived by weighted averaging the reference samples of the template in reference list 0 (RT0) and the reference samples of the template in reference list 1 (RT1) as follows. RT=((8-w)*RT0+w*RT1+4)>>3 (2-32) The weights of the reference templates in reference list 0 (8-w) and the weights of the reference templates in reference list 1 (w) are determined by the BCW index of the merge candidate. BCW indexes equal to {0, 1, 2, 3, 4} correspond to w equal to {-2, 3, 4, 5, 10} respectively. If the local illumination compensation (LIC) flag of the merge candidate is true, the reference samples of the template are derived using the LIC method. The template matching cost is calculated based on the sum of absolute differences (SAD) between T and RT. The template size is 1. This means that the width of the left template and / or the height of the top template is 1. If the codec mode is MMVD, the merge candidates used to derive the base merge candidate are not reordered. If the coding mode is GPM, the merge candidates used to derive the unidirectional prediction candidate list are not reordered. 2.28. Geometric prediction mode with motion vector differences In geometric prediction mode with motion vector differencing, each geometric partition in GPM can decide whether to use GMVD. If GMVD is selected for a geometric region, the MV of the region is calculated as the sum of the MV and MVD of the merge candidate. All other processing remains the same as in GPM. With GMVD, the MVD is signaled as a pair of direction and distance. There are nine candidate distances (1 / 4 pixel, 1 / 2 pixel, 1 pixel, 2 pixels, 3 pixels, 4 pixels, 6 pixels, 8 pixels, 16 pixels) and eight candidate directions (four horizontal / vertical directions and four diagonal directions). In addition, when pic_fpel_MVD_enabled_flag is equal to 1, the MVD in GMVD is also left-shifted by 2 like in MMVD. 2.29. Affine MMVD In affine MMVD, an affine merge candidate (called base affine merge candidate) is selected and the MV of a control point is further refined by the signaled MVD information. The MVD information of the MVs of all control points is the same in one prediction direction. When the starting MV is a bidirectional prediction MV in which the two MVs point to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), the MV offset added to the list 0MV component of the starting MV and the MV offset of the list 1MV have opposite values; otherwise, when the starting MV is a bidirectional prediction MV in which both lists point to the same side of the current picture (i.e., the POCs of the two references are both greater than the POC of the current picture, or both are less than the POC of the current picture), the MV offset added to the list 0MV component of the starting MV and the MV offset of the list 1MV are the same. 2.30. Adaptive Decoder-side Motion Vector Refinement (ADMVR) In ECM-2.0, a multi-pass decoder-side motion vector refinement (DMVR) method is applied in regular merge mode if the selected merge candidate meets the DMVR condition. In the first pass, bilateral matching (BM) is applied to the codec block. In the second pass, BM is applied to each 16x16 sub-block within the codec block. In the third pass, the MV in each 8x8 sub-block is refined by applying bidirectional optical flow (BDOF). The adaptive decoder-side motion vector refinement method consists of two new merge modes that introduce refined MVs in only one direction, L0 or L1, of bidirectional prediction for merge candidates that satisfy the DMVR condition. For the selected merge candidates, a multi-pass DMVR process is applied to refine the motion vectors, however MVD0 or MVD1 is set to zero in the first-pass (i.e. PU-level) DMVR. Like the regular merge mode, the merge candidates for the proposed merge mode are derived from spatially adjacent coded blocks, TMVP, non-adjacent blocks, HMVP, and paired candidates. The difference is that only those that satisfy the DMVR condition are added to the candidate list. The same merge candidate list (i.e., ADMVR merge list) is used by both proposed merge candidates, and the merge index is coded and decoded as in the regular merge mode. 3. Question In the current design of adaptive merge candidate reordering based on template matching, the motion information of the merge candidates is directly used in the reordering process. However, the motion information of some merge candidates may be modified by motion refinement, such as template matching and / or decoder-side motion vector refinement. Since the motion information used in the reordering process is not aligned with the motion information used in the reconstruction of the video unit, the reordering process may not be efficient. 4. Embodiments of the present disclosure The following detailed embodiments should be considered as examples to explain the general concept. These embodiments should not be interpreted in a narrow manner. In addition, these embodiments can be combined in any way. In the present disclosure, the motion candidate list may refer to a motion candidate list for TM merge mode, and / or normal merge mode, and / or adaptive DMVR, and / or CIIP with TM merge mode, and / or AMVP, and / or affine, and / or GPM (GEO), and / or TPM, and / or MMVD, and / or CIIP, and / or MHP, and / or AMVP-Merge, affine AMVP, TM AMVP, GPM with MMVD, IBC merge, IBC AMVP, template-IBC, intra-template matching and / or any inter-mode that uses a motion candidate list to derive prediction / reconstruction of a block. The motion candidate list may be a TM merge / AMVP list after block-based bilateral matching refinement and / or template matching refinement and / or sub-block-based bilateral matching refinement. The motion candidate list may be a regular merge / AMVP list after the DMVR / multi-pass DMVR process. The motion candidate list may be an ADMVR merged list after the DMVR / multi-pass DMVR process. The motion candidate list may be a GPM merge list (eg, AGPMList, LGPMList, or LAGPMList) after a template matching refinement process. In the following discussion, SatShift(x,n) is defined as In one example, offset0 and / or offset1 are set to (1 <<n)> >1 or (1<<(n-1)). In another example, offset0 and / or offset1 are set to 0. Shift(x,n) is defined as Shift(x,n)=(x+offset0)>>n. Clip3(x,y,z) is defined as Re-ordering the motion candidate list using refined motion information 1. Before the motion candidate list undergoes the first re-ranking procedure, at least one motion candidate in the proposed list may be modified or refined. a. When one or more motion candidates in a motion candidate list are modified by a motion refinement process for a video unit, it is proposed to use the refined motion information to re-order the motion candidate list. b. In one example, before the first reordering process, a second reordering process may be performed on the motion candidate list. i. In one example, when constructing the motion candidate list in the second re-ordering procedure, the motion candidate list may be re-ordered using the original motion information. ii. In one example, not all of the motion candidates in the motion candidate list after the second reordering procedure are reordered in the first reordering procedure. 1) In one example, the top M motion candidates may be reordered in a first reordering procedure. a) In one example, M may be predefined, such as M=4, 5, 6, 7, 8, 9 or 10. b) In one example, M may be signaled in the bitstream. c) In one example, M may be equal to N+M1, where N is the maximum allowed number of motion candidates for the codec (e.g., a syntax element indicated in the SPS), And M1 is an integer greater than or equal to 0, for example, M1=0, 1, 2, 3, 4, 5 or 6. c. In one example, the motion candidate list may not be reordered before being reordered using the first reordering procedure. d. In one example, the motion refinement process may refer to template matching based motion refinement, such as template matching, and / or bilateral matching based motion refinement, such as DMVR, multi-channel DMVR. e. In one example, a simplified version of the motion refinement process may be applied. i. In one example, when template matching is used as one of the processes in the motion refinement process, the template size can be set equal to 1 or 2. ii. In one example, the template may be downsampled first, and when template matching is used as one of the processes of motion refinement, the downsampled template may be used. 1) In one example, the downsampling ratio may be equal to T, for example, T=2, 3 or 4. iii. In one example, when template matching is used as one of the processes in the motion refinement process, only one of the upper template or the left template may be used. 1) In one example, when both the upper template and the left template are available, the template with the larger size may be used, for example, when the block width is greater than the block height, the upper template is used, and when the block height is greater than the block width, the left template is used. iv. In one example, whether and how to simplify the motion refinement process may depend on codec information, such as block size or QP. f. In one example, the refined motion information used in the first reordering process may be the final motion information of the motion refinement process. i. In one example, when the motion refinement process includes multiple steps, the motion information after the last step in the motion refinement process may be used in the first reordering process. 1) In one example, multiple steps can refine the motion information at different resolutions of the motion vector. a) In one example, refinement may be from integer resolution motion vectors to fractional resolution motion vectors. b) In another example, the refinement may be from fractional resolution motion vector to fractional resolution motion vector. c) In one example, different resolutions of motion vectors may refer to 16 pixels, and / or 8 pixels, and / or 4 pixels, and / or 2 pixels, and / or 1 pixel, and / or 1 / 2 pixel, and / or 1 / 4 pixel, and / or 1 / 8 pixel, and / or 1 / 16 pixel, and / or 1 / 32 pixel, and / or 1 / 64 pixel. 2) In one example, multiple steps may refer to different search shapes to refine the motion vector, such as diamond and / or cross in template matching. a) In one example, the motion information following the diamond search shape of template matching may be used in the first re-ordering process. b) In one example, at least one search shape may be skipped. i. In one example, diamond search shapes can be skipped in TM refinement. ii. In one example, the cross search shape can be skipped in TM refinement. 3) In one example, which motion vector resolution to use may depend on a syntax element indicating the resolution of the motion vector of the motion candidate, such as the value of adaptive motion vector resolution. 4) In one example, when early termination is used in the motion refinement process, the motion information output due to the early termination can be used in the first reordering process. ii. In one example, when more than one motion refinement pass is involved and processed one by one, the motion information derived after the last motion refinement pass may be used in the first reordering pass. 1) In one example, when the motion refinement process includes a first motion refinement process and a second motion refinement process, motion information of the second motion refinement process may be used in the first reordering process. a) In one example, the first motion refinement process may refer to a template matching method, such as TM, or a bilateral matching method such as DMVR / multi-channel DMVR. b) In one example, the second motion refinement process may refer to a template matching method, such as TM, or a bilateral matching method such as DMVR / multi-channel DMVR. g. In one example, the refined motion information used in the first reordering process may be the intermediate motion information of the motion refinement process. i. In one example, when the motion refinement process includes multiple steps, the motion information before the last step in the motion refinement process may be used. 1) In one example, the motion information in the first step may be used. 2) In another example, motion information in integer resolutions (eg, 16 pixels, 8 pixels, 4 pixels, 2 pixels, 1 pixel) of motion vectors may be used. ii. In one example, when more than one motion refinement pass is involved and processed one by one, the motion information derived from any motion refinement pass before the last motion refinement pass may be used in the first reordering pass. 1) In one example, when the motion refinement process includes a first motion refinement process and a second motion refinement process, motion information of the first motion refinement process may be used. a) In one example, the first motion refinement process may refer to a template matching method, such as TM, or a bilateral matching method such as DMVR / multi-channel DMVR. b) In one example, the second motion refinement process may refer to a template matching method, such as TM, or a bilateral matching method such as DMVR / multi-channel DMVR. iii. In one example, when more than one motion refinement process is involved and processed one by one, and the last motion refinement process contains multiple steps, motion information derived from any step before the last step in the last motion refinement process may be used in the first reordering process. iv. In one example, when the motion refinement process includes TM and multi-channel DMVR, and TM is processed between a first channel DMVR (e.g., block-based bilateral matching MV refinement) and a second channel DMVR (e.g., sub-block-based bilateral matching MV refinement), the motion information after TM can be used in the first reordering process. 1) Alternatively, the motion information after the first channel DMVR may be used. 2) Alternatively, the motion information after the second channel DMVR may be used. 3) In one example, when the refined motion information is used in the first reordering process, the early termination method to skip the second channel DMVR may not be applied. h. In one example, when the refined motion information is based on motion information of sub-blocks, the refined motion information of one or more sub-blocks may be used in the first reordering process. i. In one example, refined motion information of one of the sub-blocks may be used in the first reordering process. 1) In one example, refined motion information of an upper left subblock, or one of subblocks located at a left block boundary, or one of subblocks located at an upper block boundary may be used. ii. In one example, refined motion information of more than one sub-block may be used in the first reordering process. 1) In one example, motion information of all sub-blocks may be averaged and used. a) In one example, the motion information of all sub-blocks may be weighted averaged, and the weight parameter may depend on the block / sub-block dimension / size, and / or the distance between the sub-block and the template used in the re-ordering process. 2) In another example, motion information of a sub-block at an upper boundary and / or motion information of a sub-block at a left boundary may be used. i. In one example, the motion candidates may be divided into X subgroups, and the re-ranking process may be processed within each subgroup, where X is greater than 1. i. In one example, the subgroup size (ie, the number of motion candidates within a subgroup, N s ) may be the same for all subgroups, such as N s=2,3,4,5,6,7,8,9,10. ii. In another example, at least one subgroup size may be different from the other subgroups. iii. In one example, the size of the first subset is larger or smaller than the sizes of the other subsets. iv. In one example, the subgroup size may be pre-defined. v. In one example, according to the motion candidate type (eg, adjacent spatial domain, temporal domain, non-adjacent spatial domain, HMVP), the motion candidates may be divided into multiple subgroups. vi. In one example, the subgroup size may be determined based on the available number of types of motion candidates (eg, adjacent spatial and / or temporal motion candidates). j. In one example, an X-tap interpolation filter is used in the first reordering procedure, for example, X=2, 4, 6, 8, 10 or 12. i. In one example, the interpolation filter used in the first reordering procedure is different from the interpolation filter used in the second reordering procedure. ii. Alternatively, the interpolation filter used in the first reordering procedure is the same as the interpolation filter used in the second reordering procedure. iii. In one example, when the first reordering procedure is used, a Y-tap interpolation filter may be used in the second reordering procedure, eg, Y=2, 4, 6, 8, or 10. 2. In one example, whether and / or how to apply the reordering process using the refined motion information may be signaled in the bitstream, or determined on the fly, or depend on codec information. a. In one example, the reordering process using refined motion information may be applied to video units having a size greater than T1, such as T1=8, or 16, or 32, or 64, or 128. b. In another example, the reordering process using refined motion information may be applied to video units having a size smaller than T1, such as T1=64, or 128, or 256, or 512, or 1024, or 2048. c. In one example, whether the reordering process using refined motion information is applied to the video unit may depend on the temporal layer. d. In one example, whether the reordering process using refined motion information is applied to a video unit may depend on a quantization parameter (QP). e. In one example, whether the reordering process using refined motion information is applied to a video unit may depend on the slice / picture type. i. In one example, the reordering process using refined motion information may be applied to B slices / pictures. ii. The reordering process using refined motion information can be applied only to B slices / pictures that have at least one forward reference picture and one backward reference picture. f. In one example, whether the reordering process using refined motion information is applied to different codec tools depends on conditions, which may be different for different codec tools. i. In one example, the codec tool may refer to TM merge mode, and / or ADMVR mode, and / or TMAMVP mode, and / or normal merge mode. g. In one example, whether and / or how the reordering process is applied to different codecs can be controlled by a single syntax element signaled in the bitstream, such as an SPS flag. 3. In one example, when a reordering process using refined motion information is applied to an inter-frame mode with one-sided DMVR, adaptive DMVR, the reordering process using refined motion may be applied to reorder a first motion candidate list, and / or a second motion candidate list, wherein the motion candidates in the first motion candidate list are refined on one side (direction) and the motion candidates in the second motion candidate list are refined on another side (direction). a. In one example, one-sided DMVR may refer to a DMVR method in which motion information (eg, motion vectors) is refined in one direction at one step of DMVR. i. In one example, motion information may be refined in one direction at the first channel of a multi-channel DMVR. ii. In one example, motion information is refined in one direction at the first channel and / or the second channel of the multi-channel DMVR. b. In one example, inter-mode may refer to adaptive DMVR. c. In one example, a reordering process may be applied to reorder the first motion candidate list and the second motion candidate list separately. d. In another example, the first candidate list and the second motion candidate list may be combined together, and a reordering process may be used to reorder the combined motion candidate list. e. In one example, the refined motion information used in the reordering process may be the block-based MV of the first channel in a multi-channel DMVR. 4. In one example, removing redundant motion candidates from one or more motion candidate lists for adaptive DMVR is proposed. a. In one example, redundant motion candidates may refer to those motion candidates in the first motion candidate list that have the same motion information as one of the motion candidates in the second motion candidate list. i. In one example, the first motion candidate list may refer to a motion candidate list in a first direction, and the second motion candidate list may refer to a motion candidate list in a second direction. ii. In one example, the first motion candidate list may refer to the motion candidate list of the second direction, and the second motion candidate list may refer to the motion candidate list of the first direction. b. In one example, redundant motion candidates may refer to motion candidates after the first-pass DMVR. 5. Generate N motion candidates are proposed (for a motion candidate list with M as the maximum allowed number of motion candidates, where N>M. a. M motion candidates selected in sequence are used in prediction / reconstruction. b. By reordering the N motion candidates, M motion candidates are selected in order, and the selected M motion candidates are the first M (eg, with the lowest cost) of the reordered N motion candidates. c. In one example, N may be predefined, or signaled in the bitstream, or determined on the fly. i. In one example, N may be different for different codec modes. ii. In another example, N can be 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20. d. In one example, motion information derived from the motion candidate list may be used in the re-ordering process. e. In one example, the above reordering process using refined motion information may be applied. 6. In one example, the cost values ​​generated in a refinement process such as TM or DMVR / Multi-channel DMVR may be reused in a reordering process after the refinement process. a. In one example, the final (or minimum or maximum) cost value generated during the refinement process for a motion candidate may be stored. b. In one example, the final (or minimum or maximum) cost value generated in the refinement process for the first motion candidate in the re-ranking process may be compared with the final (or minimum or maximum) cost value generated in the refinement process for the second motion candidate. 7. In one example, according to the merge type, the merge candidates can be reordered, and the top X merge candidates of each merge type are used to construct the adaptive DMVR merge candidate list. a. In one example, the merge type may refer to adjacent airspace, and / or time domain, and / or non-adjacent airspace, and / or HMVP, and / or pairwise. b. In one example, X may be different for different merge types. i. In one example, for time domain merging candidates, X may be set equal to 1 or 2. ii. In another example, for spatial merging candidates, X may be set to be equal to 1, or 2, or 3, or 4, or 5, or 6, or 7, or 8, or 9, or 10. iii. In another example, for non-adjacent spatial merging candidates, X may be set to be equal to 1, or 2, or 3, or 4, or 5, or 6, or 7, or 8, or 9, or 10. iv. In another example, for HMVP merge candidates, X may be set equal to 1, or 2, or 3, or 4, or 5, or 6, or 7, or 8, or 9, or 10. c. In another example, for pair-wise merge candidates, X may be set equal to 1, or 2, or 3, or 4, or 5, or 6, or 7, or 8, or 9, or 10. General 8. In the above examples, a video unit may refer to a color component / sub-picture / slice / tile / codec tree unit (CTU) / CTU row / CTU group / codec unit (CU) / prediction unit (PU) / transform unit (TU) / codec tree block (CTB) / codec block (CB) / prediction block (PB) / transform block (TB) / block / sub-block of a block / sub-region within a block / any other region containing more than one sample or pixel. 9. Whether to apply the method disclosed above and / or how to apply the method disclosed above can be transmitted by a signal at the sequence level / picture group level / picture level / slice level / tile group level, for example in the sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / slice header / tile group header. 10. Whether and / or how to apply the method disclosed above may be signaled at a PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU row / slice / tile / sub-picture / other type of region containing more than one sample or pixel. 11. Whether to apply the method disclosed above and / or how to apply the method disclosed above may depend on the coded information, such as block size, color format, single / dual tree partitioning, color component, slice / picture type. 5. Examples 5.1. Example 1 In the current design of TM merging in ECM-3.1, first the TM merge candidate list is constructed. Then adaptive reordering of merge candidates (ARMC) can be applied to reorder the merge candidates. Finally, if the conditions for multi-pass DMVR are met, the merge candidates selected using template matching and multi-pass DMVR are refined. The original motion information in the TM merge candidate list is used in ARMC, but the refined motion information after template matching and multi-pass DMVR is used for prediction, which can limit the codec performance. ARMC using refined motion information is proposed for TM merge mode application. Specifically, first a TM merge candidate list is constructed and reordered for all merge candidates, while only the first 5 TM merge candidates in ECM-3.1 are reordered. Secondly, the TM merge candidates in the TM merge list are divided into N subgroups, and the subgroup size is set equal to 4. Template matching and multi-channel DMVR are applied to all TM merge candidates in the selected subgroup, which is determined by the merge index transmitted by the signal. Finally, the refined merge candidates in the subgroup are reordered. To reduce the complexity, the following simplifications are applied: 1) The proposed method is only applied to blocks with luma sample size larger than T1, e.g. T1=16, or 32, or 64. 2) The proposed method is only applied to blocks with luma sample size smaller than T2, e.g. T2=128, or 256, or 512, or 1024, or 2048. 3) When multi-channel DMVR is applied to the selected sub-group, only the first channel DMVR is used. 4) When template matching is applied to the selected subgroup, the template size for template matching is set equal to 1. 5) When template matching is applied to the selected subgroup, template matching is terminated early after the diamond search. 5.2. Example 2 In the current design of adaptive DMVR in ECM-3.1, an adaptive DMVR merge candidate list is first constructed, and the allowed merge candidates can be up to the maximum number of merge candidates (N) for adaptive DMVR signaled in the bitstream, and N can be selected from 0 to 6, and is set equal to 4 in ECM-3.1. Then adaptive reordering of merge candidates (ARMC) is applied to reorder the merge candidates for adaptive DMVR. When constructing the adaptive DMVR merge candidate list, increasing the allowed number of merge candidates (M) is proposed, where M is greater than N. ARMC is then applied to reorder the merge candidates. At the encoder, only the first N merge candidates are involved in the RDO of adaptive DMVR. M is set equal to 10.

[0097] Embodiments of the present disclosure relate to motion candidate list construction.

[0098] As used herein, the terms "video unit" or "codec unit" or "block" used herein may refer to one or more of the following: color component, sub-picture, slice, tile, codec tree unit (CTU), CTU row, CTU group, codec unit (CU), prediction unit (PU), transform unit (TU), codec tree block (CTB), codec block (CB), prediction block (PB), transform block (TB), block, sub-block of a block, sub-region within a block, or area including more than one sample or pixel.

[0099] In the present disclosure, with respect to “a block encoded and decoded in mode N”, the term “mode N” may be a prediction mode (e.g., MODE_INTRA, MODE_INTER, MODE_PLT, MODE_IBC, etc.) or a coding and decoding technology (e.g., AMVP, merge, SMVD, BDOF, PROF, DMVR, AMVR, TM, affine, CIIP, GPM, MMVD, BCW, HMVP, SbTMVP, etc.).

[0100] It should be noted that the terms mentioned below are not limited to the specific terms defined in existing standards. Any variants of codec tools are also applicable.

[0101] Fig.41 A flow chart of a method 4100 for video processing according to an embodiment of the present disclosure is shown. The method 4100 is implemented during conversion between a video unit of a video and a bitstream of the video.

[0102] At block 4110, a process is applied to at least one motion candidate in the motion candidate list for conversion between a video unit of a video and a bitstream of the video unit. In some embodiments, the process includes at least one of: a modification process, a motion refinement process, or a second reordering process.

[0103] At block 4120, at least some of the motion candidates in the motion candidate list are reordered by applying a first reordering procedure to the motion candidate list after the processing procedure. In one example, before the motion candidate list is subjected to the first reordering procedure, at least one motion candidate in the list may be modified or refined.

[0104] At block 4130, conversion is performed based on the reordered motion candidates. In some embodiments, conversion may include encoding the video unit into a bitstream. In some embodiments, conversion may include decoding the video unit from the bitstream. Codec efficiency and performance may be improved compared to conventional techniques.

[0105] In some embodiments, the processing procedure includes a second reordering procedure. In this case, in some embodiments, only part of the motion candidates in the motion candidate list after the second reordering procedure are reordered in the first reordering procedure. In other words, not all of the motion candidates in the motion candidate list after the second reordering procedure are reordered in the first reordering procedure.

[0106] In one example, the first M motion candidates are reordered in a first reordering procedure. M can be an integer. For example, M is predefined. As an example, M is equal to one of 4, 5, 6, 7, 8, 9, or 10. Alternatively, M can be indicated in the bitstream. In some other embodiments, M is equal to N+M1. In this case, N can be the maximum allowed number of motion candidates for the codec tool (e.g., a syntax element indicated in the SPS), and M1 can be an integer greater than or equal to 0. For example, M1 can be equal to one of 0, 1, 2, 3, 4, 5, or 6.

[0107] In some embodiments, the processing procedure includes a motion refinement process, and the motion refinement process is simplified. For example, if template matching is used as the motion refinement process, only one of the upper template or the left template can be used in the motion refinement process. In some embodiments, if both the upper template and the left template are available, the template with the larger size is used. For example, if the block width is greater than the block height, the upper template is used. Alternatively or additionally, if the block height is greater than the block width, the left template is used.

[0108] In some embodiments, whether to simplify the motion refinement process and the method of simplifying the motion refinement process depend on the encoding and decoding information of the video unit. For example, the encoding information includes at least one of a block size or a quantization parameter (QP).

[0109] In some embodiments, if template matching is used as a motion refinement process, the first downsampled template is used in the motion refinement process. For example, the downsampling ratio of the template is equal to T. T can be an integer. As an example, T can be equal to 2, 3 or 4.

[0110] In some embodiments, the processing procedure is a motion refinement procedure. The refined motion information used in the first reordering procedure may be the final motion information of the motion refinement procedure. If the motion refinement procedure includes multiple steps, the motion information after the last step in the motion refinement procedure may be used in the first reordering procedure. The multiple steps may include different search shapes to refine the motion vector in template matching. In this case, at least one search shape in the multiple steps is skipped. In some embodiments, the diamond search shape is skipped in template matching (TM) refinement. Alternatively or additionally, the cross search shape is skipped in TM refinement.

[0111] In some embodiments, an interpolation filter with X-taps is used in the first reordering procedure. X may be an integer. In some embodiments, X is equal to one of 2, 4, 6, 8, 10, or 12.

[0112] In some embodiments, the interpolation filter used in the first reordering procedure is different from another interpolation filter used in the second reordering procedure. Alternatively, the interpolation filter used in the first reordering procedure is the same as another interpolation filter used in the second reordering procedure.

[0113] In some embodiments, another interpolation filter with Y-tap is used in the second reordering procedure. Y can be an integer. In some embodiments, Y is equal to one of 2, 4, 6, 8 or 10.

[0114] In some embodiments, whether the first reordering procedure using the refined motion information is applied to the video unit depends on the QP. In some other embodiments, whether the first reordering procedure using the refined motion information is applied to the codec tool depends on a condition, and the condition is different for different codec tools. For example, the codec tool includes at least one of the following: TM merge mode, advanced decoder side motion vector refinement (ADMVR) mode, TM advanced motion vector prediction (AMVP) mode, or normal merge mode.

[0115] In some embodiments, whether to apply the first reordering procedure and the scheme of applying the first reordering procedure for different codecs are controlled by a single syntax element indicated in the bitstream. In some other embodiments, whether the first reordering procedure of refined motion information is applied to video units depends on the temporal layer.

[0116] In some embodiments, whether the first reordering procedure using the refined motion information is applied to the video unit depends on at least one of a slice type or a picture type. For example, the first reordering procedure is applied to at least one of a B slice or a B picture. In some embodiments, the first reordering procedure is applied to at least one of: a B slice or a B picture having at least one forward reference picture and one backward reference picture.

[0117] In some embodiments, a video unit refers to one of: a color component, a sub-image, a slice, a tile, a codec tree unit (CTU), a CTU row, a CTU group, a codec unit (CU), a prediction unit (PU), a transform unit (TU), a codec tree block (CTB), a codec block (CB), a prediction block (PB), a transform block (TB), a block, a sub-block of a block, a sub-region within a block, or a region containing more than one sample or pixel. In some embodiments, an indication of whether and / or how to reorder at least some of the motion candidates in the motion candidate list by applying a first reordering procedure to the motion candidate list after a processing procedure is indicated at one of: a sequence level, a picture group level, a picture level, a slice level, or a tile group level.

[0118] In some embodiments, an indication of whether and / or how to reorder at least some of the motion candidates in the motion candidate list by applying a first reordering procedure to the motion candidate list after a processing procedure is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header. In some embodiments, an indication of whether and / or how to reorder at least some of the motion candidates in the motion candidate list by applying a first reordering procedure to the motion candidate list after a processing procedure is included in one of the following: a prediction block (PB), a transform block (TB), a codec block (CB), a prediction unit (PU), a transform unit (TU), a codec unit (CU), a virtual pipeline data unit (VPDU), a codec tree unit (CTU), a CTU row, a slice, a tile, a sub-picture, or a region containing more than one sample or pixel.

[0119] In some embodiments, method 4100 also includes: determining whether and / or how to reorder at least part of the motion candidates in the motion candidate list based on codec information of the video unit by applying a first reordering procedure to the motion candidate list after the processing procedure, the codec information including at least one of the following: block size, color format, single tree partitioning and / or double tree partitioning, color component, slice type, or picture type.

[0120] In some embodiments, converting comprises encoding the video unit into a bitstream. In some embodiments, converting comprises decoding the video unit from the bitstream.

[0121] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video, which is generated by a method performed by a video processing device. The method includes: for conversion between a video unit of a video and a bitstream of the video unit, applying a processing program to at least one motion candidate in a motion candidate list; reordering at least part of the motion candidates in the motion candidate list by applying a first reordering program to the motion candidate list after the processing program; and generating a bitstream based on the reordered motion candidates.

[0122] According to some other embodiments of the present disclosure, a method for storing a bitstream of a video is provided. The method includes: for conversion between a video unit of the video and a bitstream of the video unit, applying a processing program to at least one motion candidate in a motion candidate list; reordering at least part of the motion candidates in the motion candidate list by applying a first reordering program to the motion candidate list after the processing program; generating a bitstream based on the reordered motion candidates; and storing the bitstream in a non-transitory computer-readable recording medium.

[0123] Embodiments of the present disclosure may be described according to the following clauses, features of which may be combined in any reasonable way.

[0124] Clause 1. A method for video processing, comprising: applying a processing procedure to at least one motion candidate in a motion candidate list for conversion between a video unit of a video and a bitstream of the video unit; reordering at least part of the motion candidates in the motion candidate list by applying a first reordering procedure to the motion candidate list after the processing procedure; and performing conversion based on the reordered motion candidates.

[0125] Clause 2. The method according to clause 1, wherein the processing procedure comprises a second reordering procedure, and wherein only part of the motion candidates in the motion candidate list after the second reordering procedure are reordered in the first reordering procedure.

[0126] Clause 3. According to the method of clause 2, the top M motion candidates are reordered in a first reordering procedure, where M is an integer.

[0127] Clause 4. The method of clause 3, wherein M is predefined and wherein M is equal to one of 4, 5, 6, 7, 8, 9 or 10.

[0128] Clause 5. The method of clause 3, wherein M is indicated in the bitstream.

[0129] Clause 6. The method of clause 3, wherein M is equal to N+M1, and wherein N is the maximum allowed number of motion candidates for the codec, and M1 is an integer greater than or equal to zero.

[0130] Clause 7. The method of clause 6, wherein M1 is equal to one of 0, 1, 2, 3, 4, 5 or 6.

[0131] Clause 8. The method of clause 1, wherein the processing procedure includes a motion refinement process, the motion refinement process is simplified, and wherein if template matching is used as the motion refinement process, only one of the upper template or the left template is used in the motion refinement process.

[0132] Clause 9. According to the method of clause 8, if both the upper template and the left template are available, the template with the larger size is used.

[0133] Clause 10. The method of clause 9, wherein if the block width is greater than the block height, the top template is used, and wherein if the block height is greater than the block width, the left template is used.

[0134] Clause 11. The method according to clause 1, wherein the processing procedure includes a motion refinement process, the motion refinement process is simplified, and wherein whether to simplify the motion refinement process and a scheme of simplifying the motion refinement process depend on codec information of the video unit.

[0135] Clause 12. The method of clause 11, wherein the codec information comprises at least one of: a block size or a quantization parameter (QP).

[0136] Clause 13. The method of clause 1, wherein the processing comprises a motion refinement process, the motion refinement process is simplified, and wherein if template matching is used as the motion refinement process, a previously downsampled template is used in the motion refinement process.

[0137] Clause 14. The method of clause 13, wherein the downsampling ratio of the template is equal to T, and wherein T is an integer.

[0138] Clause 15. The method of clause 13, wherein T is equal to 2, 3 or 4.

[0139] Clause 16. A method according to clause 1, wherein the processing procedure is a motion refinement process, wherein the refined motion information used in the first reordering procedure is the final motion information of the motion refinement process, wherein if the motion refinement process includes multiple steps, the motion information after the last step in the motion refinement process is used in the first reordering procedure, wherein the multiple steps include different search shapes to refine the motion vector in template matching, and wherein at least one search shape in the multiple steps is skipped.

[0140] Clause 17. The method of clause 16, wherein diamond search shapes are skipped in template matching (TM) refinement, and / or wherein cross search shapes are skipped in TM refinement.

[0141] Clause 18. The method of clause 1, wherein an interpolation filter with X-taps is used in the first reordering procedure, and wherein X is an integer.

[0142] Clause 19. The method of clause 18, wherein X is equal to one of 2, 4, 6, 8, 10, or 12.

[0143] Clause 20. The method of clause 18, wherein an interpolation filter used in the first reordering procedure is different from another interpolation filter used in the second reordering procedure.

[0144] Clause 21. The method of clause 18, wherein an interpolation filter used in the first reordering procedure is the same as another interpolation filter used in the second reordering procedure.

[0145] Clause 22. The method of clause 18, wherein another interpolation filter with Y-taps is used in the second reordering procedure, and wherein Y is an integer.

[0146] Clause 23. The method of clause 22, wherein Y is equal to one of 2, 4, 6, 8 or 10.

[0147] Clause 24. The method of clause 1, wherein whether the first reordering procedure using the refined motion information is applied to the video unit depends on the QP.

[0148] Clause 25. The method of clause 1, wherein whether the first reordering procedure using the refined motion information is applied to the codec tool depends on a condition, and the condition is different for different codec tools.

[0149] Clause 26. The method of clause 25, wherein the codec tool comprises at least one of: TM merge mode, advanced decoder side motion vector refinement (ADMVR) mode, TM advanced motion vector prediction (AMVP) mode, or normal merge mode.

[0150] Clause 27. The method of clause 1, wherein whether to apply the first reordering procedure and the scheme of applying the first reordering procedure for different codecs are controlled by a single syntax element indicated in the bitstream.

[0151] Clause 28. The method of clause 1, wherein whether the first reordering procedure using the refined motion information is applied to the video units depends on the temporal layer.

[0152] Clause 29. The method of clause 1, wherein whether the first reordering procedure using the refined motion information is applied to the video unit depends on at least one of a slice type or a picture type.

[0153] Clause 30. The method of clause 29, wherein the first reordering procedure is applied to at least one of a B slice or a B picture.

[0154] Clause 31. The method of clause 29, wherein the first reordering procedure is applied to at least one of: a B slice or a B picture having at least one forward reference picture and one backward reference picture.

[0155] Clause 32. A method according to any of clauses 1-31, wherein a video unit refers to one of the following: a color component, a sub-image, a slice, a tile, a codec tree unit (CTU), a CTU row, a CTU group, a codec unit (CU), a prediction unit (PU), a transform unit (TU), a codec tree block (CTB), a codec block (CB), a prediction block (PB), a transform block (TB), a block, a sub-block of a block, a sub-region within a block, or a region containing more than one sample or pixel.

[0156] Clause 33. A method according to any of clauses 1-31, wherein an indication of whether and / or how to reorder at least part of the motion candidates in the motion candidate list by applying a first reordering procedure to the motion candidate list after a processing procedure is indicated at one of: a sequence level, a picture group level, a picture level, a slice level, or a tile group level.

[0157] Clause 34. A method according to any of clauses 1-31, wherein an indication of whether and / or how to reorder at least some of the motion candidates in the motion candidate list by applying a first reordering procedure to the motion candidate list after a processing procedure is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.

[0158] Clause 35. A method according to any of clauses 1-31, wherein an indication of whether and / or how to reorder at least part of the motion candidates in the motion candidate list by applying a first reordering procedure to the motion candidate list after a processing procedure is included in one of the following: a prediction block (PB), a transform block (TB), a codec block (CB), a prediction unit (PU), a transform unit (TU), a codec unit (CU), a virtual pipeline data unit (VPDU), a codec tree unit (CTU), a CTU row, a slice, a tile, a sub-picture, or a region containing more than one sample or pixel.

[0159] Clause 36. The method according to any one of clauses 1-31 also includes: determining whether and / or how to reorder at least part of the motion candidates in the motion candidate list by applying a first reordering procedure to the motion candidate list after a processing procedure based on codec information of the video unit, the codec information including at least one of the following: block size, color format, single tree partitioning and / or double tree partitioning, color component, slice type, or picture type.

[0160] Clause 37. The method of any of clauses 1-36, wherein converting comprises encoding the video unit into a bitstream.

[0161] Clause 38. The method of any of clauses 1-36, wherein converting comprises decoding the video unit from a bitstream.

[0162] Clause 39. An apparatus for video processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method according to any of clauses 1-38.

[0163] Clause 40. A non-transitory computer-readable storage medium storing instructions for causing a processor to perform a method according to any of clauses 1-38.

[0164] Item 41. A non-transitory computer-readable recording medium storing a bitstream of a video, the bitstream being generated by a method performed by a video processing device, wherein the method comprises: applying a processing program to at least one motion candidate in a motion candidate list for conversion between a video unit of a video and a bitstream of the video unit; reordering at least part of the motion candidates in the motion candidate list by applying a first reordering program to the motion candidate list after the processing program; and generating a bitstream based on the reordered motion candidates.

[0165] Clause 42. A method for storing a bitstream of a video, comprising: applying a processing program to at least one motion candidate in a motion candidate list for conversion between a video unit of the video and a bitstream of the video unit; reordering at least part of the motion candidates in the motion candidate list by applying a first reordering program to the motion candidate list after the processing program; generating a bitstream based on the reordered motion candidates; and storing the bitstream in a non-transitory computer-readable recording medium. Example Device

[0166] Fig.42 A block diagram of a computing device 4200 in which various embodiments of the present disclosure may be implemented is shown. The computing device 4200 may be implemented as or included in the source device 110 (or video encoder 114 or 200) or the destination device 120 (or video decoder 124 or 300).

[0167] It should be understood that Fig.42 The computing device 4200 shown is for illustrative purposes only and is not intended to in any way imply any limitation on the functionality and scope of the embodiments of the present disclosure.

[0168] like Fig.42 As shown, computing device 4200 includes a general computing device 4200. Computing device 4200 may include at least one or more processors or processing units 4210, memory 4220, storage unit 4230, one or more communication units 4240, one or more input devices 4250, and one or more output devices 4260.

[0169] In some embodiments, the computing device 4200 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server provided by a service provider, a large computing device, etc. The user terminal can be, for example, any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet computer, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It can be expected that the computing device 4200 can support any type of interface with the user (such as a "wearable" circuit, etc.).

[0170] The processing unit 4210 may be a physical or virtual processor and may implement various processes based on a program stored in the memory 4220. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the computing device 4200. The processing unit 4210 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0171] The computing device 4200 typically includes various computer storage media. Such media can be any media accessible to the computing device 4200, including but not limited to volatile and non-volatile media, or removable and non-removable media. The memory 4220 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), or flash memory), or any combination thereof. The storage unit 4230 can be any removable or non-removable medium, and can include a machine-readable medium, such as a memory, a flash drive, a disk, or other media, which can be used to store information and / or data and can be accessed in the computing device 4200.

[0172] The computing device 4200 may also include additional removable / non-removable, volatile / non-volatile storage media. Fig.42 Not shown in the figure, but a magnetic disk drive for reading and / or writing a removable and nonvolatile magnetic disk and an optical disk drive for reading and / or writing a removable nonvolatile optical disk may be provided. In this case, each drive may be connected to a bus (not shown) via one or more data medium interfaces.

[0173] The communication unit 4240 communicates with another computing device via a communication medium. In addition, the functions of the components in the computing device 4200 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 4200 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.

[0174] Input device 4250 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 4260 may be one or more of various output devices, such as a display, speaker, printer, etc. Through communication unit 4240, computing device 4200 may also communicate with one or more external devices (not shown) such as storage devices and display devices, one or more of which enable a user to interact with computing device 4200, or any device (e.g., a network card, a modem, etc.) that enables computing device 4200 to communicate with one or more other computing devices if necessary. Such communication may be performed via an input / output (I / O) interface (not shown).

[0175] In some embodiments, some or all components of the computing device 4200 may also be arranged in a cloud computing architecture rather than being integrated in a single device. In a cloud computing architecture, components may be provided remotely and work together to implement the functions described in the present disclosure. In some embodiments, cloud computing provides computing, software, data access, and storage services without requiring end users to understand the physical location or configuration of the systems or hardware that provide these services. In various embodiments, cloud computing provides services over a wide area network (e.g., the Internet) using a suitable protocol. For example, a cloud computing provider provides applications over a wide area network that can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data may be stored on a server at a remote location. The computing resources in a cloud computing environment may be merged or distributed at the location of a remote data center. Cloud computing infrastructure may provide services through a shared data center, although they act as a single access point for users. Therefore, cloud computing architecture may be used to provide components and functions described herein from a service provider at a remote location. Alternatively, they may be provided from a conventional server or directly installed or otherwise installed on a client device.

[0176] The computing device 4200 may be used to implement video encoding / decoding in embodiments of the present invention. The memory 4220 may include one or more video encoding / decoding modules 4225 having one or more program instructions. These modules may be accessed and executed by the processing unit 4210 to perform the functions of various embodiments described herein.

[0177] In an example embodiment performing video encoding and decoding, the input device 4250 may receive video data as input to be encoded 4270. For example, the video data may be processed by the video encoding and decoding module 4225 to generate an encoded bitstream. The encoded bitstream may be provided as output 4280 via the output device 4260.

[0178] In an example embodiment performing video decoding, input device 4250 may receive an encoded bitstream as input 4270. The encoded bitstream may be processed, for example, by video codec module 4225 to generate decoded video data. The decoded video data may be provided as output 4280 via output device 4260.

[0179] Although the present disclosure has been specifically shown and described with reference to its preferred embodiments, it will be appreciated by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present application, as defined by the appended claims. These changes are intended to be covered by the scope of the present application. Therefore, the foregoing description of the embodiments of the present application is not intended to be limiting.

Claims

1. A method for video processing, comprising: For conversion between a video unit of a video and a bitstream of the video unit, applying a processing procedure to at least one motion candidate in a motion candidate list; reordering at least part of the motion candidates in the motion candidate list by applying a first reordering procedure to the motion candidate list after the processing procedure; as well as The conversion is performed based on the reordered motion candidates.

2. The method of claim 1, wherein the processing procedure comprises a second reordering procedure, and Wherein only part of the motion candidates in the motion candidate list after the second reordering procedure are reordered in the first reordering procedure. 3 . The method of claim 2 , wherein the top M motion candidates are reordered in the first reordering procedure, where M is an integer.

4. The method according to claim 3, wherein M is predefined, and Where M is equal to one of 4, 5, 6, 7, 8, 9 or 10. The method of claim 3 , wherein M is indicated in the bitstream.

6. The method of claim 3, wherein M is equal to N+M1, and Where N is the maximum allowed number of motion candidates for the codec, and M1 is an integer greater than or equal to 0. The method of claim 6 , wherein M1 is equal to one of 0, 1, 2, 3, 4, 5 or 6.

8. The method of claim 1, wherein the processing procedure includes a motion refinement process, the motion refinement process is simplified, and If template matching is used as the motion refinement process, only one of the upper template or the left template is used in the motion refinement process.

9. The method of claim 8, wherein if both the upper template and the left template are available, the template with a larger size is used.

10. The method of claim 9, wherein if the block width is greater than the block height, the upper template is used, and If the block height is greater than the block width, the left template is used.

11. The method of claim 1, wherein the processing procedure includes a motion refinement process, the motion refinement process is simplified, and Whether to simplify the motion refinement process and the scheme for simplifying the motion refinement process depend on the encoding and decoding information of the video unit.

12. The method according to claim 11, wherein the codec information comprises at least one of the following: a block size or a quantization parameter (QP).

13. The method of claim 1, wherein the processing comprises a motion refinement process, the motion refinement process being simplified, and If template matching is used as the motion refinement process, the template that is first downsampled is used in the motion refinement process. The method according to claim 13 , wherein the down-sampling ratio of the template is equal to T, and wherein T is an integer. The method of claim 13 , wherein T is equal to 2, 3 or 4.

16. The method of claim 1, wherein the processing procedure is a motion refinement process, wherein the refined motion information used in the first reordering procedure is the final motion information of the motion refinement process, wherein if the motion refinement process comprises a plurality of steps, the motion information after the last step in the motion refinement process is used in the first reordering procedure, wherein the plurality of steps include different search shapes to refine the motion vector in template matching, and At least one search shape in the plurality of steps is skipped.

17. The method of claim 16, wherein diamond search shapes are skipped in Template Matching (TM) refinement, and / or The cross search shape is skipped in the TM refinement.

18. The method of claim 1, wherein an interpolation filter with X-taps is used in the first reordering procedure, and wherein X is an integer.

19. The method of claim 18, wherein X is equal to one of 2, 4, 6, 8, 10, or 12.

20. The method of claim 18, wherein the interpolation filter used in the first reordering procedure is different from another interpolation filter used in the second reordering procedure.

21. The method of claim 18, wherein the interpolation filter used in the first reordering procedure is the same as another interpolation filter used in the second reordering procedure.

22. The method of claim 18, wherein another interpolation filter with Y-taps is used in the second reordering procedure, and wherein Y is an integer.

23. The method of claim 22, wherein Y is equal to one of 2, 4, 6, 8 or 10.

24. The method of claim 1, wherein whether the first reordering procedure using refined motion information is applied to the video unit depends on a QP.

25. The method of claim 1, wherein whether the first reordering procedure using refined motion information is applied to a codec tool depends on a condition, and The conditions are different for different codec tools.

26. The method of claim 25, wherein the encoding and decoding tools include at least one of the following: TM merge mode, Advanced Decoder-Side Motion Vector Refinement (ADMVR) mode, TM Advanced Motion Vector Prediction (AMVP) mode, or Regular merge mode.

27. The method of claim 1, wherein whether to apply the first reordering procedure and a scheme of applying the first reordering procedure for different codecs are controlled by a single syntax element indicated in the bitstream.

28. The method of claim 1, wherein whether the first reordering procedure using refined motion information is applied to the video units depends on a temporal layer.

29. The method of claim 1, wherein whether the first reordering procedure using refined motion information is applied to the video unit depends on at least one of a slice type or a picture type.

30. The method of claim 29, wherein the first reordering procedure is applied to at least one of a B slice or a B picture.

31. The method of claim 29, wherein the first reordering procedure is applied to at least one of: a B slice or a B picture having at least one forward reference picture and one backward reference picture.

32. The method of any one of claims 1-31, wherein the video unit refers to one of: Color components, sub-image, Strips, Tile, Codec Tree Unit (CTU), CTU line, CTU group, Codec Unit (CU), Prediction Unit (PU), Transformation Unit (TU), Codec Tree Block (CTB), Codec Block (CB), Prediction Block (PB), Transform Block (TB), piece, Sub-blocks of a block, A sub-region within a block, or An area that contains more than one sample or pixel.

33. The method according to any one of claims 1 to 31, wherein an indication of whether and / or how to reorder at least part of the motion candidates in the motion candidate list by applying the first reordering procedure to the motion candidate list after the processing procedure is indicated at one of: Sequence level, Picture group level, Picture level, Stripe level, or Tile group level.

34. The method according to any one of claims 1 to 31, wherein an indication of whether and / or how to reorder at least part of the motion candidates in the motion candidate list by applying the first reordering procedure to the motion candidate list after the processing procedure is indicated in one of: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependency Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), Strip header, or Tile group header.

35. The method according to any one of claims 1 to 31, wherein an indication of whether and / or how to reorder at least part of the motion candidates in the motion candidate list by applying the first reordering procedure to the motion candidate list after the processing procedure is included in one of: Prediction Block (PB), Transform Block (TB), Codec Block (CB), Prediction Unit (PU), Transformation Unit (TU), Codec Unit (CU), Virtual Pipeline Data Unit (VPDU), Codec Tree Unit (CTU), CTU line, Strips, Tile, sub-image, or An area that contains more than one sample or pixel.

36. The method according to any one of claims 1-31, further comprising: determining whether and / or how to reorder at least part of the motion candidates in the motion candidate list by applying the first reordering procedure to the motion candidate list after the processing procedure based on the codec information of the video unit, the codec information comprising at least one of the following: Block size, Color format, Single-tree partitioning and / or dual-tree partitioning, Color components, Strip type, or Image type.

37. The method of any one of claims 1-36, wherein the converting comprises encoding the video unit into the bitstream.

38. The method of any one of claims 1-36, wherein the converting comprises decoding the video unit from the bitstream.

39. An apparatus for video processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1-38.

40. A non-transitory computer-readable storage medium storing instructions for causing a processor to execute the method according to any one of claims 1-38.

41. A non-transitory computer-readable recording medium storing a bit stream of a video, the bit stream being generated by a method performed by a video processing device, wherein the method comprises: For conversion between a video unit of a video and a bitstream of the video unit, applying a processing procedure to at least one motion candidate in a motion candidate list; reordering at least part of the motion candidates in the motion candidate list by applying a first reordering procedure to the motion candidate list after the processing procedure; as well as The bitstream is generated based on the reordered motion candidates.

42. A method for storing a bitstream of a video, comprising: For conversion between a video unit of a video and a bitstream of the video unit, applying a processing procedure to at least one motion candidate in a motion candidate list; reordering at least part of the motion candidates in the motion candidate list by applying a first reordering procedure to the motion candidate list after the processing procedure; generating the bitstream based on the reordered motion candidates; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

Images