Method and device for video processing and medium
By refining and reordering the motion candidate list, the problem of insufficient coding efficiency in existing video coding technologies is solved, achieving higher coding gain and coding efficiency.
Patent Information
- Application Number
- CN202380033962.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-12
- Filing Date
- 2023-04-11
- Publication Date
- 2024-11-26
AI Technical Summary
Existing video coding technologies have room for improvement in coding efficiency, especially in the processing of motion candidate lists, where it is difficult to achieve higher coding gain and coding efficiency.
By refining and reordering the motion candidate list, including template matching (TM), decoder-side motion vector refinement (DMVR), and multiple passes of DMVR, combined with cost calculation and reordering processes, the conversion process of motion candidates is optimized.
It improves the coding gain and coding efficiency of video encoding, achieving higher coding efficiency.
Smart Images

Figure CN119032557A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate generally to video processing techniques, and more particularly, to reordering a motion candidate list using refined motion information. Background Art
[0002] Nowadays, digital video functions are being applied to all aspects of people's lives. For video encoding / decoding, various types of video compression technologies have been proposed, such as Moving Picture Experts Group (MPEG)-2, MPEG-4, ITU-TH.263, International Telecommunication Union-Telecommunication Standardization Sector (ITU-T) H.264 / MPEG-4 Part 10 Advanced Video Codec (AVC), ITU-T H.265 High Efficiency Video Codec (HEVC) standard, and Versatile Video Codec (VVC) standard. However, it is generally expected to further improve the coding efficiency of video codec coding technology. Summary of the invention
[0003] The embodiment of the present disclosure provides a video processing solution.
[0004] In a first aspect, a method for video processing is provided. The method comprises: for conversion between a video unit of a video and a bitstream of the video unit, applying a processing flow to at least one motion candidate in a motion candidate list; after the processing flow, reordering at least one motion candidate subgroup in the motion candidate list by applying a first reordering flow to the motion candidate list, wherein the processing flow comprises at least one of the following: a modification process, a motion refinement process, a second reordering process; and performing conversion based on the reordered motion candidates. Compared with conventional techniques, higher coding gain can be achieved. In addition, coding efficiency can be improved.
[0005] In a second aspect, another method for video processing is provided. The method includes: for conversion between a video unit of a video and a bitstream of the video unit, generating a cost in a refinement process of a motion candidate list for the video unit, wherein the refinement process includes at least one of the following: template matching (TM), decoding-side motion vector refinement (DMVR), multi-pass DMVR; applying a reordering process to the motion candidate list using the cost; and performing conversion based on the motion candidate list. Compared with conventional techniques, higher coding gain can be achieved. In addition, coding efficiency can be improved.
[0006] In a third aspect, another method for video processing is proposed. The method includes: for conversion between a video unit of a video and a bitstream of the video unit, applying a refinement process and a reordering process to motion candidates in a motion candidate list, wherein in the refinement process, the motion candidates are refined using a motion search method; and based on the refined and reordered motion candidates, performing conversion. Compared with conventional techniques, higher coding gain can be achieved. In addition, coding efficiency can be improved.
[0007] In a fourth aspect, a video processing device is provided. The device includes a processor and a non-volatile memory having instructions. When the instructions are executed by the processor, the processor executes the method according to the first aspect, the second aspect, or the third aspect of the present disclosure.
[0008] In a fifth aspect, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions for causing a processor to execute instructions of the method according to the first aspect, the second aspect, or the third aspect of the present disclosure.
[0009] In a sixth aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video, the bitstream of the video being generated by a method performed by an apparatus for video processing, the method comprising: applying a processing flow to at least one motion candidate in a motion candidate list; after the processing flow, reordering at least one motion candidate subgroup in the motion candidate list by applying a first reordering flow to the motion candidate list, wherein the processing flow comprises at least one of the following: a modification process, a motion refinement process, a second reordering process; and performing a conversion based on the reordered motion candidates.
[0010] In a seventh aspect, a method for storing a video bitstream is provided. The method includes: applying a processing flow to at least one motion candidate in a motion candidate list; after the processing flow, reordering at least one motion candidate subgroup in the motion candidate list by applying a first reordering flow to the motion candidate list, wherein the processing flow includes at least one of the following: a modification process, a motion refinement process, and a second reordering process; and performing conversion based on the reordered motion candidates; and storing the bitstream in a non-transitory computer-readable recording medium.
[0011] In an eighth aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video, the bitstream of the video being generated by a method performed by an apparatus for video processing, the method comprising: generating a cost in a refinement process of a motion candidate list for a video unit, wherein the refinement process comprises at least one of the following: template matching (TM), decoding-side motion vector refinement (DMVR), and multi-pass DMVR; applying a reordering process to the motion candidate list using the cost; and performing a conversion based on the motion candidate list.
[0012] In a ninth aspect, a method for storing a video bitstream is provided. The method includes: generating a cost in a refinement process of a motion candidate list for a video unit, wherein the refinement process includes at least one of the following: template matching (TM), decoding-side motion vector refinement (DMVR), and multi-pass DMVR; applying a reordering process to the motion candidate list using the cost; and performing a conversion based on the motion candidate list; and storing the bitstream in a non-transitory computer-readable recording medium.
[0013] In a tenth aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video, the bitstream of the video being generated by a method performed by an apparatus for video processing, the method comprising: applying a refinement process and a reordering process to motion candidates in a motion candidate list, wherein in the refinement process, the motion candidates are refined using a motion search method; and performing conversion based on the refined and reordered motion candidates.
[0014] In an eleventh aspect, a method for storing a video bitstream is provided. The method includes: applying a refinement process and a reordering process to motion candidates in a motion candidate list, wherein in the refinement process, the motion candidates are refined using a motion search method; and based on the refined and reordered motion candidates, performing conversion; and storing the bitstream in a non-transitory computer-readable recording medium.
[0015] This disclosure is provided to introduce a selection of concepts in a simplified form, which are further described below in the Detailed Description. This disclosure is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become more apparent through the following detailed description with reference to the accompanying drawings. In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.
[0017] Figure 1A block diagram showing an example video encoding and decoding system according to some embodiments of the present disclosure is shown;
[0018] Figure 2 A block diagram of an example video encoder according to some embodiments of the present disclosure is shown;
[0019] Figure 3 A block diagram of an example video decoder according to some embodiments of the present disclosure is shown;
[0020] Figure 4 A schematic diagram showing an example encoder block diagram is shown;
[0021] Figure 5 A schematic diagram showing 67 intra prediction modes;
[0022] Figure 6 A schematic diagram showing reference samples for wide-angle intra prediction is shown;
[0023] Figure 7 A schematic diagram showing the problem of discontinuity when the orientation exceeds 45°;
[0024] Figure 8 A schematic diagram of the MMVD search point is shown;
[0025] Fig. 9 A schematic diagram of a symmetric MVD pattern is shown;
[0026] Fig.10 shows the extended CU area used in BDOF;
[0027] Fig.11 An affine motion model based on control points is shown;
[0028] Fig.12 The affine MVF of each sub-block is shown;
[0029] Fig.13 shows the position of the inherited affine motion predictor;
[0030] Fig.14 Shows control point motion vector inheritance;
[0031] Fig.15 The positions of candidate positions for constructing an affine merge pattern are shown;
[0032] Fig.16 A diagram showing the use of motion vectors for the proposed combination method;
[0033] Fig.17 Sub-block MV VSB and pixel Δv(i,j) are shown;
[0034] Fig.18a and Fig.18b The SbTMVP process in VVC is shown, where Fig.15 a shows the spatially adjacent blocks used by ATVMP, and Fig.15 b shows the derivation of the sub-CU motion field by applying the motion shift from the spatial neighbors and scaling the motion information from the corresponding co-located sub-CU;
[0035] Fig.19 Shows local illumination compensation;
[0036] Fig. 20 No subsampling for the short edges is shown;
[0037] Fig.21 Decoding side motion vector refinement is shown;
[0038] Fig. 22 A diamond-shaped area in the search area is shown;
[0039] Fig.23 The positions of spatial merging candidates are shown;
[0040] Fig.24 shows candidate pairs considered for redundancy check of spatial merging candidates;
[0041] Fig.25 A diagram showing motion vector scaling for temporal merging candidates is shown;
[0042] Fig.26 The candidate positions of the temporal merge candidates C0 and C1 are shown;
[0043] Fig. 27 The VVC spatial neighboring blocks of the current block are shown;
[0044] Fig.28 shows a diagram of virtual blocks in the i-th round of search;
[0045] Fig.29 An example of GPM partitions grouped at the same angle is shown;
[0046] Fig.30 Unidirectional prediction MV selection for geometric partitioning mode is shown;
[0047] Fig.31 The use of a geometric partitioning pattern to generate a warp weight w_0 is shown;
[0048] Fig.32 shows spatial neighboring blocks used to derive spatial merging candidates;
[0049] Fig.33 shows the template matching performed on the search area around the initial MV;
[0050] Fig.34 A diagram showing sub-blocks of an OBMC application is shown;
[0051] Fig.35 The SBT position, type and transformation type are shown;
[0052] Fig.36 Neighboring samples used to calculate SAD are shown;
[0053] Fig.37 Neighboring samples used to calculate the SAD of sub-CU level motion information are shown;
[0054] Fig.38 The sorting process is shown;
[0055] Fig.39 shows the reordering process in the encoder;
[0056] Fig.40 shows the reordering process in the decoder;
[0057] Fig.41 A flowchart of a method for video processing according to an embodiment of the present disclosure is shown;
[0058] Fig.42 A flowchart of a method for video processing according to an embodiment of the present disclosure is shown;
[0059] Fig.43 A flowchart showing a method for video processing according to an embodiment of the present disclosure; and
[0060] Fig.44 A block diagram of a computing device is shown in which various embodiments of the present disclosure may be implemented.
[0061] Throughout the drawings, the same or similar reference numbers generally refer to the same or similar elements. DETAILED DESCRIPTION
[0062] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described only for the purpose of illustrating and helping those skilled in the art to understand and implement the present disclosure, without implying any limitation on the scope of the present disclosure. In addition to the methods described below, the disclosure described herein can also be implemented in various ways.
[0063] In the following description and claims, unless otherwise defined, all scientific and technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs.
[0064] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment must include the particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example embodiment, whether or not explicitly described, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in relation to other embodiments.
[0065] It should be understood that although the terms "first" and "second" etc. can be used to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish one element from another element. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.
[0066] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms "including", "comprising", and / or "having" when used herein indicate the presence of the features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof. Example Environment
[0067] Figure 1 1 is a block diagram illustrating an example video codec system 100 that may utilize the techniques of the present disclosure. As shown, the video codec system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a video encoding device, and the destination device 120 may also be referred to as a video decoding device. In operation, the source device 110 may be configured to generate encoded video data, and the destination device 120 may be configured to decode the encoded video data generated by the source device 110. The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.
[0068] Video source 112 may include a source such as a video capture device. Examples of a video capture device include, but are not limited to, an interface that receives video data from a video content provider, a computer graphics system for generating video data, and / or combinations thereof.
[0069] The video data may include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a code stream. The code stream may include a bit sequence that forms a coded representation of the video data. The code stream may include coded pictures and associated data. The coded picture is a coded representation of the picture. The associated data may include a sequence parameter set, a picture parameter set, and other grammatical structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The coded video data may be directly transmitted to the destination device 120 via the network 130A via the I / O interface 116. The coded video data may also be stored on a storage medium / server 130B for access by the destination device 120.
[0070] The destination device 120 may include an I / O interface 126, a video decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain encoded video data from the source device 110 or the storage medium / server 130B. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to the user. The display device 122 may be integrated with the destination device 120, or may be outside the destination device 120, which is configured to be connected to an external display device interface.
[0071] The video encoder 114 and the video decoder 124 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other existing and / or future standards.
[0072] Figure 2 is a block diagram showing an example of a video encoder 200 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video encoder 114 in the system 100 is shown.
[0073] Video encoder 200 may be configured to implement any or all of the techniques of this disclosure. Figure 2 In the example of , video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video encoder 200. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.
[0074] In some embodiments, the video encoder 200 may include a partitioning unit 201, a prediction unit 202, a residual generation unit 207, a transformation unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transformation unit 211, a reconstruction unit 212, a buffer 213 and an entropy coding unit 214, and the prediction unit 202 may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205 and an intra-frame prediction unit 206.
[0075] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, where at least one reference picture is a picture where the current video block is located.
[0076] Furthermore, although some components (such as the motion estimation unit 204 and the motion compensation unit 205) may be integrated, for the purpose of explanation, these components are described in detail below. Figure 2 are shown separately in the example.
[0077] The partitioning unit 201 may partition a picture into one or more video blocks. The video encoder 200 and the video decoder 300 (which will be discussed in detail below) may support various video block sizes.
[0078] The mode selection unit 203 may select one of a plurality of coding modes (intra-frame coding or inter-frame coding), for example, based on the error result, and provide the generated intra-frame coding block or inter-frame coding block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the coding block for use as a reference picture. In some examples, the mode selection unit 203 may select a combination of intra-frame and inter-frame prediction (CIIP) modes, where the prediction is based on an inter-frame prediction signal and an intra-frame prediction signal. In the case of inter-frame prediction, the mode selection unit 203 may also select a resolution for the motion vector (e.g., sub-pixel precision or integer pixel precision) for the block.
[0079] To perform inter-frame prediction on the current video block, the motion estimation unit 204 may generate motion information for the current video block by comparing the current video block with one or more reference frames from the buffer 213. The motion compensation unit 205 may determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from the buffer 213 other than the picture associated with the current video block.
[0080] The motion estimation unit 204 and the motion compensation unit 205 may perform different operations on the current video block, for example, depending on whether the current video block is in an I slice, a P slice, or a B slice. As used herein, an "I slice" may refer to a portion of a picture consisting of macroblocks, all of which are based on macroblocks within the same picture. Furthermore, as used herein, in some aspects, a "P slice" and a "B slice" may refer to a portion of a picture consisting of macroblocks that are independent of macroblocks in the same picture.
[0081] In some examples, the motion estimation unit 204 may perform unidirectional prediction on the current video block, and the motion estimation unit 204 may search the reference pictures of list 0 or list 1 to find the reference video block for the current video block. The motion estimation unit 204 may then generate a reference index and a motion vector, the reference index indicating the reference picture in list 0 or list 1 containing the reference video block, and the motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 204 may output the reference index, the prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.
[0082] Alternatively, in other examples, the motion estimation unit 204 may perform bidirectional prediction on the current video block. The motion estimation unit 204 may search the reference pictures in list 0 for a reference video block for the current video block, and may also search the reference pictures in list 1 for another reference video block for the current video block. The motion estimation unit 204 may then generate a plurality of reference indexes and a plurality of motion vectors, the plurality of reference indexes indicating a plurality of reference pictures in list 0 and list 1 containing a plurality of reference video blocks, and the plurality of motion vectors indicating a plurality of spatial displacements between the plurality of reference video blocks and the current video block. The motion estimation unit 204 may output the plurality of reference indexes and the plurality of motion vectors of the current video block as motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the plurality of reference video blocks indicated by the motion information of the current video block.
[0083] In some examples, motion estimation unit 204 may output a complete set of motion information for use in a decoding process by a decoder. Alternatively, in some embodiments, motion estimation unit 204 may signal motion information of a current video block with reference to motion information of another video block. For example, motion estimation unit 204 may determine that motion information of a current video block is sufficiently similar to motion information of a neighboring video block.
[0084] In one example, motion estimation unit 204 may indicate to video decoder 300 a value in a syntax structure associated with the current video block that indicates that the current video block has the same motion information as another video block.
[0085] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.
[0086] As discussed above, the video encoder 200 may signal motion vectors in a predictive manner.Two examples of prediction signaling techniques that may be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and merge mode signaling.
[0087] The intra prediction unit 206 may perform intra prediction on the current video block. When performing intra prediction on the current video block, the intra prediction unit 206 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a prediction video block and various syntax elements.
[0088] The residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) the prediction video block(s) of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample portions of samples in the current video block.
[0089] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 207 may not perform a subtraction operation.
[0090] Transform unit 208 may generate one or more transform coefficient video blocks for a current video block by applying one or more transforms to the residual video block associated with the current video block.
[0091] After transform unit 208 generates a transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values associated with the current video block.
[0092] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current video block for storage in the buffer 213.
[0093] After reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.
[0094] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the data is received, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy coded data and output a code stream including the entropy coded data.
[0095] Figure 3 is a block diagram showing an example of a video decoder 300 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video decoder 124 in the system 100 is shown.
[0096] Video decoder 300 may be configured to perform any or all of the techniques of this disclosure. Figure 3 In the example of , video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video decoder 300. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.
[0097] exist Figure 3 In the example of , the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, and a reconstruction unit 306 and a buffer 307. In some examples, the video decoder 300 can perform a decoding process that is generally opposite to the encoding process described with respect to the video encoder 200.
[0098] The entropy decoding unit 301 can retrieve the encoded code stream. The encoded code stream may include entropy encoded video data (e.g., encoded video data blocks). The entropy decoding unit 301 can decode the entropy encoded video data, and the motion compensation unit 302 can determine motion information from the entropy decoded video data, the motion information including motion vectors, motion vector precision, reference picture list indexes, and other motion information. The motion compensation unit 302 can determine the information, for example, by performing AMVP and merge mode. AMVP is used, including deriving several most likely candidates based on data and reference pictures of neighboring PBs. The motion information typically includes horizontal and vertical motion vector displacement values, one or two reference picture indexes, and in the case of prediction areas in B stripes, also includes an identification of which reference picture list is associated with each index. As used herein, in some aspects, "merge mode" may refer to deriving motion information from blocks that are adjacent in space or time.
[0099] The motion compensation unit 302 may generate motion compensated blocks, possibly performing interpolation based on interpolation filters.Identifiers for the interpolation filters used with sub-pixel precision may be included in the syntax elements.
[0100] The motion compensation unit 302 may calculate interpolated values for sub-integer pixels of a reference block using interpolation filters used by the video encoder 200 during encoding of the video block. The motion compensation unit 302 may determine the interpolation filters used by the video encoder 200 based on received syntax information, and the motion compensation unit 302 may use the interpolation filters to generate a prediction block.
[0101] The motion compensation unit 302 may use at least part of the syntax information to determine the size of blocks used to encode (multiple) frames and / or (multiple) slices of the encoded video sequence, partition information describing how each macroblock of a picture of the encoded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-coded block, and other information for decoding the encoded video sequence. As used herein, in some aspects, a "slice" may refer to a data structure that can be decoded independently of other slices of the same picture in terms of entropy coding, signal prediction, and residual signal reconstruction. A slice may be an entire picture, or it may be a region of a picture.
[0102] The intra prediction unit 303 may use, for example, an intra prediction mode received in a bitstream to form a prediction block from spatially neighboring blocks. The inverse quantization unit 304 inversely quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 305 applies an inverse transform.
[0103] The reconstruction unit 306 may obtain the decoded block, for example, by adding the residual block to the corresponding prediction block generated by the motion compensation unit 302 or the intra prediction unit 303. If necessary, a deblocking filter may also be applied to filter the decoded block in order to remove blocking artifacts. The decoded video block is then stored in a buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction, and the buffer 307 also generates the decoded video for presentation on a display device.
[0104] Some exemplary embodiments of the present disclosure will be described in detail below. It should be noted that the section titles used in this document are for ease of understanding, rather than limiting the embodiments disclosed in the section to that section. In addition, although some embodiments are described with reference to general video codecs or other specific video codecs, the disclosed technology is also applicable to other video codec technologies. In addition, although some embodiments describe the video encoding steps in detail, it should be understood that the corresponding decoding steps of the unencoding will be implemented by the decoder. In addition, the term video processing includes video encoding or compression, video decoding or decompression, and video transcoding, in which video pixels are represented from one compression format to another compression format or at different compression bit rates. 1. Brief Overview The present disclosure relates to video coding techniques. Specifically, it relates to a motion candidate list for inter-frame coding, which uses refined motion information, how and / or whether to apply a reordering process of a motion candidate list, and other coding tools in image / video coding to reorder the motion candidate list. It can be applied to existing video coding standards such as HEVC or universal video codec (VVC). It can also be applied to future video coding standards or video codecs. 1. Introduction Video coding standards have evolved primarily through the development of standards from the renowned International Telecommunication Union Telecommunication Standardization Sector (ITU-T) and the International Organization for Standardization / International Electrotechnical Commission (ISO / IEC). ITU-T developed H.261 and H.263, ISO / IEC developed MPEG-1 and MPEG-4 Vision, and the two organizations worked together to develop H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Coding (AVC) and H.265 / HEVC standards. Since H.262, video coding standards have been based on a hybrid video coding structure that utilizes temporal prediction plus transform coding. . To explore future video codec technologies beyond HEVC, the Video Coding Experts Group (VCEG) and the Moving Picture Experts Group (MPEG) jointly established the Joint Video Exploration Team (JVET) in 2015. . Since then, JVET has taken many new approaches and put them into a reference software called the Joint Exploration Model (JEM). In April 2018, the Joint Video Experts Group (JVET) between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) was established to develop the Versatile Video Coding (VVC) standard with a 50% bit rate reduction compared to HEVC. The latest version of the VVC draft, Versatile Video Coding (Draft 10), can be found at: http: / / phenix.it-sudparis.eu / jvet / doc_end_user / documents / 20_ Teleconference / wg11 / JVET-T2001-v1.zip The latest reference software for VVC (named VTM) can be found at: https: / / vcgit.hhi.fraunhofer.de / jvet / VVCSoftware_VTM / - / tags / VTM-11.0 2.1 Bit encoding process of typical video codec Figure 4 An example of an encoder block diagram for VVC is shown, which contains three in-loop filtering blocks: deblocking filter (DF), sample adaptive offset (SAO), and adaptive loop filter (ALF). Unlike DF, which uses a predefined filter, SAO and ALF use the original samples of the current picture to reduce the mean square error between the original samples and the reconstructed samples by adding an offset and by applying a finite impulse response (FIR) filter, respectively, while indicating the offset and filter coefficients with encoded side information. ALF is located at the last processing stage of each picture and can be seen as a tool that attempts to capture and repair artifacts created by previous stages. 2.2 Intra-mode coding with 67 intra-prediction modes To capture arbitrary edge directions present in natural videos, as used in HEVC, the number of directional intra modes is extended from 33 to 65, as Figure 5 , and planar and DC modes remain unchanged. These more dense directional intra prediction modes apply to all block sizes and luma and chroma intra prediction. In HEVC, each intra-coded block has a square shape with the length of each side being a power of 2. Therefore, no division operation is required to generate the intra predictor using the DC mode. In VVC, in general, blocks can have a rectangular shape requiring the use of a division operation for each block. To avoid the division operation for DC prediction, for non-square blocks, only the longer side is used to calculate the average. 2.2.1 Wide-angle intra prediction Although 67 modes are defined in VVC, the exact prediction direction for a given intra prediction mode index also depends on the block shape. Conventional angular intra prediction directions are defined from 45 degrees to 135 degrees clockwise. In VVC, for non-square blocks, several traditional angular intra prediction modes are adaptively replaced by wide-angle intra prediction modes. The replaced modes are indicated using the original mode indices, which are remapped to the indices of the wide-angle mode after parsing. The total number of intra prediction modes remains unchanged, i.e. 67, and the intra mode encoding method remains unchanged. To support these predictions, Figure 6 As shown, a top reference with a length of 2W+1 and a left reference with a length of 2H+1 are defined. The number of modes replaced in the wide-angle direction mode depends on the aspect ratio of the block. The replaced intra prediction modes are shown in Table 1. Table 1 Intra-frame prediction modes replaced by wide-angle mode Figure 7 A block diagram showing the discontinuity in the case where the orientation exceeds 45 degrees. Figure 7 As shown in FIG. 700 , in the case of wide-angle intra prediction, two vertically adjacent prediction samples can use two non-adjacent reference samples. Therefore, a low-pass reference sample filter and side smoothing are applied to wide-angle prediction to reduce the effect of the increased gap on Δp α negative impact. If the wide angle mode represents a non-fractional offset. There are 8 modes in the wide angle mode that meet this condition, which are [014, 912, 10, 96, 72, 76, 78, 80]. When predicting a block through these modes, the samples in the reference buffer are directly copied without applying any interpolation. With this modification, the number of samples that need to be smoothed is reduced. In addition, it keeps the design of the non-fractional mode in the traditional prediction mode and the wide angle mode consistent. In VVC, in addition to 4:2:0, 4:2:2 and 4:4:4 chroma formats are supported. The chroma derivative mode (DM) derivation table for the 4:2:2 chroma format was originally ported from HEVC, expanding the number of entries from 35 to 67 to keep pace with the expansion of intra prediction modes. Since the HEVC specification does not support prediction angles below -135 degrees and above 45 degrees, the luma intra prediction modes from 2 to 5 are mapped to 2. Therefore, the chroma DM derivation table for the 4:2:2 chroma format is updated by replacing some values of the mapping table entries to more accurately convert the prediction angles of the chroma blocks. 2.3 Inter-frame prediction For each inter-predicted CU, the motion parameters consist of a motion vector, a reference picture index and a reference picture list usage index, as well as additional information required by the new VVC coding features for generating inter-predicted samples. Motion parameters can be indicated explicitly or implicitly. When a CU is encoded or decoded in skip mode, the CU is associated with one PU and has no significant residual coefficients, no encoded motion vector increments or reference picture indices. Specify a merge mode, in which the motion parameters of the current CU are obtained from neighboring CUs, including spatial and temporal candidates, as well as other schemes introduced in VVC. Merge mode can be applied to any inter-predicted CU, not just skip mode. An alternative to merge mode is explicit transmission of motion parameters, where each CU explicitly indicates the motion vector, the reference picture index corresponding to each reference picture list, the reference picture list usage flag, and other required information. 2.4 Intra-block copy (IBC) Intra-block copying (IBC) is a tool adopted in the HEVC screen content coding (SCC) extension. It is well known that it significantly improves the coding efficiency of screen content material. Since the IBC mode is implemented as a block-level coding mode, block matching (BM) is performed at the encoder to find the best block vector (or motion vector) for each CU. Here, the block vector is used to indicate the displacement from the current block to the reference block that has been reconstructed inside the current picture. The luminance block vector of the CU encoded and decoded by IBC is integer precision. The chrominance block vector is also rounded to integer precision. When used in conjunction with AMVR, the IBC mode can switch between 1-pixel and 4-pixel motion vector precision. The IBC-encoded CU is regarded as a third prediction mode different from the intra or inter prediction mode. The IBC mode is applicable to CUs with width and height both less than or equal to 64 luminance samples. On the encoder side, hash-based motion estimation is performed for IBC. The encoder performs rate-distortion (RD) checks on blocks with a width or height no greater than 16 luma samples. For non-merge mode, a block vector search is first performed using a hash-based search. If the hash search does not return a valid candidate, a local search based on block matching is performed. In hash-based search, hash key matching (32-bit cyclic redundancy check (CRC)) between the current block and the reference blocks is extended to all allowed block sizes. The hash key calculation for each position in the current image is based on 4×4 sub-blocks. For larger-sized current blocks, when the hash keys of all 4×4 sub-blocks match the hash keys in the corresponding reference positions, a hash key is determined to match the hash key of the reference block. If multiple reference blocks are found whose hash keys match the hash key of the current block, the block vector cost of each matching reference is calculated and the one with the smallest cost is selected. In the block matching search, the search range is set to cover both the previous CTU and the current CTU. At the CU level, the IBC mode is represented by a flag, and it can be indicated as IBC AMVP mode or IBC Skip / Merge mode as described below: - IBC skip / merge mode: Use the merge candidate index to indicate which block vector from the list of neighboring candidate IBC coded blocks to use to predict the current block. The merge list consists of spatial, HMVP and pairwise candidates. - IBC AMVP mode: Block vector differences are encoded in the same way as motion vector differences. The block vector prediction method uses two candidates as predictors, one from the left neighbor and one from the top neighbor (if IBC is encoded and decoded). When either neighbor is not available, the default block vector will be used as the predictor. A flag is used to indicate the block vector predictor index. 2.5 Merge Mode with MVD (MMVD) In addition to the merge mode, a merge mode with motion vector difference (MMVD) is introduced in VVC when the implicitly derived motion information is directly used for prediction sample generation of the current CU. The MMVD flag is indicated immediately after the regular merge flag is sent to specify whether the MMVD mode is used for the CU. In MMVD, after selecting a merge candidate, it further indicates MVD information refinement. Further information includes a merge candidate flag, an index for specifying the magnitude of motion, and an index for indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the MV basis. The MMVD candidate flag is indicated to specify which one to use between the first and second merge candidates. The distance index specifies motion magnitude information and indicates a predefined offset from a starting point. Figure 8 Shows the MMVD search point. Figure 8 As shown, the offset is added to the horizontal or vertical component of the starting MV. The relationship between the distance index and the predefined offset is specified in Table 2-2. Table 2-2 - Relationship between distance index and predefined offset The direction index indicates the direction of the MVD relative to the starting point. The direction index can represent four directions, as shown in Table 2- Figure 3 As shown. It should be noted that the meaning of the MVD sign may vary depending on the information of the starting MV. When the starting MV is a non-predicted MV or a bidirectional predicted MV with two lists pointing to the same side of the current picture (i.e., the POCs of both references are greater than the POC of the current picture, or both are less than the POC of the current picture), the sign in Table 2-3 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectional predicted MV with two MVs to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture) and the POC difference in list 0 is greater than the POC in list 1, the sign in Table 2-3 specifies the sign of the MV offset added to the list 0 MV component of the starting MV, and the sign of the list 1 MV has the opposite value. Otherwise, if the POC difference in list 1 is greater than list 0, the sign in Table 2-3 specifies the sign of the MV offset added to the list 1 MV component of the starting MV, and the sign of the list 0 MV has the opposite value. The MVD is scaled according to the difference in POC in each direction. If the difference in POC in both lists is the same, no scaling is required. Otherwise, if the POC difference in list 0 is greater than the one in list 1, then the MVD of list 1 is scaled by defining the POC difference for L0 as td and the POC difference for L1 as tb, as in Fig. 9 If the POC difference of L1 is greater than L0, then the MVD of list 0 is scaled in the same way. If the starting MV is unidirectionally predicted, then the MVD is added to the available MVs. Table 2-3 - Signs of MV offsets specified by direction index Directions to IDX 00 01 10 11 x-axis + - N / A N / A y-axis N / A N / A + - 2.6 Symmetric MVD Codec In VVC, in addition to normal uni-prediction and bi-prediction mode MVD signaling, symmetric MVD mode of bi-prediction MVD signaling is applied. In symmetric MVD mode, motion information including both reference picture indexes of list 0 and list 1 and MVD of list 1 is not indicated, but motion information is derived. The decoding process of the symmetric MVD mode is as follows: 1. At the slice level, the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 are derived as follows: – If mvd_l1_zero_flag is 1, then BiDirPredFlag is set equal to 0. – Otherwise, if the nearest reference picture in list-0 and the nearest reference picture in list-1 form a forward and backward reference picture or a backward and forward reference picture pair, then BiDirPredFlag is set to 1, and both list-0 and list-1 reference pictures are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. 2. At the CU level, if the CU is bidirectionally predicted and BiDirPredFlag is equal to 1, the symmetric mode flag explicitly indicates whether the symmetric mode is used. When the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag and MVD0 are explicitly indicated. The reference index of list 0 and list 1 are set equal to the reference picture pair respectively. MVD1 is set equal to (-MVD0). The final motion vector is shown in the following formula: Fig. 9 is an illustration of the symmetric MVD mode. In the encoder, symmetric MVD motion estimation starts with an initial MV evaluation. The initial MV candidate set includes MVs obtained from unidirectional prediction search, MVs obtained from bidirectional prediction search, and MVs from the AMVP list. The MV with the lowest rate-distortion cost is selected as the initial MV for the symmetric MVD motion search. 2.7 Bidirectional Optical Flow (BDOF) The Bidirectional Optical Flow (BDOF) tool is included in VVC. BDOF, previously known as BIO, is included in JEM. BDOF in VVC is a simpler version that requires less computation, especially in terms of the number of multiplications and the size of the multipliers, compared to the JEM version. BDOF is used to refine the bidirectional prediction signal of a CU at the 4×4 sub-block level. BDOF is applied to a CU if the CU meets all of the following conditions: – The CU is encoded using “true” bi-prediction mode, ie, one of the two reference pictures precedes the current picture in display order, and the other follows the current picture in display order. – The distances from the two reference pictures to the current picture (ie, the POC difference) are the same. – Both reference images are short-term reference images. –Do not use affine mode or SbTMVP merge mode to encode or decode the CU. – The CU has more than 64 luma samples. – Both CU height and CU width are greater than or equal to 8 luma samples. – BCW weight index indicates equal weight. –Do not enable WP for the current CU. –CIIP mode is not used for the current CU. BDOF is applied only to the luma component. As the name suggests, the BDOF mode is based on the concept of optical flow, which assumes that the motion of objects is smooth. For each 4×4 sub-block, the motion refinement (v x ,v y ). Then, motion refinement is used to adjust the bidirectional prediction sample values in the 4×4 sub-block. The following steps are applied in the BDOF process. First, the horizontal gradient of the two prediction signals is calculated by directly calculating the difference between two adjacent samples and vertical gradient k=0,1, that is: Among them I (k) (i, j) is the sample value of the prediction signal at coordinate (i, j) in list k, k = 0, 1, and shift1 is calculated based on the brightness bit depth bitDepth, that is, shift1 = max(6, bitDepth-6). Then, the autocorrelation and cross-correlation S1, S2, S3, S5 and S6 of the gradient are calculated as follows: S1=∑ (i,j)∈Ω Abs(ψ x (i,j)),S3=∑ (i,j)∈Ω θ(i,j)·Sign(ψ x (i,j)) (2-3) S5=∑ (i,j)∈Ω Abs(ψ y (i,j)),S6=∑ (i,j)∈Ω θ(i,j)·Sign(ψ y (i,j)) in θ(i,j)=(I (1) (i,j)>>n b )-(I (0) (i,j)>>n b ) where Ω is a 6×6 window around the 4×4 sub-block, n a and n b The values are equal to min(1, bitDepth-11) and min(4, bitDepth-8) respectively. The motion refinement (v) is then derived using the cross and autocorrelation terms using the following formula x ,v y): in th′ BIO =2 max(5,BD-7) , is a floor function, and Based on the motion refinement and gradients, the following adjustments are computed for each sample in the 4×4 sub-block: Finally, the BDOF samples of the CU are calculated by adjusting the bidirectional prediction samples as follows: pred BDOF (x,y)=(I (0) (x,y)+I (1) (x,y)+b(x,y)+o offset )>>shift (2-7) These values are chosen so that the multipliers in the BDOF process do not exceed 15 bits and the maximum bit width of the intermediate parameters in the BDOF process remains within 32 bits. In order to derive the gradient value, it is necessary to generate some prediction samples I in the list k (k = 0, 1) outside the current CU boundary (k) (i,j); Fig.10 FIG. 4 is a schematic diagram showing an extended CU region used in BDOF; FIG. Fig.10 As shown in FIG. 1000 , BDOF in VVC uses a row / column extension region around the boundary of a CU. In order to control the computational complexity of generating out-of-bounds prediction samples, the prediction samples in the extension region (in Fig.10 1010 in CU) is generated by directly obtaining reference samples at nearby integer positions (using floor function operation on coordinates) without interpolation, and a normal 8-tap motion compensation interpolation filter is used to generate prediction samples inside the CU (in Fig.10 These extended sample values are only used for gradient calculations. For the remaining steps in the BDOF process, if any samples and gradient values outside the CU boundary are needed, they are filled (i.e. repeated) from their nearest neighbors. When the width and / or height of a CU is greater than 16 luma samples, it will be divided into sub-blocks with a width and / or height equal to 16 luma samples, and the sub-block boundaries are regarded as CU boundaries in the BDOF process. The maximum unit size for the BDOF process is limited to 16×16. The BDOF process can be skipped for each sub-block. When the SAD between the initial L0 and L1 prediction samples is less than a threshold, the BDOF process is not applied to the sub-block. The threshold is set equal to (8*W*(H>>1), where W indicates the sub-block width and H indicates the sub-block height. To avoid the additional complexity of the SAD calculation, the SAD between the initial L0 and L1 prediction samples calculated in the DVMR process is used again here. If BCW is enabled for the current block, that is, the BCW weight index indicates unequal weights, then bidirectional optical flow will be disabled. Similarly, if weighted prediction (WP) is enabled for the current block, that is, the luma weight flag (luma_weight_lx_flag) is 1 for any of the two reference images, then bidirectional optical flow will also be disabled. Bidirectional optical flow will also be disabled when the CU is encoded in MVD mode or CIIP mode. 2.8 Joint Inter- and Intra-frame Prediction (CIIP) 2.9 Affine Motion Compensated Prediction In HEVC, only the translational motion model is applied to motion compensated prediction (MCP). In the real world, there are many kinds of motions, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensated prediction is applied. Fig.11 As shown, the affine motion field of a block is described by the motion information of two control points (4 parameters) or three control point motion vectors (6 parameters). for Fig.11 Using the 4-parameter affine motion model 1110 in , the motion vector at the sample position (x, y) in the block is derived as: for Fig.11 Using the 6-parameter affine motion model 1120 in , the motion vector at the sample position (x, y) in the block is derived as: Where (mv 0x ,mv 0y ) is the motion vector of the upper left control point, (mv 1x ,mv 1y ) is the motion vector of the upper right control point, and (mv 2x ,mv 2y ) is the motion vector of the lower left control point. To simplify motion compensated prediction, block-based affine transform prediction is applied. Fig.12 A schematic diagram 1200 of the affine MVF of each sub-block is shown. To derive the motion vector of each 4×4 luminance sub-block, the motion vector of the center sample of each sub-block is calculated according to the above equation, such as Fig.12 As shown, and rounded to 1 / 16 fractional precision. A motion compensated interpolation filter is then applied to generate a prediction for each subblock with a derived motion vector. The subblock size of the chroma component is also set to 4×4. The MV of a 4×4 chroma subblock is calculated as the average of the MVs of the four corresponding 4×4 luminance subblocks. For translational motion inter prediction, there are also two affine motion inter prediction modes: affine merge mode and affine AMVP mode. 2.9.1 Affine Merge Prediction AF_MERGE mode can be applied to CUs whose width and height are both greater than or equal to 8. In this mode, the CPMV of the current CU is generated based on the motion information of the spatially neighboring CUs. There may be up to five CPMVP candidates, and the index is indicated to indicate the CPMVP candidate to be used for the current CU. The following three types of CPMV candidates are used to form the affine merge candidate list: – Inherited affine merge candidates, which are extrapolated from the CPMV of neighboring CUs. – Constructed affine merge candidate CPMVP derived using the translation MV of the neighboring CU – Zero MV. In VVC, there are a maximum of two inherited affine candidates derived from the affine motion models of neighboring blocks, one from the left neighboring CU and one from the neighboring CU above. Fig.13 Schematic diagram 1300 showing the location of the inherited affine motion predictor. The candidate blocks are Fig.13 As shown in . For the left predictor, the scan order is A0->A1, and for the above predictors, the scan order is B0->B1->B2. Only the first inherited candidate is selected from each side. No pruning check is performed between two inherited candidates. When a neighboring affine CU is identified, its control point motion vector is used to derive the CPMVP candidate in the affine merge list of the current CU. Fig.14 Schematic diagram 1400 showing inheritance of control point motion vectors. Fig.14 As shown, if the neighboring left bottom block A 1410 is encoded in an affine mode, motion vectors v2, v3, and v4 of the upper left corner, upper right corner, and lower left corner of the CU 1420 containing the block A 1410 are obtained. When the block A 1410 is encoded using a 4-parameter affine model, two CPMVs of the current CU are calculated according to and. In the case where the block A is encoded using a 6-parameter affine model, three CPMVs of the current CU are calculated according to v2, v3, and v4. Constructing affine candidates means constructing candidates by combining the neighboring translation motion information of each control point. The motion information of the control point is obtained from Fig.15 The specified spatial neighbors and temporal neighbors are derived as shown in Fig.15 A schematic diagram 1500 is shown showing the locations of candidate locations for constructing an affine merge pattern. k (k=1, 2, 3, 4) represents the kth control point. For CPMV1, check B2->B3->A2 blocks and use the MV of the first available block. For CPMV2, check B1>B0 blocks, and for CPMV3, check A1>A0 blocks. For TMVP, if available, use TMVP as CPMV4. After obtaining the MVs of the four control points, an affine merge candidate is constructed based on the motion information. The following combinations of control point MVs are used to construct in order: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}. The combination of 3 CPMVs constructs a 6-parameter affine merge candidate, and the combination of 2 CPMVs constructs a 4-parameter affine merge candidate. To avoid the motion scaling process, the relevant combination of control point MVs is discarded if the reference indexes of the control points are different. After checking inherited affine merge candidates and constructed affine merge candidates, if the list is still not complete, a zero MV is inserted to the end of the list. 2.9.2 Affine AMVP Prediction Affine AMVP mode can be applied to CUs with width and height both greater than or equal to 16. An affine flag in the CU level is indicated in the bitstream to indicate whether the affine AMVP mode is used, and then another flag is indicated to indicate whether 4-parameter affine or 6-parameter affine. In this mode, the difference between the CPMV of the current CU and its predicted value CPMVP is signaled in the bitstream. The affine AVMP candidate list size is 2, and is generated in sequence by the following four types of CPMV candidates: – Inherited affine AMVP candidates, which are extrapolated from the CPMV of neighboring CUs. – Constructed affine AMVP candidate CPMVP, which is derived using the translated MV of the neighboring CU. – Translated MV from neighboring CU. – Zero MV. The order of checking inherited affine AMVP candidates is the same as that of checking inherited affine merge candidates. The only difference is that for AVMP candidates, only affine CUs with the same reference picture as in the current block are considered. When an inherited affine motion predictor is inserted into the candidate list, the pruning process is not applied. The AMVP candidates are constructed from Fig.13 Derived from the specified spatial neighbors shown in . The same check order is used in the affine merge candidate construction. In addition, the reference picture index of the neighboring blocks is also checked. The first block in the check order that uses inter-frame coding and has the same reference picture as in the current CU. Only when the current CU is encoded with a 4-parameter affine mode and both mv0 and mv1 are available, they are added as a candidate in the affine AMVP list. When the current CU is encoded and decoded using a 6-parameter affine mode and all three CPMVs are available, it is added as a candidate in the affine AMVP list. Otherwise, the constructed AMVP candidate is set to unavailable. If the affine AMVP list candidate is still less than 2 after checking the inherited affine AMVP candidates and the constructed AMVP candidates, mv0, mv1, and mv2 will be added as translation MVs to predict all control point MVs of the current CU when available. Finally, if the affine AMVP list is still not full, the affine AMVP list is filled with zero MVs. 2.9.3 Affine Motion Information Storage In VVC, the CPMV of an affine CU is stored in a separate buffer. The stored CPMV is only used to generate the inherited CPMVP in the affine merge mode, and to generate the affine AMVP mode in the separately encoded CU. The sub-block MV derived from the CPMV is used for motion compensation, merging of translation MVs / MV derivation of the AMVP list, and deblocking. To avoid extra CPMVs for the picture line buffer, the affine motion data inherited from the CU of the upper CTU is treated differently from the inheritance from the normal neighboring CU. If the candidate CU for affine motion data inheritance is in the above CTU row, the lower left and lower right sub-block MVs in the row buffer are used for affine MVP derivation instead of the CPMVs. In this way, the CPMVs are only stored in the local buffer. If the candidate CU is 6-parameter affine coded, the affine model is downgraded to a 4-parameter model. Fig.16 As shown, along the top CTU boundary, the bottom left and bottom right sub-block motion vectors of the CU are used for affine inheritance of the CU in the bottom CTU. 2.9.4 Prediction Refinement with Optical Flow for Affine Mode Compared with pixel-based motion compensation, sub-block-based affine motion compensation can save memory access bandwidth and reduce computational complexity, at the cost of prediction accuracy loss. In order to achieve finer motion compensation granularity, prediction refinement using optical flow (PROF) is used to refine sub-block-based affine motion compensation predictions without increasing the memory access bandwidth for motion compensation. In VVC, after performing sub-block-based affine motion compensation, the brightness prediction samples are refined by adding the differences derived from the optical flow equation. PROF is described as the following four steps: Step 1) Perform sub-block based affine motion compensation to generate sub-block prediction I(i,j). Step 2) Calculate the spatial gradient g at each sampling position using a 3-tap filter [1,0,1] x (i,j) and sub-block prediction g y (i,j). The gradient calculation is exactly the same as that in BDOF. g x (i,j)=(I(i+1,j)>>shift1)-(I(i-1,j)>>shift1) (2-10) g y (i,j)=(I(i,j+1)>>shift1)-(I(i,j-1)>>shift1) (2-11) Shift1 is used to control the accuracy of the gradient. For gradient calculation, the sub-block (i.e. 4×4) prediction is extended by one sample on each side. To avoid additional memory bandwidth and additional interpolation calculations, those extended samples on the extended boundary are copied from the nearest integer pixel position in the reference picture. Step 3) Calculate the brightness prediction refinement by following the optical flow equation. ΔI(i,j)=g x (i,j)*Δv x (i,j)+g y (i,j)*Δv y (i,j) (2-12) like Fig.17 As shown in , where Δv(i,j) is the difference between the sample MV (denoted by v(i,j)) calculated for the sample position (i,j) and the sub-block MV of the sub-block to which the sample (i,j) belongs. Δv(i,j) is quantized in units of 1 / 32 lehm sample accuracy. Since the affine model parameters and the sample positions relative to the sub-block center do not change from sub-block to sub-block, Δv(i,j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. dx(i,j) and dy(i,j) represent the distance from the sample position (i,j) to the sub-block (x SB ,y SB) can be derived from the following equations: To maintain accuracy, the sub-block (x SB ,y SB ) is calculated as ((W SB -1) / 2,(H SB -1) / 2), where W SB and H SB are the sub-block width and height respectively. For the 4-parameter affine model, For the 6-parameter affine model, Where (v 0x ,v 0y )、(v 1x ,v 1y )、(v 2x ,v 2y ) are the upper-left, upper-right, and lower-left control point motion vectors, and are the width and height of the CU. Step 4) Finally, the luma prediction correction ΔI(i,j) is added to the sub-block prediction I(i,j). The final prediction I' is produced as the following equation. I′(i,j)=I(i,j)+ΔI(i,j) (2-17) PROF is not applicable to two cases of affine-coded CUs: 1) all control point MVs are the same, indicating that the CU has only translational motion; 2) the affine motion parameters are larger than the specified limit, because the sub-block based affine MC is downgraded to CU based MC to avoid large memory access bandwidth requirements. A fast coding method is applied to reduce the coding complexity of affine motion estimation with PROF. PROF is not applied in the following two cases at the affine motion estimation stage: a) If the CU is not a root block and its parent block does not select the affine mode as its best mode, PROF is not applied because the possibility that the current CU selects the affine mode as the best mode is low; b) If the magnitudes of the four affine parameters (C, D, E, F) are all less than a predefined threshold and the current picture is not a low-latency picture, PROF is not applied because the improvement introduced by PROF is small for this case. In this way, affine motion estimation with PROF can be accelerated. 2.10 Sub-block based temporal motion vector prediction (Sb TMVP) VVC supports a sub-block based temporal motion vector prediction (Sb TMVP) method. Similar to the temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the co-located picture to improve the motion vector prediction and merge mode of the CU in the current picture. The same co-located pictures used by TMVP are used for SbTMVP. SbTMVP differs from TMVP in the following two main aspects: –TMVP predicts motion at CU level, but SbTMVP predicts motion at sub-CU level; –While TMVP extracts the temporal motion vector from the co-located block in the co-located picture (the co-located block is the bottom-right or center block relative to the current CU), SbTMVP applies motion shifting before extracting temporal motion information from the co-located picture, where the motion shifting is obtained from the motion vector from one of the spatially neighboring blocks of the current CU. SbTMVP process Fig.18a and Fig.18b Shown in. Fig.18a Schematic diagram 1810 of the spatial neighboring blocks used by SbTMVP is shown. SbTMVP predicts the motion vector of the sub-CU within the current CU in two steps. In the first step, check Fig.18a If A1 has a motion vector that uses the co-located picture as its reference picture, then this motion vector is selected as the motion shift to be applied. If this motion is not identified, the motion shift is set to (0,0). Fig.18b Schematic diagram of deriving a sub-CU motion field by applying motion shifts from spatial neighbors and scaling motion information from corresponding co-located sub-CUs. Fig.18b As shown, the motion offset identified in step 1 is applied (ie, added to the coordinates of the current block in the current picture 1820 ) to obtain sub-CU level motion information (motion vector and reference index) from the co-located picture 1822 . Fig.18b The example in assumes that the motion shift is set to block A1" motion. Then, for each sub-CU, the motion information of its corresponding block (the minimum motion grid covering the center sample) in the co-located picture 1822 is used to derive the motion information of the sub-CU. After the motion information of the collimated sub-CU is identified, it is converted into a motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference picture of the temporal motion vector with the reference picture of the current CU. In VVC, a merge list of combined subblocks containing SbTMVP candidates and affine merge candidates is used for signaling of subblock-based merge mode. The SbTMVP mode is enabled / disabled by a sequence parameter set (SPS) flag. If SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry in the list of subblock-based merge candidates, followed by the affine merge candidates. The size of the subblock-based merge list is signaled in the SPS, and the maximum allowed size of the subblock-based merge list in VVC is 5. The sub-CU size used in SbTMVP is fixed to 8×8, and for the affine merge mode, the SbTMVP mode is only applicable to CUs whose width and height are both greater than or equal to 8. The encoding logic of the additional SbTMVP merge candidate is the same as other merge candidates, that is, for each CU in a P or B slice, an additional RD check is performed to decide whether to use the SbTMVP candidate. 2.11 Adaptive Motion Vector Resolution (AMVR) In HEVC, when use_integer_mv_flag is equal to 0 in the slice header, the motion vector difference (MVD) (between the CU's motion vector and the predicted motion vector) is signaled in units of quarter luma samples. In VVC, a CU-level adaptive motion vector resolution (AMVR) scheme is introduced. AMVR allows encoding and decoding the MVD of a CU with different precisions. Depending on the mode of the current CU (normal AMVP mode or affine AMVP mode), the MVD of the current CU can be adaptively selected as follows: – Normal AMVP mode: quarter block samples, half block samples, integer block samples, or quad block samples. – Affine AMVP mode: quarter luma samples, integer luma samples, or 1 / 16 luma samples. If the current CU has at least one non-zero MVD component, the CU level MVD resolution indication is conditionally signaled.If all MVD components (ie, both horizontal and vertical MVDs for reference list L0 and reference list L1) are zero, then quarter luma sample MVD resolution is inferred. For CUs with at least one non-zero MVD component, a first flag is indicated to indicate whether quarter luma sample MVD precision is used for the CU. If the first flag is 0, no further signaling is required and quarter luma sample MVD precision is used for the current CU. Otherwise, a second flag is signaled to indicate that half-block samples or other MVD precision (integer or four luma samples) are used for normal AMVP CUs. In the case of half-block samples, a 6-tap interpolation filter is used instead of the default 8-tap interpolation filter for half-luminance sample positions. Otherwise, a third flag is indicated to indicate whether integer luma sample or four luma sample MVD precision is used for normal AMVP CUs. For the case of affine AMVP CUs, the second flag is used to indicate whether integer luma sample or 1 / 16 luma sample MVD precision is used. In order to ensure that the reconstructed MV has the expected precision (quarter luma sample, half luma sample, integer luma sample, or four luma sample), the motion vector predictor of the CU is rounded to the same precision as the MVD before being added with the MVD. The motion vector predictor is rounded towards zero (ie, negative motion vector predictors are rounded towards positive infinity, and positive motion vector predictors are rounded towards negative infinity). The encoder uses RD check to determine the motion vector resolution of the current CU. In order to avoid always performing CU-level RD check four times for each MVD resolution, in VTM11, only the RD check of MVD precision other than quarter brightness samples is conditionally called. For normal AVMP mode, the RD cost of quarter brightness sample MVD precision and integer brightness sample MV precision is first calculated. Then, the RD cost of integer brightness sample MVD precision is compared with the RD cost of quarter brightness sample MVD precision to determine whether the RD cost of four-block sample MVD precision needs to be further checked. When the RD cost of quarter brightness sample MVD precision is much smaller than the RD cost of integer brightness sample MVD precision, the RD check of four-block sample MVD precision is skipped. Next, if the RD cost of integer brightness sample MVD precision is significantly greater than the best RD cost of the previously tested MVD precision, the check of half-block sample MVD precision is skipped. For the affine AMVP mode, if the affine inter mode is not selected after checking the rate-distortion cost of the affine merge / skip mode, merge / skip mode, quarter luma sample MVD precision normal AMVP mode, and quarter luma sample MVD precision affine AMVP mode, the 1 / 16 luma sample MV precision and 1 pixel MV precision affine inter modes are not checked. In addition, the affine parameters obtained in the quarter luma sample MV precision affine inter mode are used as the starting search points in the 1 / 16 luma sample and quarter luma sample MV precision affine inter modes. 2.12 Bidirectional Prediction with CU Level Weight (BCW) In HEVC, a bidirectional prediction signal is generated by averaging two prediction signals obtained from two different reference pictures and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging to allow a weighted average of the two prediction signals. P bi-pred =((8-w)*P0+w*P1+4)>>3 (2-18) Five weights, w∈{-2,3,4,5,10}, are allowed in weighted average bidirectional prediction. For each bidirectional prediction CU, the weight w is determined in one of two ways: 1) For non-merged CUs, the weight index is signaled after the motion vector difference; 2) For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applied to CUs with 256 or more luma samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency pictures, all 5 weights are used. For non-low-latency pictures, only 3 weights (w∈{3,4,5}) are used. – At the encoder, fast search algorithms are applied to find the weight index without significantly increasing the encoder complexity. These algorithms are summarized below. For further details, the reader is referred to VTM software and document JVET-L0646. When combined with AMVR, unequal weights are only conditionally checked for 1-pixel and 4-pixel motion vector precision if the current picture is a low-latency picture. – When combined with affine, affine ME will be performed for unequal weights if and only if the affine mode is selected as the current best mode. – When the two reference pictures in bidirectional prediction are the same, unequal weights are only checked conditionally. – Unequal weights are not searched when certain conditions are met, which depend on the POC distance between the current picture and its reference pictures, the codec QP, and the temporal level. The BCW weight index is encoded using one context coding bin followed by a bypass coding bin. The first context coding bin indicates whether equal weights are used; and if unequal weights are used, an additional bin is signaled using bypass coding to indicate the use of unequal weights. Weighted prediction (WP) is a coding tool supported by the H.264 / AVC and HEVC standards for efficiently encoding video content with gradient colors. Support for WP has also been added to the VVC standard. WP allows weighting parameters (weights and offsets) to be signaled for each reference picture in each of the reference picture lists L0 and L1. Then, during motion compensation, the weights and offsets of the corresponding reference pictures are applied. WP and BCW are designed for different types of video content. In order to avoid the interaction between WP and BCW (which will complicate the VVC decoder design), if the CU uses WP, the BCW weight index is not sent, and w is inferred to be 4 (ie, equal weights are applied). For the merged CU, the weight index is inferred from the neighboring blocks based on the merge candidate index. This can be applied to the normal merge mode and the inherited affine merge mode. For the constructed affine merge mode, affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index of the CU using the constructed affine merge mode is simply set equal to the BCW index of the first control point MV. In VVC, CIIP and BCW cannot be applied jointly to a CU. When a CU is encoded in CIIP mode, the BCW index of the current CU is set to 2, such as equal weight. 2.13 Local Illumination Compensation (LIC) Local illumination compensation (LIC) is a codec tool used to address the problem of local illumination changes between a current picture and its temporal reference picture. LIC is based on a linear model where a scaling factor and an offset are applied to a reference sample to obtain a predicted sample for the current block. Specifically, LIC can be mathematically modeled by the following equation: P(x,y)=α·P r (x+v x ,y+v y )+β Where P(x,y) is the prediction signal of the current block at the coordinate (x,y); P r (x+v x ,y+v y ) is the motion vector (v x ,x y ) points to the reference block; α and β are the corresponding scaling factor and offset applied to the reference block. Fig.19 The LIC process is shown in FIG. Fig.19 In , when LIC is applied to a block, the minimum mean square error (LMSE) method is used to derive the values of the LIC parameters (i.e., α and β) by minimizing the neighboring samples of the current block (i.e., Fig.19 The template T in the temporal reference picture) and its corresponding reference sample in the temporal reference picture (ie, Fig.19In addition, in order to reduce the computational complexity, both the template samples and the reference template samples are subsampled (adaptive subsampling) to derive the LIC parameters, that is, only Fig.19 α and β are derived from the shadow samples in . like Fig. 20 As shown, in order to improve the coding performance, the short side is not sub-sampled. 2.14 Decoder-side Motion Vector Refinement (DMVR) To improve the accuracy of the MV in merge mode, decoding-side motion vector refinement based on bilateral matching (BM) is applied in VVC. In the bidirectional prediction operation, the refined MV is searched around the initial MV in the reference picture list L0 and the reference picture list L1. The BM method calculates the distortion between the 2 candidate blocks in the reference picture list L0 and the list L1. Fig.21 Schematic diagram for explaining the refinement of motion vectors at the decoding side. Fig.21 As shown, the SAD between blocks 2110 and 2112 based on each MV candidate around the initial MV is calculated, where block 2110 is in reference picture 2101 in list L0, and block 2112 is in reference picture 2103 in list L1 of current picture 2102. The MV candidate with the lowest SAD becomes the refined MV and is used to generate a bidirectional prediction signal. In VVC, the application of DMVR is restricted and only applies to CUs coded in the following modes and features: – CU level merge mode has bi-prediction MV. – One reference picture is in the past and the other reference picture is in the future relative to the current picture. – The distances from the two reference pictures to the current picture (ie, the POC difference) are the same. – Both reference images are short-term reference images. – The CU has more than 64 luma samples. – Both CU height and CU width are greater than or equal to 8 luma samples. – BCW weight index indicates equal weight. – Do not enable WP for the current block. –CIIP mode is not used for the current block. The refined MV derived by the DMVR process is used to generate inter-frame prediction samples and is also used in temporal motion vector prediction for future picture encoding. The original MV is used in the deblocking process and is also used in spatial motion vector prediction for future CU encoding and decoding. Additional features of DMVR are mentioned in the following sub-clauses. 2.14.1 Search Scheme In DVMR, the search point is around the initial MV, and the MV offset obeys the MV difference mirror rule. In other words, any point checked by DMVR (represented by the candidate MV pair (MV0, MV1)) obeys the following two equations: MV0′=MV0+MV offset (2-19) MV1′=MV1-MV offset (2-20) Where V_offset represents the refinement offset between the initial MV and the refined MV in one of the reference pictures. The refinement search range is two integer luma samples from the initial MV. The search includes an integer sample offset search stage and a fractional sample refinement stage. A 25-point full search is applied to the integer sample offset search. The SAD of the initial MV pair is first calculated. If the SAD of the initial MV pair is less than the threshold, the integer sampling stage of DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search stage. In order to reduce the penalty of uncertainty in DMVR refinement, it is proposed to prefer the original MV during the DMVR process. The SAD between the reference blocks referenced by the initial MV candidates is reduced by 1 / 4 of the SAD value. The integer sample search is followed by fractional sample refinement. To save computational complexity, fractional sample refinement is derived by using the parameter error surface equation instead of an additional search using SAD comparison. Fractional sample refinement is conditionally called based on the output of the integer sample search stage. When the integer sample search stage terminates at the center with the minimum SAD in the first or second iteration search, fractional sample refinement is further applied. In the parametric error surface based sub-pixel offset estimation, the center location cost and the costs at four neighboring locations from the center are used to fit a 2-D parabolic error surface equation of the following form: E(x,y)=A(x min ) 2 +B(yy min ) 2 +C (2-21) Where (x min ,y min ) corresponds to the fractional position of the minimum cost and C corresponds to the minimum cost value. By solving the above equation using the cost values of the five search points, (x min ,y min ) is calculated as: x min =(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0))) (2-22) y min =(E(0,-1)-E(0,1)) / (2((E(0,-1)+E(0,1)-2E(0,0))) (2-23) x min and min The value of is automatically constrained between -8 and 8, since all cost values are positive and the minimum is E(0,0). This corresponds to a halftone shift with 1 / 16 pixel MV precision in VVC. The calculated fractional part (x min ,y min ) is added to the integer distance refined motion vector to obtain a refined incremental motion vector with sub-pixel accuracy. 2.14.2 Linear interpolation and sample filling In VVC, the resolution of MV is 1 / 16 luma sample. Samples at fractional positions are interpolated using an 8-tap interpolation filter. In DMVR, the search points are around the initial fractional pixel MV with an integer sample offset, so samples at those fractional positions need to be interpolated for the DMVR search process. In order to reduce computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in DMVR. Another important effect is that by using a bilinear filter with a 2-sample search range, DVMR does not access more reference samples than the normal motion compensation process. After the DMVR search process obtains the refined MV, a conventional 8-tap interpolation filter is applied to generate the final prediction. In order not to access more reference samples to the normal MC process, samples that are not needed for the interpolation process based on the original MV but needed for the interpolation process based on the refined MV will be filled from those available samples. 2.14.3 Maximum DMVR Processing Unit When the width and / or height of a CU is greater than 16 luma samples, it will be further split into sub-blocks with width and / or height equal to 16 luma samples. The maximum unit size of the DMVR search process is limited to 16×16. 2.15 Multi-pass decoding end motion vector refinement In this contribution, multi-pass decode-side motion vector refinement is applied instead of DMVR. In the first pass, bilateral matching (BM) is applied to the coding block. In the second pass, BM is applied to each 16×16 sub-block within the coding block. In the third pass, MVs in each 8×8 sub-block are refined by applying bidirectional optical flow (BDOF). The refined MVs are stored for both spatial and temporal motion vector prediction. 2.15.1 First pass - sub-block-based bidirectional optical flow MV refinement In the first pass, refined MVs are derived by applying BM to the coded blocks. Similar to decoding-side motion vector refinement (DMVR), refined MVs are searched around two initial MVs (MV0 and MV1) in reference picture lists L0 and L1. Refined MVs (MV0_pass1 and MV1_pass1) are derived around the starting MVs based on the minimum bilateral matching cost between 2 reference blocks in L0 and L1. BM performs a local search to derive integer sample precision intDeltaMV and half-pixel sample precision halfDeltaMv. The local search applies a 3×3 square search pattern to cycle through the search range [-sHor, sHor] in the horizontal direction and [-sVer, sVer] in the vertical direction, where the values of sHor and sVer are determined by the block size, and the maximum values of sHor and sVer are 8. The bilateral matching cost is calculated as: bilCost = mvDistanceCost + sadCost. When the block size cbW*cbH is greater than 64, the MRSAD cost function is applied to remove the DC effect of the distortion between the reference blocks. When the bilinear at the center point of the 3×3 search pattern has the minimum cost, the intDeltaMV or halfDeltaMV local search is terminated. Otherwise, the current minimum cost search point becomes the new center point of the 3×3 search pattern, and the search for the minimum cost continues until it reaches the end of the search range. The existing fractional sample refinement is further applied to derive the final delta MV. The refined MV after the first pass is then derived as: MV0_pass1 = MV0 + deltaMV, MV1_pass1=MV1-deltaMV. 2.15.2 Second pass - sub-block-based bidirectional optical flow MV refinement In the second pass, refined MVs are derived by applying BM to 16×16 grid sub-blocks. For each sub-block, the refined MVs are searched around the 3 MVs (MV0_pass1 and MV1_pass1) for reference picture lists L0 and L1. Refined MVs (MV0_pass2(sb Idx2) and MV1_pass2(sb Idx2)) are derived based on the minimum bilateral matching cost between the 2 reference sub-blocks in L0 and L1. For each subblock, BM performs a full search to derive integer sample precision intDeltaMV. The full search has a search range of [-sHor, sHor] in the horizontal direction and [-sVer, sVer] in the vertical direction, where the values of sHor and sVer are determined by the block size and the maximum values of sHor and sVer are 8. The bilateral matching cost is calculated by applying the cost factor to the SATD cost between the two reference sub-blocks, such as: bilCost = satdCost * costFactor. The search area (2*sHor+1)*(2*sVer+1) is divided into 5 diamond search areas shown in Figure 2200. Each search area is assigned a costFactor, which is determined by the distance (int Delta MV) between each search point and the starting MV, and each diamond area is processed in order starting from the center of the search area. In each area, the search points are processed in raster scan order, starting from the upper left to the lower right corner of the area. When the minimum bilinear in the current search area is less than or equal to the threshold of sbW*sbH, the pixel full search is terminated, otherwise, continue to the pixel full search of the next search area until all search points are checked. BM performs a local search to derive the half-sample accuracy half DeltaMv. The search pattern and cost function are the same as defined in 2.9.1. The existing VVC DMVR fractional sample refinement is further applied to derive the final incremental MV (sbId×2). In the second pass, the refined MV is derived: ·MV0_pass2(sbId×2)=MV0_pass1+deltaMV(sbId×2), ·MV1_pass2(sbId×2)=MV1_pass1-deltaMV(sbId×2). 2.15.3 The third pass - sub-block-based bidirectional optical flow MV refinement In the third pass, refined MVs are derived by applying BDOF to 8×8 grid sub-blocks. For each 8×8 sub-block, BDOF refinement is applied to derive scaled Vx and Vy without clipping from the refined MV of the parent sub-block of the second pass. The derived bioMv(Vx, Vy) is rounded to 1 / 16 sample precision and clipped between -32 and 32. In the third pass, the refined MVs (MV0_pass3(sbId×3) and MV1_pass3(sbId×3)) are derived: ·MV0_pass3(sbId×3)=MV0_pass2(sbId×3)+bioMv, ·MV1_pass3(sbId×3)=MV0_pass2(sbId×3)-bioMv. 2.16 Sample-based BDOF In sample-based BDOF, instead of deriving motion refinement (Vx, Vy) on a block basis, it is performed on a per-sample basis. The coding block is divided into eight 8×8 sub-blocks. For each sub-block, it is determined whether to apply BDOF by checking the SAD between two reference sub-blocks against a threshold. If it is decided to apply BDOF to a sub-block, for each sample in the sub-block, a sliding 5×5 window is used and the existing BDOF process is applied for each sliding window to derive Vx and Vy. The derived motion refinement (Vx, Vy) is applied to adjust the bidirectional prediction sample value of the center sample of the window. 2.17 Extended Merger Forecast In VVC, the merge candidate list is constructed by including the following five types of candidates in order: (1) Spatial MVP from spatially neighboring CUs. (2) Temporal MVP from the same CU. (3) History-based MVP comes from the FIFO table. (4) Average MVP. (5) Zero MV. The size of the merge list is signaled in the sequence parameter set header, and the maximum allowed size of the merge list is 6. For each CU code in merge mode, the index of the best merge candidate is encoded using truncated unary binarization (TU). The first bin of the merge index is coded using context, and bypass coding is used for other bins. The derivation process of merge candidates for each category is provided in this session. As done in HEVC, VVC also supports parallel derivation of merge candidate lists for all CUs in a region of a certain size. 2.17.1 Spatial Candidate Derivation The derivation of spatial merging candidates in VVC is the same as in HEVC, except that the positions of the first two merging candidates are swapped. Fig.23 is a schematic diagram 2300 illustrating the locations of spatial merging candidates. Fig.23 Up to four merge candidates are selected from the candidates for the positions depicted in . The derivation order is B0, A0, B1, A1 and B2. Position B2 is considered only when one or more coding units of position B0, A0, B1, A1 are not available (for example because it belongs to another slice or tile) or when intra coding is performed. After adding the candidate for position A1, the addition of the remaining candidates is subject to a redundancy check that ensures that candidates with the same motion information are excluded from the list, thereby improving coding efficiency. . In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Fig.24 is a schematic diagram 2400 showing candidate pairs considered for redundant checking of spatial merging candidates. Fig.24 The arrows in are linked to pairs, and a candidate is only added to the list if the corresponding candidate for redundancy check does not have the same motion information. 2.17.2 Time Candidate Derivation In this step, only one candidate is added to the list. Specifically, when deriving this temporal merge candidate, the scaled motion vector is derived based on the co-located CU belonging to the co-located reference picture. The reference picture list used to derive the co-located CU is explicitly signaled in the slice header. Fig.25 2500, a scaled motion vector for a temporal merge candidate is obtained, which is scaled from the motion vector of the co-located CU using POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero. Fig.26 2600 is a schematic diagram showing candidate positions C0 and C1 of temporal merge candidates. Fig.26 As shown, the position of the temporal candidate is selected between candidates C0 and C1. If the CU at position C0 is not available, intra-coded, or outside the current row of the CTU, then position C1 is used. Otherwise, position C0 is used in the derivation of the temporal merge candidate. 2.17.3 History-based Merge Candidate Derivation After spatial MVP and TMVP, history-based MVP (HMVP) merge candidates are added to the merge list. This method stores the motion information of the previously coded block in a table as the MVP of the current CU. A table with multiple HMVP candidates is maintained during the encoding / decoding process. The table is reset (cleared) when a new CTU row is encountered. Whenever there is a non-sub-block inter-coded CU, the associated motion information is added to the last entry of the table as a new HMVP candidate. The HMVP table size S is set to 6, which indicates that up to 6 history-based MVP (HMVP) candidates can be added to the table. When a new motion candidate is inserted into the table, a constrained first-in-first-out (FIFO) rule is used, where a redundancy check is first applied to find if the same HMVP exists in the table. If found, the same HMVP is removed from the table, and all HMVP candidates are moved forward, and the same HMVP is inserted into the last entry of the table. HMVP candidates can be used in the merge candidate list construction process. Check the latest few HMVP candidates in the table and insert them into the candidate list after the TMVP candidate. Apply redundancy check to the spatial or temporal merge candidates on the HMVP candidate. In order to reduce the number of redundant check operations, the following simplifications are introduced: The number of HMPV candidates for merge list generation is set to (N<=4)?M:(8,N), where N indicates the number of existing candidates in the merge list and M indicates the number of HMVP candidates available in the table. Once the total number of available merge candidates reaches the maximum allowed merge candidates minus 1, the merge candidate list construction process from HMVP is terminated. 2.17.4 Pairwise Average Merge Candidate Derivation Pairwise average candidates are generated by averaging predefined candidate pairs in the existing merge candidate list, and the predefined pairs are defined as {(0,1),(0,2),(1,2),(0,3),(1,3),(2,3)}, where the number represents the merge index to the merge candidate list. The average motion vector is calculated separately for each reference list. If two motion vectors are available in one list, the two motion vectors are averaged even when they point to different reference pictures; if only one motion vector is available, it is used directly; if no motion vector is available, the list is kept invalid. When the merge list is not full after adding pairwise average merge candidates, zero MVPs are inserted at the end until the maximum number of merge candidates is encountered. 2.17.5 Merging Estimated Regions The merge estimation region (MER) allows independent derivation of merge candidate lists for CUs in the same merge estimation region (MER). Candidate blocks within the same MER as the current CU are not included for generation of the merge candidate list for the current CU. In addition, the update process for the history-based motion vector prediction candidate list is updated only when (xCb+cbWidth)>>Log2ParMrgLevel is greater than xCb>>Log2ParMrgLevel, and (yCb+cbHeight)>>Log2ParMrgLevel is greater than (yCb>>Log2ParMrgLevel), where (xCb, yCb) is the top left luma sample position of the current CU in the picture and (cbWidth, cbHeight) is the CU size. The MER size is selected at the encoder side and signaled in the sequence parameter set as log2_parallel_merge_level_minus2. 2.18 New Merge Candidates 2.18.1 Non-adjacent merge candidate derivation Fig. 27 A schematic diagram 2700 of VVC spatial neighboring blocks of a current block is shown. In VVC, Fig. 27 Five spatial neighboring blocks and one temporal neighboring block are shown for deriving merging candidates. It is proposed to derive additional merge candidates from locations that are not adjacent to the current block using the same pattern as in VVC. To achieve this, for each search round i, a virtual block is generated based on the current block as follows: First, calculate the relative position of the virtual block to the current block: Offsetx=-i×gridX, Offsetty=-i×gridY Where Offsetx and Offsetty represent the offset of the upper left corner of the virtual block relative to the upper left corner of the current block, and gridX and gridY are the width and height of the search grid. Second, the width and height of the virtual block are calculated by: newWidth=i×2×gridX+currWidth newHeight=i×2×gridY+currHeight. Where currWidth and currHeight are the width and height of the current block. newWidth and newHeight are the width and height of the new virtual block. gridX and gridY are currently set to currWidth and currHeight respectively. Fig.28 A schematic diagram of a virtual block in the i-th search round is shown, which shows the relationship between the virtual block and the current block. After generating the virtual block, block A i , B i , C i , D i and E i can be considered as VVC spatial neighboring blocks of the virtual block, and their positions are obtained in the same pattern as in VVC. Obviously, if the search round i is 0, the virtual block is the current block. In this case, block A i , B i , C i , D i and E i is the spatial neighboring block used in VVC merge mode. When building the merge candidate list, pruning is performed to ensure that each element in the merge candidate list is unique. The maximum search loop is set to 1, which means that five non-adjacent spatial neighboring blocks are used. The non-adjacent spatial merging candidates are inserted into the merging list after the temporal merging candidates in the order of B1->A1->C1->D1->E1. 2.18.2STMVP It is proposed to use three spatial merging candidates and one temporal merging candidate to derive the average candidate as the STMVP candidate. Insert STMVP before the top left spatial merge candidate. The STMVP candidates are pruned using all previous merge candidates in the merge list. For spatial candidates, the first three candidates in the current merge candidate list are used. For the temporal candidate, the same position as the VTM / HEVC co-location is used. For spatial candidates, the first, second, and third candidates inserted into the current merge candidate list before STMVP are denoted as F, S, and T. The temporal candidate having the same position as the VTM / HEVC co-location used in TMVP is denoted as Col1. The motion vector of the STMVP candidate in prediction direction X (denoted as mvLX) is derived as follows: 1) If the reference indices of the four merge candidates are all valid and are all equal to zero in the prediction direction X (X = 0 or 1), then mvLX=(mvLX_F+mvLX_S+mvLX_T+mvLX_Col)>>2. 2) If the reference indexes of three of the four merge candidates are valid and equal to zero in prediction direction X (X=0 or 1), then mvLX=(mvLX_F×3+mvLX_S×3+mvLX_Col×2)>>3 or mvLX=(mvLX_F×3+mvLX_T×3+mvLX_Col×2)>>3 or mvLX=(mvLX_S×3+mvLX_T×3+mvLX_Col×2)>>3. 3) If the reference indexes of two of the four merge candidates are valid and equal to zero in prediction direction X (X=0 or 1), then mvLX=(mvLX_F+mvLX_Col)>>1 or mvLX=(mvLX_S+mvLX_Col)>>1 or mvLX=(mvLX_T+mvLX_Col)>>1. NOTE: If a time candidate is not available, the STMVP mode is deactivated. 2.18.3 Merge List Size If non-contiguous and STMVP merging candidates are considered, the size of the merging list is signaled in the sequence parameter set header and the maximum allowed size of the merging list is 8. 2.19 Geometric Partitioning Model (GPM) In VVC, geometric partitioning modes are supported for inter prediction. The geometric partitioning mode is signaled using a CU level flag as a type of merge mode, where other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. The geometric partitioning mode supports a total of 64 partitions for each possible coding unit (CU) size w×h=2 m ×2 n (where m,n∈{3…6}), excluding 8×64 and 64×8. like Fig.29 As shown, when this mode is used, the CU is divided into two parts by a geometrically positioned straight line. The position of the dividing line is mathematically derived from the angle and offset parameters of the specific partition. Each part of the geometric partition in the CU uses its own motion for inter-frame prediction; only unidirectional prediction is allowed for each partition, that is, each part has one motion vector and one reference index. Unidirectional prediction motion constraints are applied to ensure that only two motion compensated predictions are required for each CU, the same as conventional bidirectional prediction. The unidirectional prediction motion of each partition is derived using the process described in 2.19.1. If the current CU uses geometric partitioning mode, a geometric partition index indicating the partitioning mode (angle and offset) of the geometric partition and two merge indices (one for each partition) are further signaled. The number of maximum GPM candidate sizes is explicitly signaled in the SPS and the syntax binarization of the GPM merge indices is specified. After predicting each of the parts of the geometric partition, a hybrid process with adaptive weights as in 219.2 is used to adjust the sample values along the edges of the geometric partitions. This is the prediction signal for the entire CU, and the transform and quantization process will be applied to the entire CU as in other prediction modes. Finally, the motion field of the CU predicted using the geometric partitioning mode is stored as in 2.19.3. 2.19.1 One-way prediction candidate list construction In 2.17, the unidirectional prediction candidate list is derived directly from the merge candidate list constructed according to the extended merge prediction process. Denote n as the index of the unidirectional prediction motion in the geometric unidirectional prediction candidate list. The LX motion vector (where X equals the parity of n) of the nth extended merge candidate is used as the nth unidirectional prediction motion vector of the geometric partitioning mode. Fig.30 Shows the unidirectional prediction MV selection for geometric partitioning mode. These motion vectors are in Fig.30 In the case where there is no corresponding LX motion vector for the n-extended merge candidate, the L(1-X) motion vector of the same candidate is used instead of the unidirectional prediction motion vector for the geometric partitioning mode. 2.19.2 Blending along the edges of geometric partitions After each part of the geometric partition is predicted using its own motion, blending is applied to the two prediction signals to derive samples around the geometric partition edge. The blending weights for each position of the CU are derived based on the distance between each position and the partition edge. The distance from the location to the partition edge is derived as: where i and j are the indices of the angle and offset of the geometry partition, which depend on the indicated geometry partition index. x,j and ρ y,j The sign of depends on the angle index i. The weights for each part of the geometric partition are derived as follows: wIdxL(x,y)=partIdx? 32+d(x,y):32-d(x,y) (2-28) w1(x,y)=1-w0(x,y) (2-30) partIdx depends on the angle index i. Fig.31 The use of a geometric partitioning pattern to generate warp weights is shown. An example of a weight w0 is Fig.31 Shown in. 2.19.3 Motion Field Storage for Geometry Partitioning Mode Mv1 from the first part of the geometric partition, Mv2 from the second part of the geometric partition, and a combination Mv of Mv1 and Mv2 are stored in the motion region of the CU encoded in the geometric partitioning mode. The type of motion vector stored for each individual position in the motion field is determined as: sType=abs(motionIdx)<32?2:(motionIdx≤0?(1-partIdx):partIdx) (2-31) Where motionIdx is equal to d(4x+2,4y+2), which is recalculated according to equation (2-18). partIdx depends on the angle index i. If sType is equal to 0 or 1, then Mv0 or Mv1 is stored in the corresponding motion field, otherwise if sType is equal to 2, then the combined Mv from Mv0 and Mv2 is stored. The combined Mv is generated using the following process: 1) If Mv1 and Mv2 are from different reference picture lists (one from L0 and another from L1), then simply combine Mv1 and Mv2 to form a bi-predictive motion vector. Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored. 2.20 Multi-hypothesis prediction In Multiple Hypothesis Prediction (MHP), on top of Inter AMVP mode, Normal Merge mode, Affine Merge and MMVD mode, up to two additional predictors can be signaled. The resulting overall prediction signal is iteratively accumulated with each additional prediction signal. p n+1 =(1-α n+1 ) n +α n+1 h n+1 The weighting factor α is specified according to the following Table 2-4: Table 2-4- Weighting factors of MHP add_hyp_weight_idx α 0 1 / 4 1 -1 / 8 For inter-AMVP mode, MHP is only applied if unequal weights in BCW are selected in bi-prediction mode. The additional assumption may be a merge mode or an AMVP mode. In the case of the merge mode, the motion information is indicated by a merge index, and the merge candidate list is the same as in the geometry partitioning mode. In the case of the AMVP mode, the reference index, MVP index, and MVD are signaled. 2.21 Non-adjacent spatial candidates Non-adjacent spatial merging candidates are inserted after the TMVP in the regular merging candidate list. Fig.32 Shows the spatial neighboring blocks used to derive spatial merge candidates. The mode of spatial merge candidates is shown in Fig.32 The distance between the non-neighboring spatial candidates and the current coding block is based on the width and height of the current coding block. 2.22 Template Matching (TM) Template Matching (TM) is a decoding-side MV derivation method to improve the motion information of the current CU by finding the closest match between a template in the current picture (i.e., the top and / or left neighboring blocks of the current CU) and a block in a reference picture (i.e., the same size as the template). Fig.33 It is shown that template matching is performed on the search area around the initial MV. Fig.33 As shown, a better MV is searched around the initial motion of the current CU within the search range of [-8, +8] pixels. In this contribution, the template matching previously proposed in JVET-J0021 is adopted with two modifications: the search step size is determined based on the AMVR mode, and it can be cascaded with the bilateral matching process in the merge mode. In AMVP mode, MVP candidates are determined based on template matching error to pick the MVP candidate that achieves the minimum difference between the current block template and the reference block template, and then TM is performed only for this specific MVP candidate for MV refinement. TM refines the MVP candidate by using iterative diamond search from full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode) within the [-8, +8] pixel search range. The AMVP candidate can be further refined by using a cross search with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode), followed by half-pixel and quarter-pixel refinement in turn according to the AMVR mode specified in Table 2-5. This search process ensures that after the TM process, the AMVP candidate still maintains the same motion vector accuracy as indicated by the AMVR mode. Table 2-5 - Search mode of AMVR and merge mode with AMVR In merge mode, a similar search method is applied to the merge candidate indicated by the merge index. As shown in Table 2-5, according to the merged motion information, according to the alternative interpolation filter (the interpolation filter used when AMVR is half-pixel mode), TM can be performed all the way up to 1 / 8 pixel MVD accuracy or skip the accuracy beyond half-pixel MVD accuracy. In addition, when TM mode is enabled, template matching can be used as an independent process or an additional MV refinement process between block-based bilateral matching (BM) methods, depending on whether BM can be enabled according to its enabling condition check. 2.23 Overlapped Block Motion Compensation (OBMC) Overlapped Block Motion Compensation (OBMC) has been used previously in H.263. In the Joint Exploration Model (JEM), unlike H.263, OBMC can be turned on and off at the coding unit (CU) level through syntax. When OBMC is used in JEM, OBMC is performed on all motion compensated (MC) block boundaries except the right and bottom boundaries of the CU. In addition, it is applied to both luma and chroma components. In JEM, MC blocks correspond to coding blocks. When a CU is encoded or decoded in sub-CU mode (including sub-CU merging, affine and FRUC modes), each sub-block of the CU is an MC block. Fig.34 is a diagram of the sub-blocks of the OBMC application. Fig.34 As shown, in order to handle CU boundaries in a unified manner, OBMC is performed at the sub-block level for all MC block boundaries, where the sub-block size is set equal to 4×4. When OBMC is applied to the current sub-block, in addition to the current motion vector, the motion vectors of four connected neighboring sub-blocks, if available and different from the current motion vector, are also used to derive the prediction block of the current sub-block. These multiple prediction blocks based on multiple motion vectors are combined to generate the final prediction signal of the current sub-block. The prediction block based on the motion vectors of neighboring sub-blocks is denoted as P N , where N indicates the index of the neighboring upper, lower, left and right sub-blocks, and the prediction block based on the motion vector of the current sub-block is denoted as P C When P N When based on the motion information of the neighboring sub-block containing the same motion information as the current sub-block, no N Execute OBMC. Otherwise, P N Each sample is added to P C The same sample in P N Add four rows / columns to P C The weighting factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} are used for P N , weighting factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are used for P C The exception is small MC blocks (i.e., when the height or width of the coding block is equal to 4 or the CU is coded in sub-CU mode), for which only P N Add two rows / columns to P C In this case, weighting factors {1 / 4, 1 / 8} are used for P N , and weighting factors {3 / 4,7 / 8} are used for P C For P generated based on the motion vector of the vertical (horizontal) neighboring sub-block N , P N The samples in the same row (column) of are added to P with the same weighting factor C . In JEM, for CUs with a size less than or equal to 256 luma samples, a CU level flag is signaled to indicate whether OBMC is applied for the current CU. CUs with a size greater than 256 luma samples or not encoded in AMVP mode are applied by default. At the encoder, when OBMC is applied to a CU, its impact is taken into account during the motion estimation stage. The prediction signal formed by OBMC using the motion information of the top and left neighboring blocks is used to compensate the top and left boundaries of the original signal of the current CU, and then the normal motion estimation process is applied. 2.24 Multiple Transform Selection (MTS) for Core Transformations In addition to DCT-II already adopted in HEVC, the multi-transform selection (MTS) scheme is used for residual coding of both inter-frame and intra-frame coded blocks. It uses multiple selected transforms from DCT8 / DST7. The newly introduced transform matrices are DST-VII and DCT-VIII. Table 2-6 shows the basis functions of the selected DST / DCT. Table 2-6 - Transform basis functions for DCT-II / VIII and DST-VII for N-point input To maintain the orthogonality of the transform matrix, the transform matrix is quantized more accurately than the transform matrix in HEVC. To keep the intermediate values of the transform coefficients within the 16-bit range, all coefficients will have 10 bits after horizontal and vertical transforms. To control the MTS scheme, separate enable flags are specified at the SPS level for intra and inter, respectively. When MTS is enabled at SPS, a CU level flag is signaled to indicate whether MTS is applied. Here, MTS is applied only to luma. MTS signaling is skipped when one of the following conditions applies. - The position of the last significant coefficient of the luma TB is less than 1 (ie, DC only). – The last significant coefficient of the luminance TB is located within the MTS zeroing region. If the MTS CU flag is equal to zero, DCT2 is applied in both directions. However, if the MTS CU flag is equal to 1, two other flags are additionally signaled to indicate the transform type in the horizontal and vertical directions, respectively. The transform and signaling mapping table is shown in Table 2-7. A unified transform selection for ISP and implicit MTS is used by removing intra-mode and block shape dependencies. If the current block is in ISP mode, or if the current block is an intra-block and both intra- and inter-explicit MTS are turned on, only DST7 is used for both horizontal and vertical transform kernels. When transforming matrix precision, an 8-bit primary transform kernel is used. Therefore, all transform kernels used in HEVC remain the same, including 4-point DCT-2 and DST-7, 8-point, 16-point, and 32-point DCT-2. In addition, other transform kernels including 64-point DCT-2, 4-point DCT-8, 8-point, 16-point, 32-point DST-7, and DCT-8 use 8-bit primary transform kernels. Table 2-7 - Transformation and signaling mapping table To reduce the complexity of large-size DST-7 and DCT-8, high-frequency transform coefficients are zeroed for DST-7 and DCT-8 blocks with size (width or height, or both) equal to 32. Only coefficients within the 16×16 low-frequency region are retained. As in HEVC, the residual of the decoded block can be encoded in transform skip mode. To avoid redundancy of syntax coding, the transform skip flag is not indicated when the CU level MTS_CU_flag is not equal to zero. Note that when LFNST or MIP is activated for the current CU, the implicit MTS transform is set to DCT2. In addition, when MTS is enabled for inter-frame coded blocks, implicit MTS can still be enabled. 2.25 Sub-Block Transform (SBT) In VTM, sub-block transform is introduced for inter-predicted CUs. This transform mode encodes only a sub-portion of the residual block for the CU. When there is an inter-predicted CU with cu_cbf equal to 1, cu_sbt_flag can be indicated to indicate whether to encode the entire residual block or a sub-portion of the residual block. In the former case, the inter-MTS information is further parsed to determine the transform type of the CU. In the latter case, a portion of the residual block is encoded using the inferred adaptive transform, and another portion of the residual block is zeroed. When SBT is used for an inter-coded CU, the SBT type and SBT location information are indicated in the bitstream. Figure 2-32 As shown, there are two SBT types and two SBT positions. For SBT-V (or SBT-H), the TU width (or height) can be equal to half of the CU width (or height) or 1 / 4 of the CU width (or height), resulting in a 2:2 split or a 1:3 / 3:1 split. The 2:2 split is similar to the binary tree (BT) split, while the 1:3 / 3:1 split is similar to the asymmetric binary tree (ABT) split. In the ABT partition, only small areas contain non-zero residuals. If one dimension of the CU is 8 in luminance samples, 1:3 / 3:1 splitting along that dimension is not allowed. The CU has up to 8 SBT modes. Position-dependent transform kernel selection is applied to luma transform blocks in SBT-V and SBT-H (chroma TBs always use DCT-2). The two positions of SBT-H and SBT-V are associated with different kernel transforms. Fig.35 The SBT location, type, and transformation type are shown. More specifically, Fig.35 The horizontal and vertical transforms for each SBT position are specified in . For example, the horizontal and vertical transforms for SBT-V position 0 are DCT-8 and DST-7, respectively. When a side of the residual TU is larger than 32, the transforms for both dimensions are set to DCT-2. Therefore, the sub-block transform jointly specifies the TU partitioning, cbf, and horizontal and vertical core transform types of the residual block. SBT does not apply to CUs coded or decoded in combined inter-intra mode. 2.26 Adaptive Merging Candidate Reordering Based on Template Matching To improve coding efficiency, after building the merge candidate list, the order of each merge candidate is adjusted according to the template matching cost. The merge candidates are arranged in the list according to the template matching cost in ascending order. It operates in the form of subgroups. The template matching cost is measured by the SAD (sum of absolute differences) between the neighboring samples of the current CU and its corresponding reference samples. Fig.36 The neighboring samples used to calculate SAD are shown. If the merge candidate contains bidirectional predictive motion information, the corresponding reference sample is the average of the corresponding reference sample in reference list 0 and the corresponding reference sample in reference list 1, such as Fig.36 As described in . Fig.37 The neighboring samples used to calculate the SAD of the sub-CU level motion information are shown. If the merge candidate contains sub-CU level motion information, the corresponding reference samples are composed of the neighboring samples of the corresponding reference sub-block, such as Fig.37 as shown in . The sorting process operates in the form of subgroups, e.g. Fig.38 As shown in . The first three merge candidates are sorted together. The following three merge candidates are sorted together. Fig.38 As shown, the original merge candidate list 3810 is sorted to obtain an updated merge candidate list 380. In this example, the template size (the width of the left template or the height of the upper template) is 1, and the subgroup size is 3. 2.27 Adaptive Merge Candidate List Assume that the number of merge candidates is 8. Take the first 5 merge candidates as the first subgroup, and take the following 3 merge candidates as the second subgroup (ie, the last subgroup). Fig.39 3900 is a flowchart of a reordering process in an encoder. For the encoder, at block 3902, after constructing a merge candidate list, as Fig.39 As shown, some merge candidates are adaptively reordered in ascending order of their costs. More specifically, at box 3904, the template matching cost of the merge candidates in all subgroups except the last subgroup is calculated; then at box 3906, the merge candidates in its own subgroup except the last subgroup are reordered; finally, at box 3908, the final merge candidate list is obtained. For the decoder, after building the merge candidate list, some / no merge candidates are adaptively reordered in ascending order of their cost, e.g. Fig.40 As shown, Fig.40 A flow chart of a reordering process 4000 in a decoder is shown. Fig.40 In , a subgroup that is among the selected (signaled) merging candidates is called a selected subgroup. More specifically, at block 4002, it is determined whether the selected merge candidate is located in the last subgroup. If the selected merge candidate is located in the last subgroup, then at block 4004, after deriving the selected merge candidate, the merge candidate list construction process is terminated, and at block 4006, no reordering is performed and the merge candidate list is not changed; otherwise, the execution process is as follows: At box 4008, after all merge candidates in the selected subgroup are exported, the merge candidate list construction process is terminated; at box 4010, the template matching cost of the merge candidates in the selected subgroup is calculated; at box 4012, the merge candidates in the selected subgroup are reordered; finally, at box 4014, a new merge candidate list is obtained. For both the encoder and the decoder, the template matching cost is derived from T and RT, where T is a set of samples in the template and RT is a set of reference samples for the template. When deriving reference samples of the template for the merge candidate, the motion vector of the merge candidate is rounded to integer pixel precision. The template reference sample (RT) used for bidirectional prediction is obtained by weighted averaging the template reference sample (RT0) in reference list 0 and the template reference sample (RT1) in reference list 1, as follows: RT=((8-w)*RT0+w*RT1+4)>>3 (2-32) The weight (8-w) of the reference template in reference list 0 and the weight (w) of the reference template in reference list 1 are determined by the block-level weight (BCW) index of the merge candidate. The BCW index equals {0, 1, 2, 3, 4} and w equals {-2, 3, 4, 5, 10} respectively. If the local illumination compensation (LIC) flag of the merge candidate is true, the reference samples of the template are derived using the LIC method. The template matching cost is calculated based on the sum of absolute differences (SAD) of T and RT. The template size is 1. That is, the width of the left template and / or the height of the top template is 1. If the codec mode is MMVD, the merge candidates from which the base merge candidate is derived are not reordered. If the coding mode is GPM, the merge candidates used to derive the unidirectional prediction candidate list are not reordered. 2.28 Geometric Prediction Mode with Motion Vector Difference In geometric prediction mode with motion vector difference (GMVD), each geometric partition in GPM can decide to use GMVD. If GMVD is selected for a geometric region, the MV of the region is calculated as the sum of the MV of the merge candidate and the MVD. All other processing remains the same as GPM. With GMVD, the MVD is signaled as a pair of direction and distance. There are nine candidate distances (1 / 4 pixel, 1 / 2 pixel, 1 pixel, 2 pixels, 3 pixels, 4 pixels, 6 pixels, 8 pixels, 16 pixels) and eight candidate directions (four horizontal / vertical directions and four diagonal directions). In addition, when pic_fpel_mmvd_enabled_flag is equal to 1, the MVD in GMVD is also left-shifted by 2, as in MMVD. 2.29 Affine MMVD In affine MMVD, an affine merge candidate (which is referred to as a basic affine merge candidate) is selected, and the MV of the control point is further refined by the indicated MVD information. The MVD information of the MVs of all control points is the same in one prediction direction. When the starting MV is a bidirectional prediction MV with two MVs pointing to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), the MV offsets added to the List 0MV component of the starting MV and the MV offsets of List 1MV have opposite values; otherwise, when the starting MV is a bidirectional prediction MV with two lists pointing to the same side of the current picture (i.e., the POCs of the two references are both greater than the POC of the current picture, or both are less than the POC of the current picture), the MV offsets added to the List 0MV component of the starting MV and the MV offsets of List 1MV are the same. 2.30 Adaptive Decoder Motion Vector Refinement (ADMVR) In ECM-2.0, if the selected merge candidate meets the DMVR condition, the multi-pass decoding side motion vector refinement (DMVR) method is applied in the normal merge mode. In the first pass, bilateral matching (BM) is applied to the coding block. In the second pass, BM is applied to each 16×16 sub-block within the coding block. In the third pass, the MV in each 8×8 sub-block is refined by applying bidirectional optical flow (BDOF). The adaptive decoding-side motion vector refinement method consists of two new merge modes, which are introduced to refine the MV in only one direction (L0 or L1) of the bidirectional prediction of the merge candidate that meets the DMVR condition. A multi-pass DMVR process is applied to the selected merge candidate to refine the motion vector, however, in the first pass (i.e., PU level) DMVR, MVD0 or MVD1 is set to zero. Similar to the conventional merge mode, the merge candidates for the proposed merge mode are derived from spatially adjacent coding blocks, TMVP, non-adjacent blocks, HMVP, and pairwise candidates. The difference is that only those that satisfy the DMVR condition are added to the candidate list. The same merge candidate list (i.e., ADMVR merge list) is used by both proposed merge modes, and the merge index is encoded in the conventional merge mode. 3. Question In the current design of adaptive merge candidate reordering based on template matching, the motion information of the merge candidates is directly used in the reordering process. However, the motion information of some merge candidates may be modified by motion refinement (such as template matching and / or decoding-side motion vector refinement). Since the motion information used in the reordering process is not aligned with the motion information used in the reconstruction of the video unit, the reordering process may not be efficient. 4. Embodiments of the present disclosure The following detailed embodiments should be considered as examples to explain the general concept. These embodiments should not be interpreted in a narrow manner. In addition, these embodiments can be combined in any way. In the present disclosure, the motion candidate list may refer to a motion candidate list for TM merge mode and / or normal merge mode and / or adaptive DMVR and / or CIIP and TM merge mode and / or AMVP and / or affine and / or GPM (GEO) and / or TPM and / or MMVD and / or CIIP and / or MHP and / or AMVP merge, affine AMVP, TM AMVP, GPM with MMVD, IBC merge, IBCAMVP, template IBC, intra-template matching and / or any inter-mode prediction / reconstruction of a block deriving using the motion candidate list. The motion candidate list may be a TM merge / AMVP list after block-based bilateral matching refinement and / or template matching refinement and / or sub-block-based bilateral matching refinement. The motion candidate list may be a regular merge / AMVP list after the DMVR / multi-pass DMVR process. The motion candidate list may be an ADMVR merged list after the DMVR / multi-pass DMVR process. The motion candidate list may be a GPM merge list (eg, AGPMList, LGPMList, or LAGPMList) after a template matching refinement process. In the following discussion, SatShift(x,n) is defined as In one example, offset0 and / or offset1 are set to (1 <<n)> >1or(1<<(n-1)). In another example, offset0 and / or offset1 are set to 0. Shift(x,n) is defined as Shift(x,n)=(x+offset0)>>n. Clip3(x,y,z) is defined as Re-rank motion candidate list using refined motion information 1. It is proposed that before performing the first reordering process on the motion candidate list, at least one motion candidate in the list may be modified or refined. a. When one or more motion candidates in the motion candidate list of a video unit are modified by the motion refinement process, it is proposed to reorder the motion candidate list using the refined motion information. b. In one example, a second reordering process may be performed on the motion candidate list before the first reordering process. i. In one example, when constructing the motion candidate list in the second reordering process, the motion candidate list may be reordered using the original motion information. ii. In one example, after the second reordering process, not all motion candidates in the motion candidate list may be reordered in the first reordering process. 1) In one example, the first M motion candidates may be reordered in a first reordering process. a) In one example, M may be predefined, such as M=4, or 5, or 6, Or 7, or 8, or 9, or 10. b) In one example, M may be signaled in the bitstream. c) In one example, M may be equal to N+M1, where N is the maximum number of motion candidates allowed by the codec (e.g., the syntax element indicated in the SPS), and M1 is an integer greater than or equal to 0, e.g., M1=0, or 1, or 2, or 3, or 4, or 5, or 6. 2) In one example, the motion candidates may be divided into subgroups. Whether to reorder the motion candidates in the first reordering process and / or the method to reorder the motion candidates in the first reordering process depends on the subgroups of the motion candidates. a) In one example, only the first subset may be reordered. b) In one example, the last subgroup cannot be reordered. c) In one example, the last subgroup cannot be reordered. However, if the last subgroup is also the first subgroup, it can be reordered. d) Different subgroups can be reordered individually. e) Two candidates in different subgroups cannot be compared and / or re-ordered. f) If the first subgroup is before the second subgroup, after reordering, the first candidate of the first subgroup should also be before the second candidate of the second subgroup. c. In one example, the motion candidate list may not have been reordered before being reordered using the first reordering process. d. In one example, the motion refinement process may refer to template matching based motion refinement, such as template matching and / or bilateral matching based motion refinement, such as DMVR, multi-pass DMVR. e. In one example, a simplified motion refinement process may be applied. i. In one example, when template matching is used as one of the motion refinement processes, the template size can be set equal to 1 or 2. ii. In one example, when template matching is used as one of the motion refinement processes, the template may be downsampled first, and the downsampled template is used. 1) In one example, the downsampling ratio may be equal to T, such as T=2 or 3 or 4. iii. In one example, when template matching is used as one of the motion refinement processes, only one of the upper template and the left template may be used. Denote the block width and block height as W and H, respectively. 1) In one example, when both the upper template and the left template are available, the template with the larger size may be used, such as using the upper template when the block width is greater than the block height, and using the left template when the block height is greater than the block width. 2) In one example, whether to use only the upper template or only the left template may depend on the block width and / or the block height. a) In one example, when W>s×H, only the upper template is used. i. In one example, when W>s×H and the upper template is not available, the left template is used. b) In one example, when H>s×W, only the left template is used. i. In one example, when H>s×W and the left template is not available, the upper template is used. c) In one example, s is an integer, such as s=2, or s=4, or s=1, or s=3, or s=6, or s=8. iv. In one example, whether and how to simplify the motion refinement process may depend on codec information, such as block size or QP. f. In one example, the refined motion information used in the first reordering process may be the final motion information of the motion refinement process. i. In one example, when the motion refinement process includes multiple steps, the motion information after the last step in the motion refinement process is used in the first reordering process. 1) In one example, multiple steps may refine the motion information at different motion vector resolutions. a) In one example, refinement may be from integer resolution motion vectors to fractional resolution motion vectors. b) In another example, the refinement may be from fractional resolution motion vector to fractional resolution motion vector. c) In one example, different resolutions of motion vectors may refer to 16 pixels, and / or 8 pixels, and / or 4 pixels, and / or 2 pixels, and / or 1 pixel, and / or 1 / 2 pixel, and / or 1 / 4 pixel, and / or 1 / 8 pixel, and / or 1 / 16 pixel, and / or 1 / 32 pixel, and / or 1 / 64 pixel. 2) In one example, multiple steps may refer to different search shapes to refine the motion vector, such as diamond and / or cross in template matching. a) In one example, the motion information following the diamond search shape of template matching may be used in the first re-ordering process. b) In one example, at least one search shape may be skipped. i. In one example, diamond search shapes may be skipped in TM refinement. ii. In one example, cross-searching shapes may be skipped in TM refinement. c) In one example, at least one search step may be skipped. i. In one example, full pixel diamond search, and / or full pixel intersection, And / or half-pixel crossing, and / or quarter-pixel crossing, and / or 1 / 8-pixel crossing search steps may be skipped. 3) In one example, the resolution of the motion vector to be used may depend on a syntax element indicating the resolution of the motion vector of the motion candidate, such as the value of the adaptive motion vector resolution. 4) In one example, when early termination is used in the motion refinement process, the motion information output due to the early termination can be used in the first reordering process. ii. In one example, when more than one motion refinement process is involved and processed one by one, the motion information derived after the last motion refinement process may be used in the first reordering flow. 1) In one example, when the motion refinement process includes a first motion refinement process and a second motion refinement process, motion information of the second motion refinement process may be used in the first reordering process. a) In one example, the first motion refinement process may refer to a template matching method such as TM, or bilateral matching methods such as DMVR / multi-pass DMVR. b) In one example, the second motion refinement process may refer to a template matching method such as TM, or bilateral matching methods such as DMVR / multi-pass DMVR. g. In one example, the refined motion information used in the first reordering process may be the intermediate motion information of the motion refinement process. i. In one example, when the motion refinement process includes multiple steps, motion information before the last step in the motion refinement process may be used. 1) In one example, the motion information in the first step can be used. 2) In another example, motion information in integer resolutions (eg, 16 pixels, 8 pixels, 4 pixels, 2 pixels, 1 pixel) of motion vectors may be used. ii. In one example, when more than one motion refinement pass is involved and processed one by one, the motion information derived from any motion refinement pass before the last motion refinement pass may be used in the first reordering flow. 1) In one example, when the motion refinement process includes a first motion refinement process and a second motion refinement process, motion information of the first motion refinement process may be used. a) In one example, the first motion refinement process may refer to a template matching method such as TM, or bilateral matching methods such as DMVR / multi-pass DMVR. b) In one example, the second motion refinement process may refer to a template matching method such as TM, or bilateral matching methods such as DMVR / multi-pass DMVR. iii. In one example, when more than one motion refinement process is involved and processed one by one and the last motion refinement process includes multiple steps, motion information derived from any step before the last step in the last motion refinement process may be used in the first reordering flow. iv. In one example, when the motion refinement process includes TM and TM is processed between the first pass DMVR (e.g., block-based bilateral matching MV refinement) and the second pass DMVR (e.g., sub-block-based bilateral matching MV refinement), the motion information after TM can be used in the first reordering process. 1) Alternatively, the motion information after the first DMVR pass can be used. 2) Alternatively, the motion information after the second DMVR pass can be used. 3) In one example, when refined motion information is used in the first reordering process, the early termination method for skipping the second pass DMVR may not be applied. h. In one example, when the refined motion information is based on motion information of sub-blocks, the refined motion information of one or more sub-blocks may be used in the first reordering process. i. In one example, refined motion information of one of the sub-blocks may be used in the first reordering process. 1) In one example, refined motion information of the upper left sub-block, or one of the sub-blocks located at the left block boundary, or one of the sub-blocks located at the upper block boundary may be used. ii. In one example, refined motion information of more than one sub-block may be used in the first reordering process. 1) In one example, the motion information of all sub-blocks may be averaged and used. a) In one example, the motion information of all sub-blocks may be weighted averaged, and the weighting parameter may depend on the block / sub-block size / size and / or the distance between the sub-block and the template used in the reordering process. 2) In another example, motion information of a sub-block at an upper boundary and / or motion information of a sub-block at a left boundary may be used. i. In one example, the motion candidates may be divided into X subgroups, and the re-ranking process may be processed within each subgroup, where X is greater than 1. i. In one example, for all subgroups, the subgroup size (ie, the number of motion candidates within the subgroup, N s ) can be the same, such as N s =2, 3, 4, 5, 6, 7, 8, 9, 10. ii. In another example, at least one subgroup may be of a different size than the other subgroups. iii. In one example, the size of the first subset is larger or smaller than the sizes of the other subsets. iv. In one example, the size of the subgroups may be predefined. v. In one example, the motion candidates may be divided into multiple subgroups according to the motion candidate type (eg, adjacent spatial, temporal, non-adjacent spatial, HMVP). vi. In one example, the available number of types of motion candidates (e.g., adjacent spatial and / or or temporal motion candidates) to determine the size of the subgroup. j. In one example, an X-tap interpolation filter may be used in the first reordering process, such as X=2, or 4, or 6, or 8, or 10, or 12. i. In one example, the interpolation filters used in the first reordering process and the second reordering process may be different. ii. Alternatively, the interpolation filter used in the first reordering process and the second reordering process may be the same. iii. In one example, when the first reordering process is used, a Y-tap interpolation filter, such as Y=2 or 4, or 6 or 8 or 10, may be used in the second reordering process. k. In one example, the motion refinement process before the reordering process is denoted as a first motion refinement process. The motion information generated in the first motion refinement process can be used as the starting motion information and further refined by a second motion refinement process before the motion information is used to obtain prediction / reconstruction of the block. i. In one example, when a TM is included in the first motion refinement process, the motion information generated from the TM may be used as the starting motion information. ii. In one example, when DMVR / multi-pass DMVR is included in the first motion refinement process, the motion information generated from DMVR / multi-pass DMVR may be used as the starting motion information. iii. Alternatively, the motion information generated in the first motion refinement process may be used as intermediate motion information for the second motion refinement process. 1) In one example, when multi-pass DMVR is included in the second motion refinement process, the first-pass DMVR is skipped, and if TM is included in the multi-pass DMVR, the motion information generated in the first motion refinement process can be further refined directly from the second-pass DMVR or TM. 2) In one example, when TM is included in the second motion refinement process, the motion information generated in the first motion refinement process can be used as a starting point for any TM search mode, but not for the first search mode (eg, diamond search). 1. In one example, how many motion candidates and / or which motion candidates are refined before the reordering process can be determined in real time. A cost (eg, TM cost or BM cost) can be first calculated for each motion candidate, and the motion candidates are sorted according to the cost. i. In one example, the top M motion candidates with the lowest costs may be refined, such as M=2, or 3, or 4, or 5, or 6, or 7, or 8, or 9, or 10. 1) In one example, the selected M motion candidates may be extended to N motion candidates (N>M) by adding at least one MV offset to each motion candidate. ii. In another example, motion candidates with costs greater than a threshold are not used in the first refinement process and the re-ranking process. 1) In one example, the threshold may depend on the best cost of the motion candidate. 2) In one example, the threshold may be equal to P×C0, where C0 represents the best cost of the motion candidate, such as P=2, or 3, or 4, or 5, or 6, or 7, or 8, or 9, or 10. 3) In one example, the first N motion candidates with costs greater than a threshold may be used in the first refinement process and the reordering process, but other motion candidates with costs greater than the threshold are not used in the refinement process and the reordering process. As an example, N=1, or N=2, or N=3, or N =4, or N=5. 2. In one example, whether and / or how to use refined motion information to apply the reordering process can be signaled in the bitstream, or determined on the fly, or depend on codec information. a. In one example, a reordering process using refined motion information may be applied for video units having a size greater than T1 (eg, T1 = 8 or 16, or 32 or 64 or 128). b. In another example, the reordering process using refined motion information may be applied for video units having a size smaller than T1 (eg, T1 = 64, or 128, or 256, or 512, or 1024, or 2048). c. In one example, whether the reordering process using refined motion information is applied to video units may depend on the temporal layer. d. In one example, whether to apply the reordering process using refined motion information to a video unit may depend on a quantization parameter (QP). e. In one example, whether the reordering process using refined motion information is applied to a video unit may depend on the slice / picture type. i. In one example, the reordering process using refined motion information can be applied only to B slices / picture. ii. In another example, the reordering process using refined motion information may be applied only to B slices / pictures having at least one forward reference picture and one backward reference picture. f. In one example, whether the reordering process using refined motion information is applied for different codec tools may have different conditions for different codec tools. i. In one example, the codec tool may refer to TM merge mode and / or ADMVR mode and / or TM AMVP mode and / or regular merge mode. g. In one example, whether and / or how the reordering process is applied to different codecs may be controlled by a single syntax element (eg, an SPS flag) signaled in the bitstream. 3. In one example, when a reordering process using refined motion information is applied to an inter-frame mode with unilateral DMVR, adaptive DMVR, the reordering process using refined motion can be applied to reorder a first motion candidate list and / or a second motion candidate list, wherein the motion candidates in the first motion candidate list are refined on one side (direction) and the motion candidates in the second motion candidate list are refined on another side (direction). a. In one example, one-sided DMVR may refer to a DMVR method in which motion information (eg, motion vectors) is refined in one direction in the step of DMVR. i. In one example, in the first pass of multi-pass DMVR, motion information may be refined in one direction. ii. In one example, in the first pass and / or the second pass of multi-pass DMVR, motion information may be refined in one direction. b. In one example, the inter-mode may refer to adaptive DMVR. c. In one example, a reordering process may be applied to reorder the first motion candidate list and the second motion candidate list separately. d. In another example, the first candidate list and the second motion candidate list may be combined together, and a reordering process may be used to reorder the combined motion candidate list. e. In one example, the refined motion information used in the reordering process can be the block-based MV of the first pass in multi-pass DMVR. 4. In one example, it is proposed to remove redundant motion candidates from one or more motion candidate lists for adaptive DMVR. a. In one example, redundant motion candidates may refer to those motion candidates in the first motion candidate list that have the same motion information as one of the motion candidates in the second motion candidate list. i. In one example, the first motion candidate list may refer to a motion candidate list in a first direction, and the second motion candidate list may refer to a motion candidate list in a second direction. ii. In one example, the first motion candidate list may refer to the motion candidate list of the second direction, and the second motion candidate list may refer to the motion candidate list of the first direction. b. In one example, redundant motion candidates may refer to motion candidates after the first pass of DMVR. 5. It is proposed to generate N motion candidates for a motion candidate list with M as the maximum allowed number of motion candidates, where N>M. a. Use the sequentially selected M motion candidates in prediction / reconstruction. b. Selecting M motion candidates in order by reordering the N motion candidates, and the selected M motion candidates are the first M (eg, with the lowest cost) of the N motion candidates after the reordering. c. In one example, N may be predefined, or signaled in the bitstream, or determined on the fly. i. In one example, N may be different for different codec modes. ii. In another example, N can be 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20. d. In one example, motion information derived from the motion candidate list may be used in the re-ordering process. e. In one example, the above reordering process using refined motion information may be applied. 6. In one example, the cost values generated in the refinement process (eg, TM or DMVR / multi-pass DMVR) may be used again in the reordering process after the refinement process. a. In one example, the final (or minimum or maximum) cost values generated for motion candidates during the refinement process may be stored. b. In one example, when a TM is included in the refinement process, the TM cost can be used again in the re-ranking process. c. In another example, when DMVR / multi-pass DMVR is included in the refinement process, the bilateral matching cost from DMVR can be reused in the re-ranking process. i. In one example, the bilateral matching cost of the first pass in multi-pass DMVR is reused. ii. In one example, the bilateral matching cost of the second pass in multi-pass DMVR is reused. iii. In one example, the bilateral matching cost of the third pass in multi-pass DMVR is used again. d. In one example, the final (or minimum or maximum) cost value generated during the refinement process of the first motion candidate may be compared with the final (or minimum or maximum) cost value generated during the refinement process of the second motion candidate during the re-ranking process. e. In one example, a method to reuse the cost is to use the cost directly in the re-ranking process, rather than determining the cost for each motion candidate. f. In one example, whether to reuse the cost and / or the method to reuse the cost is determined based on the codec information of the video unit, or is indicated in the bitstream. 7. In one example, merge candidates may be reordered according to merge type, and the top X merge candidates of each merge type are used to construct an adaptive DMVR merge candidate list. a. In one example, the merge type may refer to adjacent space and / or time and / or non-adjacent space and / or HMVP and / or in pairs. b. In one example, X may be different for different merge types. i. In one example, for a temporal merging candidate, X may be set equal to 1 or 2. ii. In another example, for the spatial merging candidate, X may be set equal to 1, or 2, or 3, or 4, or 5, or 6, or 7, or 8, or 9, or 10. iii. In another example, for non-adjacent spatial merging candidates, X may be set equal to 1, or 2, or 3, or 4, or 5, or 6, or 7, or 8, or 9, or 10. iv. In another example, for HMVP merge candidates, X may be set equal to 1, or 2, or 3, or 4, or 5, or 6, or 7, or 8, or 9, or 10. v. In another example, for pair-wise merge candidates, X may be set equal to 1, or 2, or 3, or 4, or 5, or 6, or 7, or 8, or 9, or 10. 8. In the above examples, in addition to the existing refinement process, a motion search method may be used to refine at least one motion candidate. a. In one example, the motion candidates may be refined before the first motion refinement process before the re-ordering process. b. In one example, the motion candidates may be refined after a first motion refinement process prior to the re-ordering process. c. In one example, motion candidates may be refined after the re-ranking process. d. In one example, the motion candidates may be refined after a second motion refinement process after the re-ranking process. e. In one example, the search pattern may be in the shape of a diamond or a cross or a square. i. In one example, the neighboring 8 motion candidates of a motion candidate may be searched. ii. In one example, N rounds of searching may be performed. As an example, N=1, or N=2, or N=3. iii. In one example, the motion accuracy to be searched may be predefined, or the motion accuracy is dynamically determined, or the motion accuracy is indicated in the bitstream. f. In one example, the manner of refining the motion candidate may refer to calculating the costs of the motion candidate and other motion candidates in a search mode, wherein the motion candidate with the best cost is used as the refined motion candidate. i. In one example, the cost may be a TM cost. ii. In one example, the cost may be a BM cost. General claims 9. In the above examples, the video unit may refer to a color component / sub-picture / slice / tile / codec tree unit (CTU) / CTU row / CTU group / codec unit CU / prediction unit PU / transform unit TU / codec tree block CTB / codec block CB / prediction block PB / transform block TB / block / sub-block of a block / sub-region within a block / region containing more than one sample or pixel. 10. Whether and / or how to apply the above disclosed method can be notified in one of the following: sequence level / picture level / slice level / tile group level, for example in sequence header / picture header / SPS / VPS / DPS / DCI / PPS / PPS / slice header / tile group header. 11. Whether and / or how to apply the above disclosed method can be notified in one of the following: PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU row / slice / tile / sub-picture / region containing more than one sample or pixel. 12. Whether and / or how to apply the above disclosed method may depend on codec information, such as block size, color format, single / dual tree partitioning, color component, slice / picture type. 5. Embodiment 5.1 Example 1 In the current TM merge design in ECM-3.1, a TM merge candidate list is first constructed. Then adaptive reordering of merge candidates (ARMC) is applied to reorder the merge candidates. Finally, if the conditions for multi-pass DMVR are met, template matching and multi-pass DMVR are used to refine the selected merge candidates. The initial motion information in the TM merge candidate list is used in ARMC, but the refined motion information after template matching and multi-pass DMVR is used for prediction, which may limit the encoding performance. It is proposed to apply ARMC using refined motion information for TM merge mode. Specifically, first, a TM merge candidate list is constructed and all merge candidates are reordered, while only the first 5 TM merge candidates are reordered in ECM-3.1. Second, the TM merge candidates in the TM merge list are divided into N subgroups, and the subgroup size is set equal to 4. Template matching and multi-pass DMVR are applied to all TM merge candidates in the selected subgroup determined by the signaled merge index. Finally, the refined merge candidates in the subgroup are reordered. To reduce the complexity, the following simplification method is applied: 1) The proposed method is only applied to blocks with luma sample size larger than T1, e.g., T1=16 or 32 or 64. 2) The proposed method is only applied to blocks with luma sample size smaller than T2, for example, T2=128, or 256, or 512, or 1024, or 2048. 3) When multiple passes of DMVR are applied to the selected subgroup, only the first pass of DMVR is used. 4) When applying template matching to the selected subgroup, the template size of template matching is set equal to 1. 5) When applying template matching to the selected subgroup, the template matching is terminated early after the diamond search. 5.2 Example 2 In the current adaptive DMVR design in ECM-3.1, an adaptive DMVR merge candidate list is first constructed, and the number of allowed merge candidates can be up to the maximum number of merge candidates (N) of adaptive DMVR signaled in the bitstream, and N can be selected from 0 to 6, and it is set equal to 4 in ECM-3.1. Then adaptive reordering of merge candidates (ARMC) is applied to reorder the merge candidates of adaptive DMVR. . When constructing the adaptive DMVR merge candidate list, it is recommended to increase the allowed number of merge candidates (M), where M is greater than N. ARMC is then applied to reorder the merge candidates. In the encoder, only the first N merge candidates participate in the rate-distortion optimization (RDO) of adaptive DMVR. M is set equal to 10. 5.3 Example 3
[0105] First, when only blocks are flat (w>2×h) or narrow (h>2×w), only the upper template or the left template is used during motion refinement of TM. Second, only the first four merge candidates are reordered in TM merge mode, instead of the first eight candidates in test #2.5b. Third, when building the AMVP list, MVP candidates with TM cost greater than a threshold (the threshold is equal to five times the cost of the first MVP candidate) are skipped. Fourth, TM is extended to perform 1 / 16 pixel MVD accuracy in TM merge mode. Fifth, in TM merge mode, the MV derived from the first motion refinement process is used as the starting point for the final motion refinement process.
[0106] Embodiments of the present disclosure relate to motion candidate list construction.
[0107] As used herein, the term “video unit” or “codec unit” or “block” used herein may refer to one of the following: a color component, a sub-picture, a slice, a tile, a codec tree unit (CTU), a CTU row, a CTU group, a codec unit (CU), a prediction unit (PU), a transform unit (TU), a codec unit (CTB), a codec block (CB), a prediction block (PB), a transform block (TB), a block, a sub-block of a block, a sub-region within a block, or a region containing more than one sample or pixel.
[0108] In the present disclosure, with respect to “blocks encoded using mode N”, the term “mode N” may be a prediction mode (e.g., MODE_INTRA, MODE_INTER, MODE_PLT, MODE_IBC, etc.) or a coding technology (e.g., AMVP, merge, SMVD, BDOF, PROF, DMVR, AMVR, TM, affine, CIIP, GPM, MMVD, BCW, HMVP, SbTMVP, etc.).
[0109] Note that the terms mentioned below are not limited to the specific terms defined in existing standards. Any changes in encoding tools are also applicable.
[0110] Fig.41 A flow chart illustrating a method 4100 for video processing according to an embodiment of the present disclosure. The method 4100 is implemented during conversion between a video unit of a video and a bitstream of the video.
[0111] At block 4110, for conversion between a video unit of a video and a bitstream of the video unit, a processing flow is applied to at least one motion candidate in the motion candidate list. The processing flow includes at least one of the following: a modification process, a motion refinement process, and a second reordering process.
[0112] At block 4120, after the processing flow, at least one subgroup of motion candidates in the motion candidate list is reordered by applying a first reordering process to the motion candidate list. For example, the motion candidates in the motion candidate list are divided into subgroups. In this case, whether to reorder the motion candidates in the first reordering process and / or the method used to reorder the motion candidates in the first reordering process depends on the subgroup of motion candidates.
[0113] At block 4130, conversion is performed based on the reordered motion candidates. In some embodiments, conversion may include encoding the video unit into a bitstream. In some embodiments, conversion may include decoding the video unit from the bitstream. Coding efficiency and performance may be improved compared to conventional techniques.
[0114] In some embodiments, a first subset of motion candidates in the motion candidate list is reordered. For example, only the first subset may be reordered.
[0115] In some embodiments, the last subgroup of motion candidates in the motion candidate list is not reordered. In some other embodiments, the last subgroup of motion candidates is also the first subgroup of motion candidates, and the last subgroup of motion candidates in the motion candidate list is reordered.
[0116] In some embodiments, the first subgroup of motion candidates and the second subgroup of motion candidates are reordered separately. In other words, different subgroups may be reordered separately.
[0117] In some embodiments, a first motion candidate in a first subset of motion candidates and a second motion candidate in a second subset of motion candidates are not compared. Alternatively or additionally, the first motion candidate and the second motion candidate are not reordered. For example, two candidates in different subsets cannot be compared and / or reordered. In some embodiments, if a first subset of motion candidates precedes a second subset of motion candidates, after reordering, the first motion candidate in the first subset precedes the second motion candidate in the second subset.
[0118] In some embodiments, the processing flow includes a simplified motion refinement process. In this case, whether the upper template or the left template is used depends on at least one of the following: block width, or block height. As an example, the block width is represented as W and the block height is represented as H. For example, if W is greater than s×H, the upper template is used. As another example, if W is greater than s×H and the upper template is not available, the left template is used. In some embodiments, if H is greater than s×W, the left template is used. In some other embodiments, if H is greater than s×W and the left template is not available, the upper template is used. In these cases, s can be an integer. For example, s is equal to one of the following: 2, 4, 1, 3, 6, or 8.
[0119] In some embodiments, the processing flow is a motion refinement process. The refined motion information used in the first reordering process may be the final motion information of the motion refinement process. If the motion refinement process includes multiple steps, the motion information after the last step in the motion refinement process may be used in the first reordering process. The multiple steps may include different search shapes to refine the motion vector in template matching. In this case, at least one search step may be skipped. In some embodiments, at least one of the following is skipped: a full pixel diamond search, a full pixel cross search step, a half pixel cross search step, a quarter pixel cross search step, or a 1 / 8 pixel cross search step.
[0120] In some embodiments, the process flow includes a first motion refinement process, in which case the motion information generated in the first motion refinement process is used as starting motion information and further refined by a second motion refinement process before the motion information is used to obtain a prediction or reconstruction of the video unit.
[0121] In some embodiments, if template matching (TM) is included in the first motion refinement process, the motion information generated from TM is used as the starting motion information. In some embodiments, if decoder-side motion vector refinement (DMVR) or multi-pass DMVR is included in the first motion refinement process, the motion information generated from DMVR or multi-pass DMVR is used as the starting motion information.
[0122] Alternatively, the motion information generated in the first motion refinement process is used as intermediate motion information for the second motion refinement process. In some embodiments, if multiple passes of DMVR are included in the second motion refinement process, the first pass of DMVR is skipped. The motion information generated in the first motion refinement process is further refined from the second pass of DMVR. Alternatively, if TM is included in the multiple passes of DMVR, the motion information generated in the first motion refinement process is further refined from the TM. In some embodiments, if TM is included in the second motion refinement process, the motion information generated in the first motion refinement process is used as a starting point for the TM search pattern instead of the first search pattern.
[0123] In some embodiments, the number of motion candidates and / or which motion candidates are refined before reordering is dynamically determined. In this case, a cost is determined for each motion candidate, and the motion candidates are ordered based on the cost. For example, the cost can be a TM cost or a bilateral matching (BM) cost.
[0124] In some embodiments, the top M motion candidates with the lowest cost are refined. In this case, M can be an integer. In some embodiments, M is equal to one of the following: 2, 3, 4, 5, 6, 7, 8, 9 or 10.
[0125] In some embodiments, the M motion candidates are extended to N motion candidates by adding at least one motion vector MV offset to each of the M motion candidates. In this case, N may be an integer greater than M.
[0126] In some embodiments, motion candidates with a cost greater than a threshold are not used in the first refinement process and reordering. In some embodiments, the threshold depends on the best cost of the motion candidates. In some embodiments, the threshold is equal to P×C0, where C0 represents the best cost of the motion candidate and P is an integer. In some embodiments, P is equal to one of the following: 2, 3, 4, 5, 6, 7, 8, 9, or 10.
[0127] In some embodiments, the first N motion candidates with costs greater than a threshold are used in the first refinement process and reordering, and other motion candidates with costs greater than the threshold are not used in the refinement process and reordering. In some embodiments, N is equal to one of the following: 1, 2, 3, 4 or 5.
[0128] In some embodiments, the video unit refers to one of the following: a color component, a sub-picture, a slice, a tile, a codec tree unit (CTU), a CTU row, a CTU group, a codec unit (CU), a prediction unit (PU), a transform unit (TU), a codec unit (CTB), a codec block (CB), a prediction block (PB), a transform block (TB), a block, a sub-block of a block, a sub-region within a block, or a region containing more than one sample or pixel. In some embodiments, after the processing flow, whether and / or how to re-order at least one motion candidate subgroup in the motion candidate list by applying the first re-ordering flow to the motion candidate list is indicated at one of the following: a sequence level, a set of picture level, a picture level, a slice level, a tile group level.
[0129] In some embodiments, after the processing flow, whether and / or how, by applying the first reordering process to the motion candidate list, an indication of at least one motion candidate subgroup in the reordered motion candidate list is indicated at one of the following: sequence header, picture header, sequence parameter set (SPS), video parameter set (VPS), dependency parameter set (DPS), decoding capability information (DCI), picture parameter set (PPS), adaptation parameter set (APS), slice header, tile group header. In some embodiments, after the processing flow, whether and / or how, by applying the first reordering process to the motion candidate list, an indication of at least one motion candidate subgroup in the reordered motion candidate list is included in one of the following: prediction block (PB), transform block (TB), codec block (CB), prediction unit (PU), transform unit (TU), codec unit (CU), virtual pipeline data unit (VPDU), codec tree unit (CTU), CTU row, slice, tile, sub-picture, area containing more than one sample or pixel.
[0130] In some embodiments, method 4100 also includes: determining, based on codec information of the video unit, whether and / or how to reorder at least one motion candidate subgroup in the motion candidate list after the processing flow by applying a first reordering process to the motion candidate list, the codec information including at least one of the following: block size, color format, single tree and / or double tree partitioning, color component, slice type, picture type.
[0131] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video, the bitstream of the video is generated by a method performed by a device for video processing, the method comprising: applying a processing flow to at least one motion candidate in a motion candidate list; after the processing flow, reordering at least one motion candidate subgroup in the motion candidate list by applying a first reordering flow to the motion candidate list, wherein the processing flow comprises at least one of the following: a modification process, a motion refinement process, a second reordering process; and performing conversion based on the reordered motion candidates.
[0132] According to some further embodiments of the present disclosure, a method for storing a video bitstream is provided. The method includes: applying a processing flow to at least one motion candidate in a motion candidate list; after the processing flow, reordering at least one motion candidate subgroup in the motion candidate list by applying a first reordering flow to the motion candidate list, wherein the processing flow includes at least one of the following: a modification process, a motion refinement process, a second reordering process; and performing conversion based on the reordered motion candidates; and storing the bitstream in a non-transitory computer-readable recording medium.
[0133] Fig.42 Flowchart illustrating a method 4200 for video processing according to an embodiment of the present disclosure. The method 4200 is implemented during conversion between a video unit of a video and a bitstream of the video.
[0134] At block 4210, for conversion between a video unit of a video and a bitstream of the video unit, a cost is generated in a refinement process of a motion candidate list for the video unit. The refinement process includes at least one of: template matching (TM), decoding-side motion vector refinement (DMVR), and multi-pass DMVR.
[0135] At block 4220, a re-ranking process is applied to the motion candidate list using the costs. In one example, the cost values generated in a refinement process (eg, TM or DMVR / multi-pass DMVR) may be reused in a re-ranking process after the refinement process.
[0136] At block 4230, conversion is performed based on the motion candidate list. In some embodiments, conversion may include encoding the video unit into a bitstream. In some embodiments, conversion may include decoding the video unit from the bitstream. Coding efficiency and performance may be improved compared to conventional techniques.
[0137] In some embodiments, if the refinement process includes a TM, the TM costs are reused in the re-ranking process. In some embodiments, the method to reuse the costs is to use the costs directly in the re-ranking process, rather than determining the costs for each motion candidate.
[0138] In some embodiments, if the refinement process includes DMVR, the bilateral matching costs from the DMVR are reused in the reordering process. Alternatively or additionally, if the refinement process includes multi-pass DMVR, the bilateral matching costs from the multi-pass DMVR are reused in the reordering process.
[0139] In some embodiments, the bilateral matching cost of the first pass in the multi-pass DMVR is reused. In some embodiments, the bilateral matching cost of the second pass in the multi-pass DMVR is reused. In some embodiments, the bilateral matching cost of the third pass in the multi-pass DMVR is reused.
[0140] In some embodiments, whether to reuse the cost and / or the method for reusing the cost is determined based on the codec information of the video unit. Alternatively or additionally, whether to reuse the cost and / or the method for reusing the cost is determined based on the codec information of the video unit. In some other embodiments, whether to reuse the cost and / or the method for reusing the cost is indicated in the bitstream.
[0141] In some embodiments, a video unit refers to one of the following: a color component, a sub-picture, a slice, a tile, a codec tree unit (CTU), a CTU row, a CTU group, a codec unit (CU), a prediction unit (PU), a transform unit (TU), a codec unit (CTB), a codec block (CB), a prediction block (PB), a transform block (TB), a block, a sub-block of a block, a sub-region within a block, or a region containing more than one sample or pixel. In some embodiments, an indication of whether and / or how to use a cost to apply a reordering process to a motion candidate list is indicated at one of the following: a sequence level, a set of picture level, a picture level, a slice level, a tile group level.
[0142] In some embodiments, an indication of whether and / or how to use the cost to apply the reordering process to the motion candidate list is indicated in one of the following: sequence header, picture header, sequence parameter set (SPS), video parameter set (VPS), dependency parameter set (DPS), decoding capability information (DCI), picture parameter set (PPS), adaptation parameter set (APS), slice header, tile group header. In some embodiments, an indication of whether and / or how to use the cost to apply the reordering process to the motion candidate list is included in one of the following: prediction block (PB), transform block (TB), codec block (CB), prediction unit (PU), transform unit (TU), codec unit (CU), virtual pipeline data unit (VPDU), codec tree unit (CTU), CTU row, slice, tile, sub-picture, area containing more than one sample or pixel.
[0143] In some embodiments, method 4200 also includes: determining whether and / or how to use the cost to apply the reordering process to the motion candidate list based on the codec information of the video unit, the codec information including at least one of the following: block size, color format, single tree and / or dual tree partitioning, color component, slice type, picture type.
[0144] According to another embodiment of the present disclosure, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video, and the bitstream of the video is generated by a method performed by a device for video processing, the method comprising: generating a cost in a refinement process of a motion candidate list for a video unit, wherein the refinement process comprises at least one of the following: template matching (TM), decoding-side motion vector refinement (DMVR), and multi-pass DMVR; applying a reordering process to the motion candidate list using the cost; and performing a conversion based on the motion candidate list.
[0145] According to some further embodiments of the present disclosure, a method for storing a video bitstream is provided. The method includes: generating a cost in a refinement process of a motion candidate list for a video unit, wherein the refinement process includes at least one of the following: template matching (TM), decoding-side motion vector refinement (DMVR), and multi-pass DMVR; applying a reordering process to the motion candidate list using the cost; and performing conversion based on the motion candidate list; and storing the bitstream in a non-transitory computer-readable recording medium.
[0146] Fig.43 A flow chart illustrating a method 4300 for video processing according to an embodiment of the present disclosure. The method 4300 is implemented during conversion between a video unit of a video and a bitstream of the video.
[0147] At block 4310, a refinement process and a reordering process are applied to motion candidates in the motion candidate list for conversion between a video unit of a video and a bitstream of the video unit. During the refinement process, a motion search method is used to refine the motion candidates. For example, in the above-described embodiment, in addition to the existing refinement process, a motion search method may also be used to refine at least one motion candidate.
[0148] At block 4320, based on the refined and reordered motion candidates, conversion is performed. In some embodiments, conversion may include encoding the video unit into a bitstream. In some embodiments, conversion may include decoding the video unit from the bitstream. Coding efficiency and performance may be improved compared to conventional techniques.
[0149] In some embodiments, the way to refine the motion candidate is to determine the cost of the motion candidate and other motion candidates in the search mode of the motion search method. In this case, the target motion candidate with the best cost is used as the refined motion candidate. In some embodiments, the cost is a TM cost. Alternatively or additionally, the cost is a bilateral matching (BM) cost. In one example, the way to refine the motion candidate may refer to calculating the cost of the motion candidate and other motion candidates in the search mode, wherein the motion candidate with the best cost is used as the refined motion candidate. In one example, the cost may be a TM cost. In one example, the cost may be a BM cost.
[0150] In some embodiments, the search pattern of the motion search method is one of the following: diamond, cross, square. In some embodiments, a predetermined number of neighboring motion candidates of a motion candidate are searched. For example, eight neighboring motion candidates of a motion candidate may be searched.
[0151] In some embodiments, N rounds of searching are performed, where N is an integer. For example, N is equal to 1, 2, or 3.
[0152] In some embodiments, the motion accuracy to be searched is predefined; wherein the motion accuracy is dynamically determined. In some other embodiments, the motion accuracy is indicated in the bitstream.
[0153] In some embodiments, the motion candidates are refined before a first motion refinement process before the reordering process. Alternatively, the motion candidates are refined after a first motion refinement process before the reordering process. In some other embodiments, the motion candidates are refined after the reordering process. In some embodiments, the motion candidates are refined after a second motion refinement process after the reordering process.
[0154] In some embodiments, the video unit refers to one of the following: a color component, a sub-picture, a slice, a tile, a codec tree unit (CTU), a CTU row, a CTU group, a codec unit (CU), a prediction unit (PU), a transform unit (TU), a codec unit (CTB), a codec block (CB), a prediction block (PB), a transform block (TB), a block, a sub-block of a block, a sub-region within a block, or a region containing more than one sample or pixel. In some embodiments, whether and / or how to apply the refinement process and the reordering process to the motion candidates in the motion candidate list is indicated at one of the following: a sequence level, a set of picture level, a picture level, a slice level, a tile group level.
[0155] In some embodiments, an indication of whether and / or how to apply the refinement process and the reordering process to the motion candidates in the motion candidate list is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), a decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, a tile group header. In some embodiments, an indication of whether and / or how to apply the refinement process and the reordering process to the motion candidates in the motion candidate list is included in one of the following: a prediction block (PB), a transform block (TB), a codec block (CB), a prediction unit (PU), a transform unit (TU), a codec unit (CU), a virtual pipeline data unit (VPDU), a codec tree unit (CTU), a CTU row, a slice, a tile, a sub-picture, an area containing more than one sample or pixel.
[0156] In some embodiments, method 4300 also includes: determining whether and / or how to apply the refinement process and the reordering process to the motion candidates in the motion candidate list based on the codec information of the video unit, the codec information including at least one of the following: block size, color format, single tree and / or double tree partitioning, color component, slice type, picture type.
[0157] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bit stream of a video, and the bit stream of the video is generated by a method performed by a device for video processing, the method comprising: applying a refinement process and a reordering process to motion candidates in a motion candidate list, wherein in the refinement process, the motion candidates are refined using a motion search method; and based on the refined and reordered motion candidates, performing conversion.
[0158] According to some further embodiments of the present disclosure, a method for storing a video bitstream is provided. The method includes: applying a refinement process and a reordering process to motion candidates in a motion candidate list, wherein in the refinement process, the motion candidates are refined using a motion search method; and based on the refined and reordered motion candidates, performing conversion; and storing the bitstream in a non-transitory computer-readable recording medium.
[0159] Embodiments of the present disclosure may be described according to the following clauses, features of which may be combined in any reasonable way.
[0160] Clause 1. A video processing method, comprising: applying a processing flow to at least one motion candidate in a motion candidate list for conversion between a video unit of a video and a bitstream of the video unit; after the processing flow, reordering at least one subgroup of motion candidates in the motion candidate list by applying a first reordering flow to the motion candidate list, wherein the processing flow includes at least one of the following: a modification process, a motion refinement process, a second reordering process; and performing conversion based on the reordered motion candidates.
[0161] Clause 2. A method according to clause 1, wherein the motion candidates in the motion candidate list are divided into subgroups, and wherein whether to reorder the motion candidates in the first reordering process and / or the method used to reorder the motion candidates in the first reordering process depends on the subgroups of the motion candidates.
[0162] Clause 3. The method of clause 2, wherein a first subset of motion candidates in the motion candidate list is reordered.
[0163] Clause 4. The method of clause 2, wherein a last subset of motion candidates in the motion candidate list is not reordered.
[0164] Clause 5. The method of clause 4, wherein the last subset of motion candidates is also the first subset of motion candidates, the last subset of motion candidates in the motion candidate list being reordered.
[0165] Clause 6. The method of clause 2, wherein the first subset of motion candidates and the second subset of motion candidates are reordered separately.
[0166] Clause 7. The method of clause 2, wherein a first motion candidate in the first subset of motion candidates and a second motion candidate in the second subset of motion candidates are not compared, and / or wherein the first motion candidate and the second motion candidate are not reordered.
[0167] Clause 8. The method of clause 2, wherein if a first subset of motion candidates precedes a second subset of motion candidates, after reordering, a first motion candidate in the first subset precedes a second motion candidate in the second subset.
[0168] Clause 9. The method of clause 1, wherein the processing flow includes a simplified motion refinement process, and wherein whether the top template or the left template is used depends on at least one of: a block width, or a block height.
[0169] Clause 10. The method according to clause 9, wherein the upper template is used if W is greater than s×H, where s is an integer, W represents the block width, and H represents the block height.
[0170] Clause 11. The method of clause 9, wherein if W is greater than s×H and the upper template is not available, the left template is used, where s is an integer, W represents the block width, and H represents the block height.
[0171] Clause 12. The method of clause 9, wherein the left template is used if H is greater than s×W, where s is an integer, W represents the block width, and H represents the block height.
[0172] Clause 13. The method of clause 9, wherein if H is greater than s×W and the left template is not available, the upper template is used, where s is an integer, W represents the block width, and H represents the block height.
[0173] Clause 14. The method of any one of clauses 10-13, wherein s is equal to one of the following: 2, 4, 1, 3, 6 or 8.
[0174] Clause 15. A method according to clause 1, wherein the processing flow is a motion refinement process, wherein the refined motion information used in the first reordering process is the final motion information of the motion refinement process; wherein if the motion refinement process includes multiple steps, the motion information after the last step in the motion refinement process is used in the first reordering process, wherein the multiple steps include different search shapes to refine the motion vector in template matching, and wherein at least one search step is skipped.
[0175] Item 16. The method according to Item 15, wherein at least one of the following is skipped: a full-pixel diamond search, a full-pixel cross-search step, a half-pixel cross-search step, a quarter-pixel cross-search step, a 1 / 8-pixel cross-search step.
[0176] Clause 17. A method according to clause 1, wherein the processing flow includes a first motion refinement process, and before the motion information is used to obtain a prediction or reconstruction of the video unit, the motion information generated in the first motion refinement process is used as the starting motion information and is further refined by a second motion refinement process.
[0177] Clause 18. The method of clause 17, wherein if Template Matching (TM) is included in the first motion refinement process, the motion information generated from the TM is used as the starting motion information.
[0178] Clause 19. The method of clause 17, wherein if decoding-side motion vector refinement (DMVR) or multi-pass DMVR is included in the first motion refinement process, the motion information generated from the DMVR or the multi-pass DMVR is used as the starting motion information.
[0179] Clause 20. The method of clause 17, wherein the motion information generated in the first motion refinement pass is used as intermediate motion information for the second motion refinement pass.
[0180] Clause 21. A method according to Clause 20, wherein if a multi-pass DMVR is included in the second motion refinement process, the first-pass DMVR is skipped, and wherein the motion information generated in the first motion refinement process is further refined from the second-pass DMVR, or wherein if TM is included in the multi-pass DMVR, the motion information generated in the first motion refinement process is further refined from the TM.
[0181] Clause 22. The method of clause 20, wherein if TM is included in the second motion refinement process, the motion information generated in the first motion refinement process is used as a starting point for the TM search pattern instead of the first search pattern.
[0182] Clause 23. The method of clause 1, wherein the number of motion candidates and / or which motion candidates are refined prior to re-ranking is determined dynamically, and wherein a cost is determined for each motion candidate and the motion candidates are ranked based on the cost.
[0183] Clause 24. The method of clause 23, wherein the top M motion candidates with lowest costs are refined, and wherein M is an integer.
[0184] Clause 25. The method of clause 24, wherein M is equal to one of the following: 2, 3, 4, 5, 6, 7, 8, 9 or 10.
[0185] Clause 26. The method of clause 24, wherein the M motion candidates are extended to N motion candidates by adding at least one motion vector MV offset to each of the M motion candidates, and wherein N is an integer greater than M.
[0186] Clause 27. The method of clause 23, wherein motion candidates having a cost greater than a threshold are not used in the first refinement process and the re-ranking.
[0187] Clause 28. The method of clause 27, wherein the threshold value depends on the best cost of the motion candidate.
[0188] Clause 29. The method of clause 27, wherein the threshold is equal to P×C0, where C0 represents the best cost of the motion candidate and P is an integer.
[0189] Clause 30. The method of clause 29, wherein P is equal to one of the following: 2, 3, 4, 5, 6, 7, 8, 9 or 10.
[0190] Clause 31. The method of clause 27, wherein the first N motion candidates having costs greater than a threshold are used in a first refinement process and re-ordering, and other motion candidates having costs greater than the threshold are not used in the refinement process and re-ordering.
[0191] Clause 32. The method of clause 31, wherein N is equal to one of the following: 1, 2, 3, 4 or 5.
[0192] Clause 33. A method according to any one of clauses 1 to 32, wherein the video unit refers to one of the following: color component, sub-picture, slice, tile, codec tree unit (CTU), CTU row, CTU group, codec unit CU, prediction unit PU, transform unit TU, codec tree block CTB, codec block CB, prediction block PB, transform block TB, block, sub-block of a block, sub-region within a block, region containing more than one sample or pixel.
[0193] Clause 34. A method according to any one of clauses 1 to 32, wherein after the processing process, whether and / or how an indication of at least one motion candidate subgroup in the motion candidate list is reordered by applying a first reordering process to the motion candidate list is indicated at one of the following: sequence level, a group of picture level, picture level, slice level, tile group level.
[0194] Clause 35. A method according to any one of clauses 1 to 32, wherein after the processing flow, whether and / or how an indication of at least one motion candidate subgroup in the motion candidate list is reordered by applying a first reordering process to the motion candidate list is indicated at one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptive parameter set (APS), a slice header, a tile group header.
[0195] Clause 36. A method according to any one of clauses 1 to 32, wherein after the processing flow, whether and / or how an indication of at least one motion candidate subgroup in the motion candidate list is reordered by applying a first reordering flow to the motion candidate list is included in one of the following: prediction block (PB), transform block (TB), codec block (CB), prediction unit (PU), transform unit (TU), codec unit (CU), virtual pipeline data unit (VPDU), codec tree unit (CTU), CTU row, slice, tile, sub-picture, area containing more than one sample or pixel.
[0196] Clause 37. The method according to any one of clauses 1 to 32 further includes: determining, based on the codec information of the video unit, whether and / or how to reorder at least one motion candidate subgroup in the motion candidate list after the processing flow by applying a first reordering process to the motion candidate list, the codec information including at least one of the following: block size, color format, single tree and / or double tree partitioning, color component, slice type, picture type.
[0197] Clause 38. A video processing method, comprising: for conversion between a video unit of a video and a bitstream of the video unit, generating a cost in a refinement process of a motion candidate list for the video unit, wherein the refinement process comprises at least one of the following: template matching (TM), decoding-side motion vector refinement (DMVR), multi-pass DMVR; applying a reordering process to the motion candidate list using the cost; and performing conversion based on the motion candidate list.
[0198] Clause 39. The method of clause 38, wherein if the refinement process includes a TM, the TM cost is reused in the reordering process.
[0199] Clause 40. The method of clause 38, wherein the method for reusing the costs is to use the costs directly in the reordering process rather than determining the costs for each motion candidate.
[0200] Clause 41. A method according to Clause 38, wherein if the refinement process includes DMVR, the bilateral matching costs from the DMVR are reused in the reordering process, and / or wherein if the refinement process includes multiple passes of DMVR, the bilateral matching costs from the multiple passes of DMVR are reused in the reordering process.
[0201] Clause 42. The method of clause 41, wherein the bilateral matching cost of the first pass in the multi-pass DMVR is reused.
[0202] Clause 43. The method of clause 41, wherein the bilateral matching cost of the second pass in the multi-pass DMVR is reused.
[0203] Clause 44. The method of clause 41, wherein the bilateral matching cost of the third pass in the multi-pass DMVR is reused.
[0204] Clause 45. A method according to Clause 38, wherein whether to reuse the cost and / or the method for reusing the cost is determined based on the encoding and decoding information of the video unit, wherein whether to reuse the cost and / or the method for reusing the cost is dynamically determined, or wherein whether to reuse the cost and / or the method for reusing the cost is indicated in the bitstream.
[0205] Clause 46. A method according to any one of clauses 38 to 45, wherein the video unit refers to one of the following: color component, sub-picture, slice, tile, codec tree unit (CTU), CTU row, TU group, codec unit (CU), prediction unit (PU), transform unit (TU), codec tree block (CTB), codec block (CB), prediction block (PB), transform block (TB), block, sub-block of a block, sub-region within a block, region containing more than one sample or pixel.
[0206] Clause 47. A method according to any one of clauses 38 to 45, wherein an indication of whether and / or how to use a cost to apply a reordering process to a motion candidate list is indicated at one of: a sequence level, a group of pictures level, a picture level, a slice level, a tile group level.
[0207] Clause 48. A method according to any one of clauses 38 to 45, wherein an indication of whether and / or how to use the cost to apply the reordering process to the motion candidate list is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptive parameter set (APS), a slice header, a tile group header.
[0208] Clause 49. A method according to any one of clauses 38 to 45, wherein an indication of whether and / or how to use the cost to apply the reordering process to the motion candidate list is included in one of the following: prediction block (PB), transform block (TB), codec block (CB), prediction unit (PU), transform unit (TU), codec unit (CU), virtual pipeline data unit (VPDU), codec tree unit (CTU), CTU row, slice, tile, sub-picture, area containing more than one sample or pixel.
[0209] Clause 50. The method according to any one of clauses 38 to 45 further includes: determining whether and / or how to use the cost to apply the reordering process to the motion candidate list based on the codec information of the video unit, the codec information including at least one of the following: block size, color format, single tree and / or dual tree partitioning, color component, slice type, picture type.
[0210] Item 51. A video processing method, comprising: for conversion between a video unit of a video and a bitstream of the video unit, applying a refinement process and a reordering process to motion candidates in a motion candidate list, wherein in the refinement process, the motion candidates are refined using a motion search method; and performing conversion based on the refined and reordered motion candidates.
[0211] Clause 52. The method of clause 51, wherein the motion candidate is refined by determining costs of the motion candidate and other motion candidates in a search mode of the motion search method, and wherein a target motion candidate having the best cost is used as the refined motion candidate.
[0212] Clause 53. The method of clause 52, wherein the cost is a TM cost, or wherein the cost is a bilateral matching (BM) cost.
[0213] Clause 54. The method according to clause 51, wherein the search pattern of the motion search method is one of the following: diamond, cross, square.
[0214] Clause 55. The method of clause 54, wherein a predetermined number of neighboring motion candidates of the motion candidate are searched.
[0215] Clause 56. The method of clause 54, wherein N rounds of searching are performed, where N is an integer.
[0216] Clause 57. The method of clause 56, wherein N is equal to one of the following: 1, 2 or 3.
[0217] Clause 58. The method of clause 54, wherein the motion accuracy to be searched is predefined; wherein the motion accuracy is dynamically determined, or wherein the motion accuracy is indicated in the bitstream.
[0218] Clause 59. The method of clause 51, wherein the motion candidates are refined prior to a first motion refinement process prior to the reordering process.
[0219] Clause 60. The method of clause 51, wherein the motion candidates are refined after a first motion refinement process prior to the reordering process.
[0220] Clause 61. The method of clause 51, wherein the motion candidates are refined after a reordering process.
[0221] Clause 62. The method of clause 51, wherein the motion candidates are refined after a second motion refinement process after the reordering process.
[0222] Clause 63. A method according to any one of clauses 51 to 62, wherein the video unit refers to one of the following: color component, sub-picture, slice, tile, codec tree unit (CTU), CTU row, CTU group, codec unit (CU), prediction unit (PU), transform unit (TU), codec tree block (CTB), codec block (CB), prediction block (PB), transform block (TB), block, sub-block of a block, sub-region within a block, region containing more than one sample or pixel.
[0223] Clause 64. A method according to any one of clauses 51 to 62, wherein an indication of whether and / or how a refinement process and a reordering process are applied to motion candidates in a motion candidate list is indicated at one of: a sequence level, a group of pictures level, a picture level, a slice level, a tile group level.
[0224] Clause 65. A method according to any one of clauses 51 to 62, wherein an indication of whether and / or how to apply a refinement process and a reordering process to motion candidates in a motion candidate list is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptive parameter set (APS), a slice header, and a tile group header.
[0225] Clause 66. A method according to any one of clauses 51 to 62, wherein an indication of whether and / or how to apply a refinement process and a reordering process to a motion candidate in a motion candidate list is included in one of the following: a prediction block (PB), a transform block (TB), a codec block (CB), a prediction unit (PU), a transform unit (TU), a codec unit (CU), a virtual pipeline data unit (VPDU), a codec tree unit (CTU), a CTU row, a slice, a tile, a sub-picture, an area containing more than one sample or pixel.
[0226] Clause 67. The method according to any one of clauses 51 to 62 further includes: determining whether and / or how to apply the refinement process and the reordering process to the motion candidates in the motion candidate list based on the codec information of the video unit, the codec information including at least one of the following: block size, color format, single tree and / or double tree partitioning, color component, slice type, picture type.
[0227] Clause 68. The method of any one of clauses 1 to 67, wherein converting comprises encoding the video unit into a bitstream.
[0228] Clause 69. The method of any one of clauses 1 to 67, wherein converting comprises decoding the video unit from a bitstream.
[0229] Clause 70. An apparatus for video processing, comprising a processor and a non-volatile memory having instructions, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of clauses 1 to 69.
[0230] Clause 71. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method of any one of clauses 1 to 69.
[0231] Item 72. A non-transitory computer-readable recording medium storing a video bitstream, the video bitstream being generated by a method performed by a device for video processing, wherein the method comprises: applying a processing flow to at least one motion candidate in a motion candidate list for conversion between a video unit of a video and a bitstream of the video unit; after the processing flow, reordering at least one subgroup of motion candidates in the motion candidate list by applying a first reordering flow to the motion candidate list, wherein the processing flow comprises at least one of the following: a modification process, a motion refinement process, or a second reordering process; and generating a bitstream based on the reordered motion candidates.
[0232] Clause 73. A method for storing a bitstream of a video, comprising: applying a processing flow to at least one motion candidate in a motion candidate list for a video unit of the video; after the processing flow, reordering at least one subgroup of motion candidates in the motion candidate list by applying a first reordering flow to the motion candidate list, wherein the processing flow includes at least one of the following: a modification process, a motion refinement process, or a second reordering process; generating a bitstream based on the reordered motion candidates; and storing the bitstream in a non-transitory computer-readable recording medium.
[0233] Clause 74. A non-transitory computer-readable recording medium storing a bitstream of a video, the bitstream of the video being generated by a method performed by a device for video processing, wherein the method comprises: generating a cost in a refinement process of a motion candidate list for a video unit, wherein the refinement process comprises at least one of the following: template matching (TM), decoding-side motion vector refinement (DMVR), multi-pass DMVR; applying a reordering process to the motion candidate list using the cost; and generating a bitstream based on the motion candidate list.
[0234] Clause 75. A method for storing a bitstream of a video, comprising: generating costs during a refinement process of a motion candidate list for a video unit of the video, wherein the refinement process comprises at least one of the following: template matching (TM), decoder-side motion vector refinement (DMVR), multi-pass DMVR; applying a reordering process to the motion candidate list using the costs; generating a bitstream based on the motion candidate list; and storing the bitstream in a non-transitory computer-readable recording medium.
[0235] Clause 76. A non-transitory computer-readable recording medium storing a bitstream of a video, the bitstream of the video being generated by a method performed by a device for video processing, wherein the method comprises: applying a refinement process and a reordering process to motion candidates in a motion candidate list for a video unit of the video, wherein in the refinement process, the motion candidates are refined using a motion search method; and generating a bitstream based on the refined and reordered motion candidates.
[0236] Clause 77. A method for storing a bitstream of a video, comprising: applying a refinement process and a reordering process to motion candidates in a motion candidate list for a video unit of the video, wherein in the refinement process, the motion candidates are refined using a motion search method; generating a bitstream based on the refined and reordered motion candidates; and storing the bitstream in a non-transitory computer-readable recording medium. Example Device
[0237] Fig.44A block diagram of a computing device 4400 in which various embodiments of the present disclosure may be implemented is shown. The computing device 4400 may be implemented as a source device 110 (or video encoder 114 or 200) or a destination device 120 (or video decoder 124 or 300), or may be included in a source device 110 (or video encoder 114 or 200) or a destination device 120 (or video decoder 124 or 300).
[0238] It should be understood that Fig.44 The computing device 4400 shown in the figure is for illustration purposes only and does not in any way imply any limitation on the functionality and scope of the embodiments of the present disclosure.
[0239] like Fig.44 As shown, computing device 4400 includes a general computing device 4400. Computing device 4400 may include at least one or more processors or processing units 4410, memory 4420, storage unit 4430, one or more communication units 4440, one or more input devices 4450, and one or more output devices 4460.
[0240] In some embodiments, the computing device 4400 can be implemented as any user terminal or server terminal with computing power. The server terminal can be a server provided by a service provider, a large computing device, etc. The user terminal can be, for example, any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet computer, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device, or any combination thereof, and includes the accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 4400 can support any type of interface to the user (such as a "wearable" circuit device, etc.).
[0241] The processing unit 4410 may be a physical processor or a virtual processor and may implement various processes based on a program stored in the memory 4420. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the computing device 4400. The processing unit 4410 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.
[0242] The computing device 4400 typically includes various computer storage media. Such media can be any media accessible by the computing device 4400, including but not limited to volatile media and non-volatile media, or removable media and non-removable media. The memory 4420 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM) or flash memory) or any combination thereof. The storage unit 4430 can be any removable or non-removable medium, and can include machine-readable media, such as a memory, a flash drive, a disk, or other media that can be used to store information and / or data and can be accessed in the computing device 4400.
[0243] The computing device 4400 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Fig.44 Although not shown in the figure, a magnetic disk drive for reading from and / or writing to a removable nonvolatile magnetic disk and an optical disk drive for reading from and / or writing to a removable nonvolatile optical disk may be provided. In this case, each drive may be connected to the bus (not shown) via one or more data medium interfaces.
[0244] The communication unit 4440 communicates with another computing device via a communication medium. In addition, the functions of the components in the computing device 4400 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 4400 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general network nodes.
[0245] Input device 4450 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 4460 may be one or more of various output devices, such as a display, speaker, printer, etc. With the aid of communication unit 4440, computing device 4400 may also communicate with one or more external devices (not shown), such as storage devices and display devices, and computing device 4400 may also communicate with one or more devices that enable a user to interact with computing device 4400, or any device that enables computing device 4400 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.), if necessary. Such communication may be performed via an input / output (I / O) interface (not shown).
[0246] In some embodiments, some or all components of the computing device 4400 may also be arranged in a cloud computing architecture rather than being integrated in a single device. In a cloud computing architecture, components may be provided remotely and work together to implement the functions described in the present disclosure. In some embodiments, cloud computing provides computing, software, data access and storage services, which will not require the end user to know the physical location or configuration of the system or hardware that provides these services. In various embodiments, cloud computing provides services via a wide area network (e.g., the Internet) using a suitable protocol. For example, a cloud computing provider provides applications via a wide area network, which can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data may be stored on a remote server. The computing resources in a cloud computing environment may be merged or distributed at the location of a remote data center. Cloud computing infrastructures may provide services through a shared data center, although they appear as a single access point for users. Therefore, cloud computing architectures may be used to provide components and functions described herein from a service provider at a remote location. Alternatively, they may be provided by a conventional server, or installed directly or otherwise on a client device.
[0247] In an embodiment of the present disclosure, the computing device 4400 may be used to implement video encoding / decoding. The memory 4420 may include one or more video codec modules 4425 having one or more program instructions. These modules can be accessed and executed by the processing unit 4410 to perform the functions of the various embodiments described herein.
[0248] In an example embodiment of performing video encoding, an input device 4450 may receive video data as input 4470 to be encoded. The video data may be processed by, for example, a video codec module 4425 to generate an encoded bitstream. The encoded bitstream may be provided as output 4480 via an output device 4460.
[0249] In an example embodiment of performing video decoding, the input device 4450 may receive an encoded bitstream as input 4470. The encoded bitstream may be processed by, for example, the video codec module 4425 to generate decoded video data. The decoded video data may be provided as output 4480 via the output device 4460.
[0250] Although the present disclosure has been specifically shown and described with reference to the preferred embodiments of the present disclosure, it will be understood by those skilled in the art that various changes can be made in form and detail without departing from the spirit and scope of the present application as defined by the appended claims. These changes are intended to be encompassed by the scope of the present application. Therefore, the foregoing description of the embodiments of the present application is not intended to be limiting.
Claims
1. A video processing method comprising: For conversion between a video unit of a video and a bitstream of the video unit, applying a processing flow to at least one motion candidate in a motion candidate list; After the processing flow, at least one subgroup of motion candidates in the motion candidate list is reordered by applying a first reordering process to the motion candidate list, wherein the processing flow includes at least one of the following: a modification process, a motion refinement process, and a second reordering process; as well as Based on the reordered motion candidates, the conversion is performed.
2. The method according to claim 1, wherein the motion candidates in the motion candidate list are divided into subgroups, and Whether to reorder motion candidates in the first reordering process and / or a method used to reorder motion candidates in the first reordering process depends on the subgroup of the motion candidates. The method of claim 2 , wherein a first subset of motion candidates in the motion candidate list is reordered. The method of claim 2 , wherein a last subgroup of motion candidates in the motion candidate list is not reordered. 5 . The method of claim 4 , wherein the last subgroup of motion candidates is also the first subgroup of motion candidates, the last subgroup of motion candidates in the motion candidate list being reordered. The method of claim 2 , wherein the first subset of motion candidates and the second subset of motion candidates are reordered separately.
7. The method of claim 2, wherein a first motion candidate in the first subgroup of motion candidates and a second motion candidate in the second subgroup of motion candidates are not compared, and / or The first motion candidate and the second motion candidate are not reordered.
8. The method of claim 2, wherein if a first subset of motion candidates precedes a second subset of motion candidates, after the reordering, a first motion candidate in the first subset precedes a second motion candidate in the second subset.
9. The method of claim 1, wherein the processing flow includes a simplified motion refinement process, and Whether the top template or the left template is used depends on at least one of the following: block width, or block height.
10. The method of claim 9, wherein if W is greater than s×H, the upper template is used, Wherein s is an integer, W represents the block width, and H represents the block height.
11. The method of claim 9, wherein if W is greater than s×H and the upper template is not available, the left template is used, Wherein s is an integer, W represents the block width, and H represents the block height.
12. The method of claim 9, wherein if H is greater than s×W, the left template is used, Wherein s is an integer, W represents the block width, and H represents the block height.
13. The method according to claim 9, if H is greater than s×W and the left template is not available, the upper template is used, Wherein s is an integer, W represents the block width, and H represents the block height.
14. The method according to any one of claims 10-13, wherein s is equal to one of the following: 2, 4, 1, 3, 6 or 8.
15. The method of claim 1, wherein the processing flow is a motion refinement process, The refined motion information used in the first reordering process is the final motion information of the motion refinement process; wherein if the motion refinement process comprises a plurality of steps, the motion information after the last step in the motion refinement process is used in the first reordering process, wherein the plurality of steps include different search shapes to refine the motion vector in template matching, and At least one of the search steps is skipped.
16. The method of claim 15, wherein at least one of the following is skipped: Full pixel diamond search, Full pixel cross search step, Half-pixel cross search step, Quarter-pixel cross search step, 1 / 8 pixel cross search step.
17. The method of claim 1, wherein the processing flow comprises a first motion refinement process, The motion information generated in the first motion refinement process is used as starting motion information and is further refined by a second motion refinement process before the motion information is used to obtain a prediction or reconstruction of the video unit.
18. The method of claim 17, wherein if Template Matching (TM) is included in the first motion refinement process, the motion information generated from TM is used as the starting motion information.
19. The method of claim 17, wherein if decoding-side motion vector refinement (DMVR) or multi-pass DMVR is included in the first motion refinement process, the motion information generated from the DMVR or the multi-pass DMVR is used as the starting motion information.
20. The method of claim 17, wherein the motion information generated in the first motion refinement process is used as intermediate motion information for the second motion refinement process.
21. The method of claim 20, wherein if multiple passes of DMVR are included in the second motion refinement process, a first pass of DMVR is skipped, and wherein the motion information generated in the first motion refinement process is further refined from a second pass DMVR, or Wherein if TM is included in the multi-pass DMVR, the motion information generated in the first motion refinement process is further refined from the TM.
22. The method of claim 20, wherein if TM is included in the second motion refinement process, the motion information generated in the first motion refinement process is used as a starting point for a TM search pattern instead of the first search pattern.
23. The method of claim 1, wherein the number of motion candidates and / or which motion candidates are refined prior to said re-ordering is dynamically determined, and A cost is determined for each motion candidate, and the motion candidates are sorted based on the cost.
24. The method of claim 23, wherein the top M motion candidates with lowest costs are refined, and wherein M is an integer.
25. The method of claim 24, wherein M is equal to one of the following: 2, 3, 4, 5, 6, 7, 8, 9 or 10.
26. The method of claim 24, wherein the M motion candidates are expanded to N motion candidates by adding at least one motion vector (MV) offset to each of the M motion candidates, and Wherein N is an integer greater than M.
27. The method of claim 23, wherein motion candidates having a cost greater than a threshold are not used in the first refinement process and the re-ranking.
28. The method of claim 27, wherein the threshold depends on a best cost of a motion candidate.
29. The method of claim 27, wherein the threshold is equal to P x C0, where C0 represents the best cost of the motion candidate and P is an integer.
30. The method of claim 29, wherein P is equal to one of the following: 2, 3, 4, 5, 6, 7, 8, 9, or 10.
31. The method of claim 27, wherein the top N motion candidates having costs greater than the threshold are used in the first refinement process and the re-ranking, and Other motion candidates having costs greater than the threshold are not used in the refinement process and the re-ranking.
32. The method of claim 31, wherein N is equal to one of the following: 1, 2, 3, 4 or 5.
33. The method according to any one of claims 1 to 32, wherein the video unit is one of: Color component, Sub-picture, Strip, tiles, Codec Tree Unit (CTU), CTU line, CTU group, Codec unit CU, Prediction unit PU, Transformation unit TU, Codec tree block CTB, Codec block CB, Prediction block PB, Transformation block TB, piece, Sub-blocks of blocks, Sub-regions within a block, An area that contains more than one sample or pixel.
34. The method according to any one of claims 1 to 32, wherein after the processing flow, an indication of whether and / or how to reorder at least one motion candidate subgroup in the motion candidate list by applying the first reordering flow to the motion candidate list is indicated at one of the following: Sequence level, A set of picture levels, Picture level, Stripe level, Tile group level.
35. The method according to any one of claims 1 to 32, wherein after the processing flow, an indication of whether and / or how to reorder at least one motion candidate subgroup in the motion candidate list by applying the first reordering flow to the motion candidate list is indicated at one of the following: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependent Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), Strip head, Tile group header.
36. The method according to any one of claims 1 to 32, wherein after the processing flow, an indication of whether and / or how to reorder at least one motion candidate subgroup in the motion candidate list by applying the first reordering flow to the motion candidate list is included in one of the following: Prediction Block (PB), Transform Block (TB), Codec Block (CB), Prediction Unit (PU), Transformation Unit (TU), Codec Unit (CU), Virtual pipeline data unit (VPDU), Codec Tree Unit (CTU), CTU line, Strip, tiles, Sub-picture, An area that contains more than one sample or pixel.
37. The method according to any one of claims 1 to 32, further comprising: determining, based on codec information of the video unit, whether and / or how to reorder at least one motion candidate subgroup in the motion candidate list by applying the first reordering process to the motion candidate list after the processing flow, wherein the codec information includes at least one of the following: Block size, Color format, Single-tree and / or dual-tree partitioning, Color component, Strip type, Image type.
38. A video processing method, comprising: For conversion between a video unit of a video and a bitstream of the video unit, a cost is generated in a refinement process of a motion candidate list for the video unit, wherein the refinement process comprises at least one of: template matching (TM), decoder-side motion vector refinement (DMVR), and multi-pass DMVR; applying a re-ranking process to the motion candidate list using the costs; as well as The converting is performed based on the motion candidate list.
39. The method of claim 38, wherein if the refinement process includes the TM, TM costs are reused in the reordering process.
40. The method of claim 38, wherein the method for reusing the cost is to use the cost directly in the reordering process rather than determining a cost for each motion candidate.
41. The method of claim 38, wherein if the refinement process includes the DMVR, the bilateral matching costs from the DMVR are reused in the reordering process, and / or Wherein if the refinement process includes the multi-pass DMVR, the bilateral matching costs from the multi-pass DMVR are reused in the reordering process.
42. The method of claim 41, wherein the bilateral matching cost of the first pass in the multi-pass DMVR is reused.
43. The method of claim 41, wherein the bilateral matching cost of the second pass in the multi-pass DMVR is reused.
44. The method of claim 41, wherein the bilateral matching cost of the third pass in the multi-pass DMVR is reused.
45. The method of claim 38, wherein whether to reuse the cost and / or a method for reusing the cost is determined based on codec information of the video unit, wherein whether to reuse the cost and / or the method for reusing the cost is dynamically determined, or Whether the cost is reused and / or a method for reusing the cost is indicated in the bitstream.
46. The method of any one of claims 38 to 45, wherein the video unit is one of: Color component, Sub-picture, Strip, tiles, Codec Tree Unit (CTU), CTU line, CTU group, Codec Unit (CU), Prediction Unit (PU), Transformation Unit (TU), Codec Tree Block (CTB), Codec Block (CB), Prediction Block (PB), Transform Block (TB), piece, Sub-blocks of blocks, Sub-regions within a block, An area that contains more than one sample or pixel.
47. A method according to any one of claims 38 to 45, wherein an indication of whether and / or how to apply the reordering process to the motion candidate list using the cost is indicated at one of: Sequence level, A set of picture levels, Picture level, Stripe level, Tile group level.
48. A method according to any one of claims 38 to 45, wherein an indication of whether and / or how to apply the reordering process to the motion candidate list using the cost is indicated in one of: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependent Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), Strip head, Tile group header.
49. The method of any one of claims 38 to 45, wherein an indication of whether and / or how to apply the reordering process to the motion candidate list using the cost is comprised in one of: Prediction Block (PB), Transform Block (TB), Codec Block (CB), Prediction Unit (PU), Transformation Unit (TU), Codec Unit (CU), Virtual pipeline data unit (VPDU), Codec Tree Unit (CTU), CTU line, Strip, tiles, Sub-picture, An area that contains more than one sample or pixel.
50. The method according to any one of claims 38 to 45, further comprising: determining whether and / or how to use the cost to apply the reordering process to the motion candidate list based on codec information of the video unit, the codec information comprising at least one of the following: Block size, Color format, Single-tree and / or dual-tree partitioning, Color component, Strip type, Image type.
51. A video processing method, comprising: For conversion between a video unit of a video and a bitstream of the video unit, applying a refinement process and a reordering process to motion candidates in a motion candidate list, wherein in the refinement process, the motion candidates are refined using a motion search method; as well as Based on the refined and reordered motion candidates, the conversion is performed.
52. The method of claim 51, wherein the motion candidate is refined by determining the cost of the motion candidate and other motion candidates in a search mode of the motion search method, and The target motion candidate with the best cost is used as the refined motion candidate.
53. The method of claim 52, wherein the cost is a TM cost, or The cost is the bilateral matching (BM) cost.
54. The method according to claim 51, wherein the search pattern of the motion search method is one of the following: diamond, cross, square.
55. The method of claim 54, wherein a predetermined number of neighboring motion candidates of the motion candidate are searched.
56. The method of claim 54, wherein N rounds of searching are performed, where N is an integer.
57. A method according to claim 56, wherein N is equal to one of the following: 1, 2 or 3.
58. The method of claim 54, wherein the motion accuracy to be searched is predefined; wherein the motion accuracy is determined dynamically, or Wherein the motion accuracy is indicated in the bitstream.
59. The method of claim 51, wherein the motion candidates are refined prior to a first motion refinement process prior to the reordering process.
60. The method of claim 51, wherein the motion candidates are refined after a first motion refinement process prior to the reordering process.
61. The method of claim 51 , wherein the motion candidates are refined after the re-ranking process.
62. The method of claim 51, wherein the motion candidates are refined after a second motion refinement process after the reordering process.
63. The method of any one of claims 51 to 62, wherein the video unit is one of: Color component, Sub-picture, Strip, tiles, Codec Tree Unit (CTU), CTU line, CTU group, Codec Unit (CU), Prediction Unit (PU), Transformation Unit (TU), Codec Tree Block (CTB), Codec Block (CB), Prediction Block (PB), Transform Block (TB), piece, Sub-blocks of blocks, Sub-regions within a block, An area that contains more than one sample or pixel.
64. The method according to any one of claims 51 to 62, wherein an indication of whether and / or how to apply the refinement process and the reordering process to the motion candidates in the motion candidate list is indicated at one of: Sequence level, A set of picture levels, Picture level, Stripe level, Tile group level.
65. The method according to any one of claims 51 to 62, wherein the indication of whether and / or how to apply the refinement process and the reordering process to the motion candidates in the motion candidate list is indicated in one of the following: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependent Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), Strip head, Tile group header.
66. The method according to any one of claims 51 to 62, wherein an indication of whether and / or how to apply the refinement process and the reordering process to the motion candidates in the motion candidate list is comprised in one of the following: Prediction Block (PB), Transform Block (TB), Codec Block (CB), Prediction Unit (PU), Transformation Unit (TU), Codec Unit (CU), Virtual pipeline data unit (VPDU), Codec Tree Unit (CTU), CTU line, Strip, tiles, Sub-picture, An area that contains more than one sample or pixel.
67. The method of any one of claims 51 to 62, further comprising: determining whether and / or how to apply the refinement process and the reordering process to the motion candidates in the motion candidate list according to codec information of the video unit, the codec information comprising at least one of the following: Block size, Color format, Single-tree and / or dual-tree partitioning, Color component, Strip type, Image type.
68. The method of any one of claims 1 to 67, wherein the converting comprises encoding the video unit into the bitstream.
69. The method of any one of claims 1 to 67, wherein the converting comprises decoding the video unit from the bitstream.
70. An apparatus for video processing, comprising a processor and a non-volatile memory having instructions, wherein when the instructions are executed by the processor, the processor is caused to perform a method according to any one of claims 1 to 69.
71. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method of any one of claims 1 to 69.
72. A non-transitory computer-readable recording medium storing a bit stream of a video, wherein the bit stream of the video is generated by a method performed by an apparatus for video processing, wherein the method comprises: For conversion between a video unit of a video and a bitstream of the video unit, applying a processing flow to at least one motion candidate in a motion candidate list; After the processing flow, at least one subgroup of motion candidates in the motion candidate list is reordered by applying a first reordering process to the motion candidate list, wherein the processing flow includes at least one of: a modification process, a motion refinement process, or a second reordering process; as well as The bitstream is generated based on the reordered motion candidates.
73. A method for storing a bitstream of a video, comprising: applying a process to at least one motion candidate in a motion candidate list for a video unit of the video; After the processing flow, at least one subgroup of motion candidates in the motion candidate list is reordered by applying a first reordering process to the motion candidate list, wherein the processing flow includes at least one of: a modification process, a motion refinement process, or a second reordering process; generating the bitstream based on the reordered motion candidates; as well as The bit stream is stored in a non-transitory computer-readable recording medium.
74. A non-transitory computer-readable recording medium storing a bit stream of a video, wherein the bit stream of the video is generated by a method performed by an apparatus for video processing, wherein the method comprises: generating a cost in a refinement process of a motion candidate list for the video unit, wherein the refinement process comprises at least one of: template matching (TM), decoder-side motion vector refinement (DMVR), multi-pass DMVR; applying a re-ranking process to the motion candidate list using the costs; as well as The bitstream is generated based on the motion candidate list.
75. A method for storing a bitstream of a video, comprising: generating a cost in a refinement process of a motion candidate list for a video unit of the video, wherein the refinement process comprises at least one of: template matching (TM), decoder-side motion vector refinement (DMVR), multi-pass DMVR; applying a re-ranking process to the motion candidate list using the costs; generating the bitstream based on the motion candidate list; as well as The bit stream is stored in a non-transitory computer-readable recording medium.
76. A non-transitory computer-readable recording medium storing a bit stream of a video, wherein the bit stream of the video is generated by a method performed by an apparatus for video processing, wherein the method comprises: Applying a refinement process and a reordering process to motion candidates in a motion candidate list for a video unit of the video, wherein in the refinement process, the motion candidates are refined using a motion search method; as well as The bitstream is generated based on the refined and reordered motion candidates.
77. A method for storing a bitstream of a video, comprising: Applying a refinement process and a reordering process to motion candidates in a motion candidate list for a video unit of the video, wherein in the refinement process, the motion candidates are refined using a motion search method; generating the bitstream based on the refined and reordered motion candidates; as well as The bit stream is stored in a non-transitory computer-readable recording medium.
