Method and device for video processing and medium

Abstract

Embodiments of the present disclosure provide a solution for video processing. A method for video processing is presented. The method comprises: for a conversion between a current video block of a video and a code stream of the video, determining that a first prediction mode is applied to the current block; determining a block vector for the current video block based on the at least one motion candidate and the block vector difference information in the prediction mode; and performing a conversion based on the block vector.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to video coding techniques, and more particularly, to history-based motion vector prediction (HMVP) table and motion candidate list construction. Background Art

[0002] Nowadays, digital video capabilities are being applied to all aspects of people's lives. Various types of video compression technologies have been proposed for video encoding / decoding, such as MPEG-2, MPEG-4, ITU-TH.263, ITU-TH.264 / MPEG-4 Part 10 Advanced Video Codec (AVC), ITU-TH.265 High Efficiency Video Codec (HEVC) standard, and Versatile Video Codec (VVC) standard. However, the encoding and decoding efficiency of conventional video encoding and decoding technologies is usually very low, which is undesirable. Summary of the invention

[0003] Embodiments of the present disclosure provide a solution for video processing.

[0004] Embodiments of the present disclosure provide a solution for video processing.

[0005] In a first aspect, a method for video processing is proposed. The method comprises: for conversion between a current video block of a video and a bitstream of the video, determining that a first prediction mode is applied to the current block; in the prediction mode, determining a block vector for the current video block based on at least one motion candidate and block vector difference information; and performing conversion based on the block vector. The method according to the first aspect of the present disclosure can improve the efficiency of video encoding and decoding.

[0006] In a second aspect, a device for video processing is provided. The device includes a processor and a non-volatile memory having instructions thereon. The instructions, when executed by the processor, cause the processor to perform the method according to the first aspect of the present disclosure.

[0007] In a third aspect, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions for causing a processor to execute the method according to the first aspect of the present disclosure.

[0008] In a fourth aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a code stream of a video generated by a method performed by a video processing device. The method includes: for a conversion between a current video block of a video and a code stream of the video, determining that a first prediction mode is applied to the current block; in the prediction mode, determining a block vector for the current video block based on at least one motion candidate and block vector difference information; and generating a code stream based on motion information.

[0009] In a fifth aspect, a method for storing a bitstream of a video is provided. The method includes: for conversion between a current video block of the video and the bitstream of the video, determining that a first prediction mode is applied to the current block; in the prediction mode, determining a block vector for the current video block based on at least one motion candidate and block vector difference information; and storing the bitstream in a non-transitory computer-readable recording medium.

[0010] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become more apparent through the following detailed description with reference to the accompanying drawings. In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0012] Figure 1 A block diagram showing an example video encoding and decoding system according to some embodiments of the present disclosure is shown;

[0013] Figure 2 A block diagram of a first example video encoder according to some embodiments of the present disclosure is shown;

[0014] Figure 3 A block diagram of an example video decoder according to some embodiments of the present disclosure is shown;

[0015] Figure 4 An example diagram showing example locations of spatial merge candidates is shown;

[0016] Figure 5 shows an example diagram illustrating candidate pairs considered for redundancy checking of spatial merging candidates;

[0017] Figure 6 An example diagram illustrating example motion vector scaling for temporal merging candidates is shown;

[0018] Figure 7 An example diagram showing candidate positions for time domain merging candidates C0 and C1 is shown;

[0019] Figure 8 An example diagram showing VVC spatial neighboring blocks of a current block is shown;

[0020] Fig. 9 An example virtual block in the i-th round of search is shown;

[0021] Fig.10 An example diagram showing spatial neighboring blocks used to derive spatial merge candidates is shown;

[0022] Fig.11A and Fig. 11B The SbTMVP process in VVC is shown;

[0023] Figures 12A-12D The current CTU processing order and the available samples in the current CTU and the left CTU are shown;

[0024] Fig.13 shows the neighboring samples used to calculate the SAD;

[0025] Fig.14 shows neighboring samples used to calculate SAD for sub-CU level motion information;

[0026] Fig.15 An example diagram showing the sorting process is shown;

[0027] Fig.16 An example diagram illustrating a reordering process in an encoder is shown;

[0028] Fig.17 An example diagram illustrating a reordering process in a decoder is shown;

[0029] Fig.18 shows an example diagram illustrating the execution of template matching on a search area around an initial MV;

[0030] Fig.19 An example graph demonstrating template matching prediction is shown;

[0031] Fig. 20 An example diagram showing the intra-frame template matching search area used is shown;

[0032] Fig.21 shows an example diagram showing a template used in TIMD and its reference sample;

[0033] Fig. 22 An example diagram showing a display template and a reference sample of the template is shown;

[0034] Fig.23 An example diagram showing templates in reference list 0 and reference list 1 and reference samples of the templates is shown;

[0035] Fig.24 An example diagram showing a template for a block with sub-block motion using motion information of a sub-block of a current block and a reference sample of the template is shown;

[0036] Fig.25 An example diagram showing a template for a block with sub-block motion using motion information of each sub-template and a reference sample of the template is shown;

[0037] Fig.26 An example diagram showing a template for a block with OBMC and a reference sample of the template is shown;

[0038] Fig. 27 An example diagram showing motion estimation of a rectangular block with hash values ​​of square sub-blocks is shown;

[0039] Fig.28 An example luminance mapping with a chroma scaling architecture is shown;

[0040] Fig.29 The MMVD search point is shown;

[0041] Fig. 30A shows inter-frame prediction based on triangulation, where triangleDir is equal to 0;

[0042] Fig. 30B shows inter-frame prediction based on triangulation, where triangleDir is equal to 1;

[0043] Fig.31 Unidirectional prediction MV selection for triangulation mode is shown;

[0044] Fig.32 The weights used in the blending process are shown;

[0045] Fig.33A Three MV storage areas are shown for triangleDir equal to 0 and 32x16 blocks;

[0046] Fig.33B Three MV storage areas are shown for triangleDir equal to 0 and 16x32 blocks;

[0047] Fig.33C Three MV storage areas are shown for triangleDir equal to 0 and 32x32 blocks;

[0048] Fig.34 An example of GPM segmentation grouped at the same angle is shown;

[0049] Fig.35 Unidirectional prediction MV selection for geometric partitioning mode is shown;

[0050] Fig.36 An exemplary generation of warp weights w0 using a geometric partitioning pattern is shown;

[0051] Fig.37 The top and left neighboring blocks used in the derivation of CIIP weights are shown;

[0052] Fig.38AAn example diagram showing candidate locations for spatial domain candidates is shown;

[0053] Fig.38B An example diagram showing candidate positions for time domain candidates is shown;

[0054] Fig.39 An example diagram showing the derivation of a sub-CUbv motion field from a corresponding co-located sub-CU by applying a motion shift from a spatial neighbor is shown;

[0055] Fig.40 An example diagram illustrating example intra-frame template matching is shown;

[0056] Fig.41A An example diagram showing a reference template outside the current image is shown;

[0057] Fig.41B An example diagram showing cropping of the BV so that the reference template is located inside the current image is shown;

[0058] Fig.42A An example implementation of adding diagonal angles is shown;

[0059] Fig.42B Another example implementation of adding diagonal angles is shown;

[0060] Fig.42C Another example implementation of adding diagonal angles is shown;

[0061] Fig.43A An example implementation of adding diagonal angles with exact similar distances around a circle is shown;

[0062] Fig.43B Another example implementation of adding diagonal angles with exact similar distances around a circle is shown;

[0063] Fig.44 Some example implementations of adding any combination of steps and angles asymmetrically are shown;

[0064] Fig.45 Some example implementations of removing every other distance offset are shown;

[0065] Fig.46 A schematic diagram of the dependent direction offset based on the proposed MV is shown;

[0066] Fig.47 A template for a block with sub-block motion and a reference sample of the template are shown;

[0067] Fig.48 The template for the entire block and a reference sample of the template are shown;

[0068] Fig.49 The adjacent spatial neighbor blocks used are shown;

[0069] Fig.50 The top and left neighboring blocks used in the derivation of CIIP_N1 and CIIP_N2 weights are shown;

[0070] Fig.51 The triangulation-based IBC prediction is shown;

[0071] Fig.52A A diamond search pattern is shown;

[0072] Fig.52B A cross search pattern is shown;

[0073] Fig.53 shows the IBC reference area depending on the current CU position;

[0074] Fig.54 Candidate BVP clipping at the IBC buffer boundary is shown;

[0075] Fig.55 The candidate BVPs selected to complete the IBC merge / AMVP list are shown;

[0076] Fig.56 The adjacent spatial neighbor blocks used are shown;

[0077] Fig.57 A cross search pattern is shown;

[0078] Fig.58 A flowchart showing a method for video processing according to some embodiments of the present disclosure; and

[0079] Fig.59 A block diagram of a computing device is shown in which various embodiments of the present disclosure may be implemented.

[0080] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. DETAILED DESCRIPTION

[0081] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described only for the purpose of illustrating and helping those skilled in the art to understand and implement the present disclosure, without implying any limitation on the scope of the present disclosure. In addition to the methods described below, the disclosure described herein can also be implemented in various ways.

[0082] In the following description and claims, unless defined otherwise, all scientific and technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs.

[0083] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment must include the particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example embodiment, whether or not explicitly described, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in relation to other embodiments.

[0084] It should be understood that although the terms "first" and "second" etc. can be used to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish one element from another element. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0085] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms "including", "comprising", and / or "having" when used herein indicate the presence of the features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof. Example Environment

[0086] Figure 1 1 is a block diagram illustrating an example video codec system 100 that may utilize the techniques of the present disclosure. As shown, the video codec system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a video encoding device, and the destination device 120 may also be referred to as a video decoding device. In operation, the source device 110 may be configured to generate encoded video data, and the destination device 120 may be configured to decode the encoded video data generated by the source device 110. The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0087] Video source 112 may include a source such as a video capture device. Examples of a video capture device include, but are not limited to, an interface that receives video data from a video content provider, a computer graphics system for generating video data, and / or combinations thereof.

[0088] The video data may include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a code stream. The code stream may include a bit sequence that forms a coded representation of the video data. The code stream may include coded pictures and associated data. The coded picture is a coded representation of the picture. The associated data may include a sequence parameter set, a picture parameter set, and other grammatical structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The coded video data may be directly transmitted to the destination device 120 via the network 130A via the I / O interface 116. The coded video data may also be stored on a storage medium / server 130B for access by the destination device 120.

[0089] The destination device 120 may include an I / O interface 126, a video decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain encoded video data from the source device 110 or the storage medium / server 130B. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to the user. The display device 122 may be integrated with the destination device 120, or may be outside the destination device 120, which is configured to be connected to an external display device interface.

[0090] The video encoder 114 and the video decoder 124 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other existing and / or further standards.

[0091] Figure 2 is a block diagram showing an example of a video encoder 200 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video encoder 114 in the system 100 is shown.

[0092] Video encoder 200 may be configured to implement any or all of the techniques of this disclosure. Figure 2 In the example of , video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video encoder 200. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0093] In some embodiments, the video encoder 200 may include a partitioning unit 201, a prediction unit 202, a residual generation unit 207, a transformation unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transformation unit 211, a reconstruction unit 212, a buffer 213 and an entropy coding and decoding unit 214, and the prediction unit 202 may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205 and an intra-frame prediction unit 206.

[0094] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, where at least one reference picture is a picture where the current video block is located.

[0095] Furthermore, although some components (such as the motion estimation unit 204 and the motion compensation unit 205) may be integrated, for the purpose of explanation, these components are described in detail below. Figure 2 are shown separately in the example.

[0096] The partitioning unit 201 may partition a picture into one or more video blocks. The video encoder 200 and the video decoder 300 may support various video block sizes.

[0097] The mode selection unit 203 may select one of a plurality of codec modes (intra-frame coding or inter-frame coding), for example, based on the error result, and provide the generated intra-frame codec block or inter-frame codec block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the codec block for use as a reference picture. In some examples, the mode selection unit 203 may select a combination of intra-frame and inter-frame prediction (CIIP) modes, where the prediction is based on an inter-frame prediction signal and an intra-frame prediction signal. In the case of inter-frame prediction, the mode selection unit 203 may also select a resolution for the motion vector (e.g., sub-pixel precision or integer pixel precision) for the block.

[0098] To perform inter-frame prediction on the current video block, the motion estimation unit 204 may generate motion information for the current video block by comparing the current video block with one or more reference frames from the buffer 213. The motion compensation unit 205 may determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from the buffer 213 other than the picture associated with the current video block.

[0099] The motion estimation unit 204 and the motion compensation unit 205 may perform different operations on the current video block, for example, depending on whether the current video block is in an I slice, a P slice, or a B slice. As used herein, an "I slice" may refer to a portion of a picture consisting of macroblocks, all of which are based on macroblocks within the same picture. Furthermore, as used herein, in some aspects, a "P slice" and a "B slice" may refer to a portion of a picture consisting of macroblocks that are independent of macroblocks in the same picture.

[0100] In some examples, the motion estimation unit 204 may perform unidirectional prediction on the current video block, and the motion estimation unit 204 may search the reference pictures of list 0 or list 1 to find the reference video block for the current video block. The motion estimation unit 204 may then generate a reference index and a motion vector, the reference index indicating the reference picture in list 0 or list 1 containing the reference video block, and the motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 204 may output the reference index, the prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0101] Alternatively, in other examples, the motion estimation unit 204 may perform bidirectional prediction on the current video block. The motion estimation unit 204 may search the reference pictures in list 0 for a reference video block for the current video block, and may also search the reference pictures in list 1 for another reference video block for the current video block. The motion estimation unit 204 may then generate a plurality of reference indexes and a plurality of motion vectors, the plurality of reference indexes indicating a plurality of reference pictures in list 0 and list 1 containing a plurality of reference video blocks, and the plurality of motion vectors indicating a plurality of spatial displacements between the plurality of reference video blocks and the current video block. The motion estimation unit 204 may output the plurality of reference indexes and the plurality of motion vectors of the current video block as motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the plurality of reference video blocks indicated by the motion information of the current video block.

[0102] In some examples, motion estimation unit 204 may output a complete set of motion information for use in a decoding process by a decoder. Alternatively, in some embodiments, motion estimation unit 204 may signal motion information of a current video block with reference to motion information of another video block. For example, motion estimation unit 204 may determine that motion information of a current video block is sufficiently similar to motion information of a neighboring video block.

[0103] In one example, motion estimation unit 204 may indicate to video decoder 300 a value in a syntax structure associated with the current video block that indicates that the current video block has the same motion information as another video block.

[0104] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0105] As discussed above, the video encoder 200 may signal motion vectors in a predictive manner.Two examples of prediction signaling techniques that may be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and merge mode signaling.

[0106] The intra prediction unit 206 may perform intra prediction on the current video block. When the intra prediction unit 206 performs intra prediction on the current video block, the intra prediction unit 206 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a prediction video block and various syntax elements.

[0107] The residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) the prediction video block(s) of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample portions of samples in the current video block.

[0108] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 207 may not perform a subtraction operation.

[0109] Transform processing unit 208 may generate one or more transform coefficient video blocks for a current video block by applying one or more transforms to the residual video block associated with the current video block.

[0110] After transform processing unit 208 generates a transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0111] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current video block for storage in the buffer 213.

[0112] After reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[0113] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the entropy coding unit 214 receives data, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy coded data and output a code stream including the entropy coded data.

[0114] Figure 3 is a block diagram showing an example of a video decoder 300 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video decoder 124 in the system 100 is shown.

[0115] Video decoder 300 may be configured to perform any or all of the techniques of this disclosure. Figure 3 In the example of , video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video decoder 300. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0116] exist Figure 3 In the example of , the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, and a reconstruction unit 306 and a buffer 307. In some examples, the video decoder 300 can perform a decoding process that is generally opposite to the encoding process described with respect to the video encoder 200.

[0117] The entropy decoding unit 301 can retrieve the encoded code stream. The encoded code stream may include entropy encoded video data (e.g., encoded video data blocks). The entropy decoding unit 301 can decode the entropy encoded video data, and the motion compensation unit 302 can determine motion information from the entropy decoded video data, the motion information including motion vectors, motion vector precision, reference picture list indexes, and other motion information. The motion compensation unit 302 can determine the information, for example, by performing AMVP and merge mode. AMVP is used, including deriving several most likely candidates based on data and reference pictures of adjacent PBs. The motion information typically includes horizontal and vertical motion vector displacement values, one or two reference picture indexes, and in the case of prediction areas in B strips, also includes an identification of which reference picture list is associated with each index. As used herein, in some aspects, "merge mode" may refer to deriving motion information from blocks that are adjacent in space or time.

[0118] The motion compensation unit 302 may generate motion compensated blocks, possibly performing interpolation based on interpolation filters.Identifiers for the interpolation filters used with sub-pixel precision may be included in the syntax elements.

[0119] The motion compensation unit 302 may calculate interpolated values ​​for sub-integer pixels of a reference block using interpolation filters used by the video encoder 200 during encoding of the video block. The motion compensation unit 302 may determine the interpolation filters used by the video encoder 200 based on received syntax information, and the motion compensation unit 302 may use the interpolation filters to generate a prediction block.

[0120] The motion compensation unit 302 may use at least part of the syntax information to determine the size of blocks used to encode (multiple) frames and / or (multiple) slices of the encoded video sequence, partition information describing how each macroblock of a picture of the encoded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-frame coding block, and other information for decoding the encoded video sequence. As used herein, in some aspects, a "slice" may refer to a data structure that can be decoded independently of other slices of the same picture in terms of entropy coding, signal prediction, and residual signal reconstruction. A slice may be an entire picture, or it may be a region of a picture.

[0121] The intra prediction unit 303 may use, for example, an intra prediction mode received in the bitstream to form a prediction block from spatially adjacent blocks. The inverse quantization unit 304 inversely quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 305 applies an inverse transform.

[0122] The reconstruction unit 306 may obtain the decoded block, for example, by adding the residual block to the corresponding prediction block generated by the motion compensation unit 302 or the intra prediction unit 303. If necessary, a deblocking filter may also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in a buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction, and the buffer 307 also generates the decoded video for presentation on a display device.

[0123] Some example embodiments of the present disclosure will be described in detail below. It should be noted that the section titles used in this document are for ease of understanding, and the embodiments disclosed in the section are not limited to that section. In addition, although some embodiments are described with reference to general video codecs or other specific video codecs, the disclosed technology is also applicable to other video coding and decoding technologies. In addition, although some embodiments describe the video encoding steps in detail, it should be understood that the corresponding decoding steps of canceling the encoding will be implemented by the decoder. In addition, the term video processing includes video coding or compression, video decoding or decompression, and video transcoding, in which video pixels are represented from one compression format to another compression format or at different compression bit rates. 1. Brief Overview The present disclosure relates to video coding technology. Specifically, it is about IBC prediction and related technologies in video coding. It can be applied to existing video coding standards such as HEVC, VVC, etc. It can also be applied to future video coding standards or video codecs. 2. Introduction Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, ISO / IEC developed MPEG-1 and MPEG-4 Visual, and the two organizations jointly developed H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Codec (AVC) and H.265 / HEVC standards. Since H.262, video codec standards have been based on a hybrid video codec structure, in which temporal prediction plus transform codec is used. In order to explore future video codec technologies beyond HEVC, VCEG and MPEG jointly founded the Joint Video Exploration Team (JVET) in 2015. JVET meetings are held simultaneously every quarter, and the new video codec standard was officially named Versatile Video Codec (VVC) at the JVET meeting in April 2018, and the first version of the VVC Test Model (VTM) was released at that time. The VVC working draft and the test model VTM are updated after each meeting. The VVC project achieved technical completion (FDIS) at the July 2020 meeting. 2.1. Extended Merge Prediction In VVC, the merge candidate list is constructed by including the following five types of candidates in order: 1) Spatial MVP from the spatial neighbor CU, 2) Time domain MVP from the same CU, 3) History-based MVP from FIFO table, 4) Pairwise average MVP, 5) Zero MV. The size of the merge list is signaled in the sequence parameter set header, and the maximum allowed size of the merge list is 6. For each CU code in merge mode, the index of the best merge candidate is encoded using truncated unary binarization (TU). The first bin of the merge index uses context encoding and decoding, while bypass encoding and decoding is used for the other bins. The derivation process of merge candidates for each category is provided in this section. As in HEVC, VVC also supports parallel derivation of merge candidate lists for all CUs within a certain size region. 2.1.1. Spatial Candidate Derivation The derivation of spatial merge candidates in VVC is the same as that in HEVC, except that the positions of the first two merge candidates are swapped. Figure 4 An example diagram 400 showing example locations of spatial merging candidates is shown. Figure 4 Among the candidates at the positions shown, up to four merge candidates are selected. The derivation order is B1, A1, B0, A0, and B2. Position B2 is considered only when one or more CUs at positions B0, A0, B1, and A1 are not available (for example, because it belongs to another slice or tile) or is intra-coded. After the candidate at position A1 is added, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving coding efficiency. Figure 5 An example diagram 500 showing candidate pairs considered for redundancy check of spatial merging candidates is shown. In order to reduce computational complexity, not all possible candidate pairs are considered in the redundancy check mentioned. Instead, only Figure 5 The pairs are linked by arrows in , and candidates are added to the list only if the corresponding candidates used for redundancy checking do not have the same motion information. 2.2.2. Time Domain Candidate Derivation In this step, only one candidate is added to the list. In particular, in the derivation of this temporal merge candidate, the scaled motion vector is derived based on the co-located CU belonging to the co-located reference picture. The reference picture list to be used for deriving the co-located CU is explicitly signaled in the slice header. Figure 6 An example diagram 600 showing example motion vector scaling for a temporal merge candidate is shown. Figure 6 As shown by the dotted line in , the scaled motion vector for the temporal merge candidate is obtained, which is scaled from the motion vector of the co-located CU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero. Figure 7 An example diagram 700 showing candidate positions for time domain merging candidates C0 and C1 is shown. Figure 7 As shown, the position of the temporal candidate is selected between candidates C0 and C1. If the CU at position C0 is not available, intra-coded, or outside the current row of the CTU, position C1 is used. Otherwise, position C0 is used in the derivation of the temporal merge candidate. 2.1.3. History-based Merge Candidate Derivation History-based MVP (HMVP) merge candidates are added to the merge list after spatial MVP and TMVP. In this method, the motion information of the previous codec block is stored in a table and used as the MVP of the current CU. A table with multiple HMVP candidates is maintained during the encoding / decoding process. The table is reset (cleared) when a new CTU row is encountered. Whenever there is a non-sub-block inter-coded CU, the associated motion information is added as a new HMVP candidate to the last entry of the table. The HMVP list size S is set to 6, which indicates that up to 6 history-based MVP (HMVP) candidates can be added to the table. When a new motion candidate is inserted into the table, a constrained first-in, first-out (FIFO) rule is used, where a redundancy check is first applied to find if there is an identical HMVP in the table. If found, the identical HMVP is removed from the table, and then all subsequent HMVP candidates are moved forward. HMVP candidates can be used in the merge candidate list construction process. Check the nearest HMVP candidates in the table in order and insert them into the candidate list after the TMVP candidate. Apply redundancy check to spatial merge candidates or temporal merge candidates for HMVP candidates. To reduce the number of redundant checking operations, the following simplifications are introduced: 1. The number of HMPV candidates for merge list generation is set to (N<=4)?M:(8-N), where N indicates the number of existing candidates in the merge list and M indicates the number of available HMVP candidates in the table. 2. Once the total number of available merge candidates reaches the maximum allowed merge candidate minus 1, the merge candidate list construction process from HMVP is terminated. 2.1.4. Pairwise Average Merging Candidate Derivation The pairwise average candidate is generated by averaging the predefined candidate pairs in the existing merge candidate list, and the predefined pairs are defined as {(0, 1), (0, 2), (1, 2), (0, 3), (1, 3), (2, 3)}, where the number represents the merge index of the merge candidate list. The averaged motion vector is calculated separately for each reference list. If two motion vectors are available in one list, they are averaged even if they point to different reference pictures; if only one motion vector is available, it is used directly; if no motion vector is available, this list is kept invalid. When the merge list is not full after adding pairwise average merge candidates, zero MVPs will be inserted at the end until the maximum number of merge candidates is encountered. 2.2. New Merge Candidates 2.2.1 Non-adjacent merge candidate export Figure 8 An example diagram 800 showing VVC spatial neighboring blocks of a current block is shown. In VVC, Figure 8 The five spatial neighboring blocks and one temporal neighboring block shown are used to derive merge candidates. It is proposed to derive additional merge candidates from positions non-adjacent to the current block using the same style as in VVC. To achieve this, for each search round i, a virtual block is generated based on the current block as follows: First, the relative position of the virtual block to the current block is calculated using the following formula: Offsetx=-i×gridX, Offsety=-i×gridY Where Offsetx and Offsety represent the offset of the upper left corner of the virtual block relative to the upper left corner of the current block, and gridX and gridY are the width and height of the search grid. Second, the width and height of the virtual block are calculated using the following formula: newWidth = i×2×gridX+currWidth newHeight = i×2×gridY+currHeight. currWidth and currHeight are the width and height of the current block. newWidth and newHeight are the width and height of the new virtual block. gridX and gridY are currently set to currWidth and currHeight respectively. Fig. 9 An example diagram 900 showing virtual blocks in an i-th round of searching is shown. Fig. 9 The relationship between the virtual block and the current block is shown. After the virtual block is generated, block A i , B i , C i , D i and E i can be considered as the VVC spatial neighbors of the virtual block, and their positions are obtained using the same pattern as in VVC. Obviously, if the search round i is 0, the virtual block is the current block. In this case, block A i , B i , C i , D i and E i It is the spatial neighboring block used in VVC merge mode. When constructing the merge candidate list, pruning is performed to ensure that each element in the merge candidate list is unique. The maximum search round is set to 1, which means that five non-adjacent spatial neighbor blocks are used. Non-adjacent spatial merging candidates are inserted into the merging list after the temporal merging candidates in the order of B1->A1->C1->D1->E1. 2.2.2 Non-adjacent airspace candidates Non-adjacent spatial merging candidates are inserted after TMVP in the regular merging candidate list. Fig.10 An example diagram 1000 showing spatial neighboring blocks used to derive spatial merge candidates is shown. The pattern of spatial merge candidates is as follows: Fig.10 As shown in Figure 2. The distance between non-adjacent spatial candidates and the current codec block is based on the width and height of the current codec block. The line buffer limit is not applied. 2.2.3STMVP It is proposed to use three spatial merging candidates and one temporal merging candidate to derive an average candidate as the STMVP candidate. The STMVP is inserted before the upper left spatial merging candidate. The STMVP candidate is pruned along with all previous merge candidates in the merge list. For spatial candidates, the first three candidates in the current merged candidate list are used. For the temporal candidates, the same positions as the VTM / HEVC co-location positions are used. For spatial candidates, the first, second, and third candidates inserted before STMVP in the current merge candidate list are denoted as F, S, and T. The temporal candidate having the same position as the VTM / HEVC co-location position used in TMVP is denoted as Col. The motion vector of the STMVP candidate in prediction direction X (denoted as mvLX) is derived as follows: 1) If the reference indices of the four merge candidates are all valid and equal to zero in the prediction direction X (X=0 or 1), mvLX=(mvLX_F+mvLX_S+mvLX_T+mvLX_Col)>>2 2) If the reference indexes of three of the four merge candidates are valid and in the prediction direction X(X =0 or 1) is equal to zero, mvLX=(mvLX_F×3+mvLX_S×3+mvLX_Col×2)>>3 or mvLX=(mvLX_F×3+mvLX_T×3+mvLX_Col×2)>>3 or mvLX=(mvLX_S×3+mvLX_T×3+mvLX_Col×2)>>3. 3) If the reference indexes of two of the four merge candidates are valid and in the prediction direction X (X =0 or 1) is equal to zero, mvLX=(mvLX_F+mvLX_Col)>>1 or mvLX=(mvLX_S+mvLX_Col)>>1 or mvLX=(mvLX_T+mvLX_Col)>>1. NOTE: If time domain candidates are not available, STMVP mode is turned off. 2.2.4 Merge List Size If both non-adjacent merge candidates and STMVP merge candidates are considered, the size of the merge list is signaled in the sequence parameter set header and the maximum allowed size of the merge list is increased (eg, 8). 2.3. Sub-block based temporal motion vector prediction (SbTMVP) VVC supports the sub-block-based temporal motion vector prediction (SbTMVP) method. Similar to the temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the co-located picture to improve the motion vector prediction and merge mode of the CU in the current picture. The same co-located picture used by TMVP is used for SbTMVP. SbTMVP differs from TMVP in the following two main aspects: –TMVP predicts motion at CU level, but SbTMVP predicts motion at sub-CU level; – While TMVP pre-fetches the temporal motion vector from the co-located block in the co-located picture (the co-located block is the bottom-right or center block relative to the current CU), SbTMVP applies motion shifting before pre-fetching the temporal motion information from the co-located picture, where the motion shifting is obtained from the motion vector of one of the spatial neighboring blocks from the current CU. SbTMVP process Fig.11A and Fig. 11B As shown in Figure 2, SbTMVP predicts the motion vector of the sub-CU within the current CU in two steps. Fig.11A shows the spatial neighboring blocks used by SbTMVP. In the first step, check Fig.11A The spatial neighbor A1 in . If A1 has a motion vector that uses the co-located picture as its reference picture, then that motion vector is selected as the motion offset to be applied. If no such motion is identified, then the motion shift is set to (0, 0). Fig. 11B An example diagram showing driving the sub-CU motion field by applying motion shifts from spatial neighbors and scaling the motion information from the corresponding co-located sub-CU. In the second step, the motion shift identified in step 1 is applied (i.e., added to the coordinates of the current block) to Fig. 11B The co-located picture shown obtains motion information (motion vector and reference index) at the sub-CU level. Fig. 11B The example in assumes that the motion shift is set to the motion of block A1. Then, for each sub-CU, the motion information of its corresponding block in the co-located picture (the minimum motion grid covering the center sample) is used to derive the motion information for the sub-CU. After the motion information of the co-located sub-CU is identified, it is converted into a reference index and motion vector of the current sub-CU in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference picture of the temporal motion vector with the reference picture of the current CU. In VVC, a combined sub-block based merge list containing both SbTMVP candidates and affine merge candidates is used for signaling of sub-block based merge mode. The SbTMVP mode is enabled / disabled by a sequence parameter set (SPS) flag. If the SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry in the list of sub-block based merge candidates, followed by the affine merge candidates. The size of the sub-block based merge list is signaled in the SPS, and the maximum allowed size of the sub-block based merge list in VVC is 5. The sub-CU size used in SbTMVP is fixed to 8x8, and like the affine merge mode, the SbTMVP mode is only applicable to CUs whose width and height are both greater than or equal to 8. The encoding logic of the additional SbTMVP merge candidate is the same as that of other merge candidates, ie, for each CU in a P or B slice, an additional RD check is performed to decide whether to use the SbTMVP candidate. 2.4. Intra-block copy (IBC) Intra-block copying (IBC) is a tool adopted in the HEVC extension on SCC. As we all know, it significantly improves the coding efficiency of screen content materials. Since the IBC mode is implemented as a block-level coding mode, block matching (BM) is performed at the encoder to find the best block vector (or motion vector) for each CU. Here, the block vector is used to indicate the displacement from the current block to the reference block that has been reconstructed inside the current picture. The luminance block vector of the CU encoded and decoded by IBC is in integer precision. The chrominance block vector is also rounded to integer precision. When combined with AMVR, the IBC mode can switch between 1-pixel motion vector precision and 4-pixel motion vector precision. The CU encoded and decoded by IBC is regarded as a third prediction mode different from the intra or inter prediction mode. The IBC mode is applicable to CUs with width and height less than or equal to 64 luminance samples. On the encoder side, hash-based motion estimation is performed for IBC. The encoder performs RD check on blocks with width or height no greater than 16 luma samples. For non-merge mode, block vector search is first performed using hash-based search. If the hash search does not return a valid candidate, a local search based on block matching will be performed. In hash-based search, the hash key matching (32-bit CRC) between the current block and the reference blocks is extended to all allowed block sizes. The hash key calculation for each position in the current picture is based on 4x4 sub-blocks. For larger-sized current blocks, the hash key is determined to match the hash key of the reference block when all hash keys of all 4×4 sub-blocks match the hash keys in the corresponding reference positions. If the hash keys of multiple reference blocks are found to match the hash key of the current block, the block vector cost of each matching reference is calculated, and the one with the smallest cost is selected. In the block matching search, the search range is set to cover both the previous CTU and the current CTU. At CU level, the IBC mode is signaled using a flag, and it can be signaled in IBC AMVP mode or IBC Skip / Merge mode as follows: - IBC skip / merge mode: The merge candidate index is used to indicate which block vectors from a list of neighboring candidate IBC-coded blocks are used to predict the current block. The merge list consists of spatial, HMVP and pairwise candidates. – IBC AMVP mode: Block vector differences are encoded in the same way as motion vector differences. The block vector prediction method uses two candidates as predictors, one from the left neighbor and one from the top neighbor (if encoded by IBC). When either neighbor is not available, the default block vector will be used as the predictor. A flag is signaled to indicate the block vector predictor index. 2.4.1 Simplification of IBC vector prediction The BV predictors in IBC for merge mode and AMVP mode will share a common predictor list consisting of the following elements: ·2 adjacent airspace locations (in Figure 4 A0, B0 in 5 HMVP entries, The default is zero vector. For merge mode, up to the first 6 entries of the list will be used; for AMVP mode, the first 2 entries of the list will be used. And the list meets the shared merge list area requirements (sharing the same list within SMR). 2.4.2 IBC Reference Area To reduce memory consumption and decoder complexity, IBC in VVC only allows the reconstruction of a predefined area, which includes the area of ​​the current CTU and some areas of the left CTU. Figures 12A-12D Example diagrams 1210 , 1230 , 1250 , and 1270 are shown illustrating the current CTU processing order and the available samples in the current CTU and the left CTU. Figures 12A-12D The reference area of ​​the IBC mode is shown, where each block represents a 64x64 luma sample unit. Depending on the location of the current codec CU location within the current CTU, the following applies: – If the current block falls into the upper left 64x64 block of the current CTU, in addition to the samples already reconstructed in the current CTU, the CPR mode can also be used to refer to the reference samples in the lower right 64x64 block of the left CTU. The current block can also use the CPR mode to refer to the reference samples in the lower left 64x64 block of the left CTU and the reference samples in the upper right 64x64 block of the left CTU. – If the current block falls into the upper right 64x64 block of the current CTU, in addition to the samples already reconstructed in the current CTU, if the luma position (0,64) relative to the current CTU has not been reconstructed, the current block can also use the CPR mode to refer to the reference samples in the lower left 64x64 block and the lower right 64x64 block of the left CTU; otherwise, the current block can also refer to the reference samples in the lower right 64x64 block of the left CTU. – If the current block falls into the lower left 64x64 block of the current CTU, in addition to the samples that have been reconstructed in the current CTU, if the luma position (64,0) relative to the current CTU has not been reconstructed, the current block can also use the CPR mode to refer to the reference samples in the upper right 64x64 block and the lower right 64x64 block of the left CTU. Otherwise, the current block can also use the CPR mode to refer to the reference samples in the lower right 64x64 block of the left CTU. – If the current block falls into the lower right 64x64 block of the current CTU, it can only refer to the already reconstructed samples in the current CTU using the CPR mode. This restriction allows the IBC mode to be implemented using local on-chip memory for hardware implementation. 2.4.3 Interaction between IBC and other codecs The interaction between IBC mode and other inter-frame codec tools in VVC, such as paired merge candidates, history-based motion vector predictor (HMVP), combined intra / inter prediction mode (CIIP), merge mode with motion vector difference (MMVD), and geometric partitioning mode (GPM), is as follows: – IBC can be used with pairwise merge candidates and HMVP. A new pairwise IBC merge candidate can be generated by averaging two IBC merge candidates. For HMVP, IBC motion is inserted into the history buffer for future reference. – IBC cannot be used in conjunction with the following interframe tools: Affine Motion, CIIP, MMVD, and GPM. – When DUAL_TREE partitioning is used, IBC is not allowed for chroma codec blocks. Unlike the HEVC screen content codec extension, the current picture is no longer included as one of the reference pictures in reference picture list 0 for IBC prediction. The derivation process of motion vectors for IBC mode excludes all neighboring blocks in inter mode and vice versa. The following IBC design aspects are applied: – IBC shares the same process as in regular MV merging, including the use of pairwise merging candidates and history-based motion predictors, but TMVP and zero vectors are not allowed since they are invalid for IBC mode. – Separate HMVP buffers (5 candidates each) are used for regular MV and IBC. – Block vector constraints are implemented as codestream consistency constraints, the encoder needs to ensure that there are no invalid vectors in the codestream, and if the merge candidate is invalid (out of range or 0), the merge is not used. Such codestream consistency constraints are represented by virtual buffers as described below. – For deblocking, IBC is handled as inter mode. - If the current block is encoded using IBC prediction mode, AMVR does not use quarter-pels; instead, AMVR is signaled to only indicate whether the MV is inter-pel or 4integer-pel. - The number of IBC merge candidates may be signaled in the slice header separately from the number of normal, sub-block and geometric merge candidates. The virtual buffer concept is used to describe the allowable reference area and valid block vectors for the IBC prediction mode. Denoting the CTU size as ctbSize, the virtual buffer ibcBuf has a width wIbcBuf=128x128 / ctbSize and a height hIbcBuf=ctbSize. For example, for a CTU size of 128x128, the size of ibcBuf is also 128x128; for a CTU size of 64x64, the size of ibcBuf is 256x64; and for a CTU size of 32x32, the size of ibcBuf is 512x32. The size of the VPDU is min(ctbSize, 64) in each dimension. v =min(ctbSize, 64). The virtual IBC buffer ibcBuf is maintained as follows. – At the beginning of decoding each CTU line, flush the entire ibcBuf with an invalid value of -1. – Start decoding the VPDU (xVPDU, yVPDU) relative to the upper left corner of the picture, and set ibcBuf[x][y] = -1, where x = xVPDU% wIbcBuf, ..., xVPDU% wIbcBuf+W v -1;y=yVPDU%ctbSize,…,yVPDU%ctbSize+W v -1. – After decoding the CU containing (x, y) relative to the top-left corner of the picture, set –ibcBuf[x%wIbcBuf][y%ctbSize]=recSample[x][y]. For a block covering coordinates (x, y), if the following is true for the block vector bv = (bv[0], bv[1]), then it is valid; otherwise, it is invalid: ibcBuf[(x+bv[0])%wIbcBuf][(y+bv[1])%ctbSize] should not be equal to -1. 2.4.4IBC Virtual Buffer Test The luma block vector bvL (a luma block vector in 1 / 16 fractional sample precision) shall obey the following constraints: –CtbSizeY is greater than or equal to ((yCb+(bvL[1]>>4))&(CtbSizeY-1))+cbHeight. – For x = xCb ... xCb + cbWidth - 1 and y = yCb ... yCb + cbHeight - 1, IbcVirBuf[0][(x + (bvL[0] >> 4)) & (IbcBufWidthY - 1)][(y + (bvL[1] >> 4)) & (CtbSizeY - 1)] shall not be equal to -1. Otherwise, bvL is considered to be an invalid bv. Samples are processed in CTB units. For each luma CTB, the array size is CtbSizeY in samples in both width and height. – (xCb, yCb) is the luminance position of the top left sample of the current luminance codec block relative to the top left luminance sample of the current picture, –cbWidth specifies the width of the current codec block in luma samples. –cbHeight specifies the height of the current codec block in luma samples. 2.5. Adaptive Merging Candidate Reordering Based on Template Matching In order to improve the encoding and decoding efficiency, after constructing the merge candidate list, the order of each merge candidate is adjusted according to the template matching cost. The merge candidates are arranged in the list according to the template matching cost in ascending order. It is operated in the form of subgroups. Fig.13 An example graph 1300 showing neighboring samples used to calculate SAD is shown. Fig.14 An example diagram 1400 showing neighboring samples used to calculate SAD for sub-CU level motion information is shown. The template matching cost is measured by the SAD (sum of absolute differences) between the neighboring samples of the current CU and its corresponding reference samples. If the merge candidate includes bi-directionally predicted motion information, then Fig.13 As shown, the corresponding reference sample is the average of the corresponding reference sample in reference list 0 and the corresponding reference sample in reference list 1. If the merge candidate contains motion information at the sub-CU level, then Fig.14 As shown, the corresponding reference sample is composed of neighboring samples of the corresponding reference sub-block. Fig.15 An example diagram 1500 illustrating the sorting process is shown. Fig.15 As shown, the sorting process is operated in the form of subgroups. The first three merge candidates are sorted together. The next three merge candidates are sorted together. The template size (width of the left template or height of the upper template) is 1. The subgroup size is 3. 2.6. Adaptive Merge Candidate List Assume that the number of merging candidates is 8. The first 5 merging candidates are used as the first subgroup, and the following 3 merging candidates are used as the second subgroup (ie, the last subgroup). Fig.16 An example diagram 1600 illustrating the reordering process in an encoder is shown. For an encoder, such as Fig.16 As shown, after the merge candidate list is constructed at block 1602 , some merge candidates are adaptively reordered in ascending order of merge candidate cost. More specifically, at block 1604, the template matching costs of the merge candidates in all subgroups except the last subgroup are calculated; then at block 1606, the merge candidates in their own subgroups except the last subgroup are reordered; finally, at block 1608, the final merge candidate list will be obtained. Fig.17 An example diagram 1700 illustrating the reordering process in a decoder is shown. For the decoder, after the merge candidate list is constructed, as Fig.17 As shown, some / no merge candidates are adaptively reordered in ascending order of merge candidate cost. Fig.17 In the example, the subgroup in which the selected (signal) merging candidate is located is called the selected subgroup. More specifically, at block 1702, if the selected merge candidate is located in the last subgroup, then at block 1704, the merge candidate list construction process is terminated after the selected merge candidate is derived, and at block 1706, no reordering is performed and the merge candidate list is not changed; otherwise, the execution process is as follows: At block 1708, after all merge candidates in the selected subgroup are derived, the merge candidate list construction process is terminated; at block 1710, the template matching cost of the merge candidates in the selected subgroup is calculated; at block 1712, the merge candidates in the selected subgroup are reordered; finally, at block 1714, a new merge candidate list will be obtained. For both the encoder and the decoder, the template matching cost is derived as a function of T and RT, where T is the set of samples in the template and RT is the set of reference samples for the template. When deriving the reference samples of the template of the merge candidate, the motion vector of the merge candidate is rounded to integer pixel precision. It can also use an 8-tap or 12-tap luma interpolation filter. The reference samples of the template (RT) for bidirectional prediction are derived by weighted averaging the reference samples of the template in reference list 0 (RT0) and the reference samples of the template in reference list 1 (RT1) as follows. RT=((8-w)*RT0+w*RT1+4)>>3 The weight of the reference template in reference list 0 (8-w) and the weight of the reference template in reference list 1 (w) are determined by the BCW index of the merge candidate. BCW indexes equal to {0, 1, 2, 3, 4} correspond to w equal to {-2, 3, 4, 5, 10} respectively. If the local illumination compensation (LIC) flag of the merge candidate is true, the reference samples of the template are derived using the LIC method. The template matching cost is calculated based on the sum of absolute differences (SAD) between T and RT. The template size is 1. This means that the width of the left template and / or the height of the template above is 1. If the codec mode is MMVD, the merge candidates used to derive the base merge candidate are not reordered. If the coding mode is GPM, the merge candidates used to derive the unidirectional prediction candidate list are not reordered. 2.7. Template Matching (TM) Template Matching (TM) is a decoder-side MV derivation method used to refine the motion information of the current CU by finding the closest match between a template in the current picture (i.e., the top and / or left neighboring blocks of the current CU) and a block in a reference picture (i.e., the same size as the template). Fig.18 An example diagram 1800 illustrating the execution of template matching on a search area around an initial MV is shown. Fig.18 As shown, in the [-8, +8] pixel search range, a better MV is searched around the initial motion of the current CU. This paper adopts template matching and makes two modifications: the search step size is determined based on the AMVR mode, and the TM can be cascaded using the bilateral matching process in the merge mode. In AMVP mode, the MVP candidate is determined based on the template matching error by selecting the one that achieves the minimum difference between the current block template and the reference block template, and then TM performs MV refinement only on this specific MVP candidate. TM refines the MVP candidate by using an iterative diamond search, starting with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode) within the [-8, +8] pixel search range. The AMVP candidate can be further refined by using a cross search with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode), and then using half-pixels and quarter-pixels in turn according to the AMVR mode specified in Table 2-1. This search process ensures that the MVP candidate still maintains the same MV accuracy as indicated by the AMVR mode after the TM process. Table 2-1. AMVR search patterns and merge patterns using AMVR In merge mode, a similar search method is applied to the merge candidates indicated by the merge index. As shown in Table 1, TM can be performed all the way to 1 / 8 pixel MVD accuracy, or skip those accuracies exceeding half-pixel MVD accuracy, depending on whether an alternative interpolation filter is used according to the merged motion information (i.e., used when AMVR is half-pixel mode). In addition, when TM mode is enabled, template matching can work as an independent process between a block-based bilateral matching (BM) method and a sub-block-based bilateral matching method, or as an additional MV refinement process, depending on whether BM can be enabled according to its enabling condition check. On the encoder side, the TM merge mode will perform MV refinement for each merge candidate. 2.8. Intra-frame template matching Template Matching Prediction (TMP) is a special intra prediction mode that copies the best prediction block from the reconstructed part of the current frame, whose L-shaped template matches the current template. Fig.19 An example diagram 1900 showing template matching prediction is shown. For a predefined search range, the encoder searches for the template that is most similar to the current template in the reconstructed part of the current frame and uses the corresponding block as the prediction block. The encoder then signals the use of this mode and the inverse operation is performed on the decoder side. It is a codec tool mainly applicable for encoding and decoding of screen content. A prediction signal is generated at the decoder side by matching the L-shaped causal neighbors of the current block with another block in a predefined search area. Fig. 20 An example diagram 2000 showing the intra-frame template matching search area used is shown. Specifically, the search range is divided into three areas: R1: In the current CTU, R2: upper left corner outside the current CTU, R3: Above the current CTU, R4: On the left side of the current CTU. In each region, the decoder searches for the template with the minimum SAD relative to the current region and uses its corresponding block as the prediction block. The sizes of all regions (SearchRange_w, SearchRange_h) are set proportionally to the block size (BlkW, BlkH) so that there is a fixed number of SAD comparisons per pixel. That is: SearchRange_w = a*BlkW, SearchRange_h=a*BlkH. where "a" is a constant that controls the gain / complexity tradeoff. 2.9. Template-based Intra-mode export using MPM Fig.21An example diagram 2100 showing templates used in TIMD and their reference samples is shown. TIMD mode is derived from MPM using neighboring templates. TIMD mode is used as an additional intra prediction method for a CU. Fig.21 As shown in , using the reference samples of the template for each candidate mode, the prediction samples of the template are generated. The cost is calculated as the sum of absolute transform differences (SATD) between the predicted and reconstructed samples of the template. The intra prediction mode with the minimum cost is selected as the TIMD mode and is used for intra prediction of the CU. 2.9.1 TIMD mode export For each intra prediction mode in the MPM, the SATD between the prediction and reconstructed samples of the template is calculated. The intra prediction mode with the smallest SATD is selected as the TIMD mode and is used for intra prediction of the current CU. Position-dependent intra prediction combination (PDPC) and gradient PDPC are supported in the derivation of TIMD mode. 2.9.2 TIMD Signaling A flag is signaled in the sequence parameter set (SPS) to enable / disable TIMD. When the flag is true, a CU level flag is signaled to indicate whether TIMD is used for the CU. The TIMD flag is signaled after the MIP flag. If the TIMD flag is equal to true, the remaining syntax elements related to the luma intra prediction mode are skipped. 2.9.3 Interaction with new codecs in ECM-1.0 When the DIMD flag or the MIP flag is equal to true, then the TIMD flag is not signaled and is set equal to false. TIMD is allowed to be combined with ISP and MRL. When TIMD is combined with ISP or MRL and the TIMD flag is equal to true, the derived TIMD mode is used as the intra prediction mode of ISP or MRL. When the auxiliary MPM is enabled, both the primary MPM and the auxiliary MPM are used to derive the TIMD mode. The 6-tap interpolation filter is not used for the TIMD mode export. 2.9.4 Modification of MPM list construction in TIMD mode export During the construction of the MPM list, the intra prediction mode of the neighboring blocks is derived as a plane when they are inter-coded. In order to improve the accuracy of the MPM list, when the neighboring blocks are inter-coded, the propagated intra prediction mode is derived using the motion vector and the reference picture and is used in the construction of the MPM list. 2.10. Adaptive Merge Candidate List In the following, a template is a set of reconstructed samples adjacent to or non-adjacent to a current block. According to the same motion information of the current block, the reference samples of the template are derived. For example, the reference samples of the template are a mapping of the template depending on the motion information. In this case, the reference samples of the template are located by the motion vector of the motion information in the reference picture indicated by the reference index of the motion information. Fig. 22 An example diagram 2200 showing templates and reference samples of templates is shown. Fig. 22 An example is shown, where RT represents a reference sample of template T. When the merge candidate utilizes bidirectional prediction, the reference sample of the template of the merge candidate is represented by RT, and RT may be generated according to RT0 derived from the reference picture in reference picture list 0 and RT1 derived from the reference picture in reference picture list 1. In one example, RT0 includes a set of reference samples on the reference picture of the current block indicated by the reference index of the merge candidate that references the reference picture in reference list 0, wherein the MV of the merge candidate references reference list 0), and in one example, RT1 includes a set of reference samples on the reference picture of the current block indicated by the reference index of the merge candidate that references the reference picture in reference list 1, wherein the MV of the merge candidate references reference list 1). Fig.23 An example diagram 2300 showing templates in reference list 0 and reference list 1 and reference samples of the templates is shown. In one example, the reference samples (RT) of the templates used for bidirectional prediction are derived by taking an equally weighted average of the reference samples (RT0) of the templates in reference list 0 and the reference samples (RT1) of the templates in reference list 1. An example is as follows: RT=(RT0+RT1+1)>>1 In one example, the reference sample (RT) of the template used for bidirectional prediction bi-pred ) is derived by taking a weighted average of the reference samples of the template in reference list 0 (RT0) and the reference samples of the template in reference list 1 (RT1). An example is as follows: RT=((2 N -w)*RT0+w*RT1+2 N-1 )>>N, for example, N=3. In one example, the weight of the reference template in reference list 0 such as (8-w) and the weight of the reference template in reference list 1 such as (w) may be determined by the BCW index of the merge candidate. The merge candidates can be divided into several groups according to some criteria. Each group is called a subgroup. For example, adjacent spatial and temporal merge candidates can be used as the first subgroup, and the remaining merge candidates can be used as the second subgroup. In another example, the first N (N≥2) merge candidates can also be used as the first subgroup, the following M (M≥2) merge candidates can be used as the second subgroup, and the remaining merge candidates can be used as the third subgroup. Note that the proposed method can be applied to the merge candidate list construction process for inter-frame encoded blocks (e.g., translational motion), affine-encoded blocks; or other motion candidate list construction processes (e.g., AMVP list; IBC AMVP list; IBC merge list). W and H are the width and height of the current block (e.g., luminance block). The following description takes the merge candidate list construction process as an example: 1. The merging candidates can be adaptively rearranged in the final merging candidate list according to one or some criteria. a. In one example, firstly, part or all of the process of building the current merge candidate list is called, and then the candidates in the list are reordered. i. Alternatively, the candidates in the first subgroup may be reordered and they should be added before those in the second subgroup, where the first subgroup is added before the second subgroup. (i) In one example, multiple merge candidates of a first category may be first derived and then reordered within the first category; then merge candidates from the second category may be determined based on the reordered candidates in the first category (eg, how to apply pruning). ii. Alternatively, the first merge candidate in the first category may be compared with the second merge candidate in the second category to determine the order of the first merge candidate or the second merge candidate in the final merge candidate list. b. In one example, merge candidates may be adaptively rearranged before retrieval. i. In one example, the process of adaptively arranging merge candidates may be processed before obtaining merge candidates to be used in the motion compensation process. c. In one example, if the width of the current block is greater than the height of the current block, the above candidate is added before the left candidate. d. In one example, if the width of the current block is smaller than the height of the current block, the above candidate is added after the left candidate. e. Whether the merge candidates are adaptively rearranged may depend on the selected merge candidate or the selected merge candidate index. i. In one example, if the selected merge candidate is in the last sub-group, the merge candidates are not adaptively rearranged. f. In one example, costs are assigned to merge candidates, and the merge candidates are adaptively reordered in ascending order of their costs. i. In one example, the cost of merging candidates may be a template matching cost. ii. In one example, the template is a set of reconstructed samples that are adjacent or non-adjacent to the current block. iii. The template matching cost is derived as a function of T and RT, where T is the set of samples in the template and RT is the set of reference samples for the template. (i) How to obtain the reference sample of the template of the merge candidate may depend on the motion information of the merge candidate a) In one example, when deriving reference samples for a template, the motion vector of the merge candidate is rounded to integer pixel precision, where the integer motion vector may be its nearest integer motion vector. b) In one example, when deriving the reference samples of the template, an N-tap interpolation filter is used to obtain the reference samples of the template at the sub-pixel position. For example, N can be 2, 4, 6 or 8. c) In one example, when deriving reference samples for a template, the motion vectors of the merge candidates may be scaled to a given reference picture (eg, for each reference picture list, if available). d) For example, Fig. 22 As shown, a reference sample of the template of the merge candidate is obtained on the reference picture of the current block indicated by the reference index of the merge candidate, which has the MV of the merge candidate or the modified MV (eg, according to items a)-b)). e) For example, when the merge candidate utilizes bidirectional prediction, the reference sample of the template of the merge candidate is represented by RT, and RT can be generated according to RT0 derived from the reference picture in reference picture list 0 and RT1 derived from the reference picture in reference picture list 1. [1] In one example, RT0 includes a set of reference samples on a reference picture of the current block indicated by a reference index of a merge candidate that references a reference picture in reference list 0, where the MV of the merge candidate references reference list 0), [2] In one example, RT1 includes a set of reference samples on a reference picture of the current block indicated by a reference index of a merge candidate that references a reference picture in reference list 1, where the MV of the merge candidate references reference list 1). [3] For example Fig.23 shown. f) In one example, the reference samples (RT) of the templates used for bidirectional prediction are derived by taking an equally weighted average of the reference samples (RT0) of the templates in reference list 0 and the reference samples (RT1) of the templates in reference list 1. An example is as follows: RT=(RT0+RT1+1)>>1 g) In one example, the reference sample (RT) of the template used for bidirectional prediction bi-pred ) is derived by taking a weighted average of the reference samples of the template in reference list 0 (RT0) and the reference samples of the template in reference list 1 (RT1). An example is as follows: RT=((2 N -w)*RT0+w*RT1+2 N-1 )>>N, for example, N=3. h) In one example, the weight of the reference template in reference list 0 such as (8-w) and the weight of the reference template in reference list 1 such as (w) may be determined by the BCW index of the merge candidate. [1] In one example, the BCW index is equal to 0 and w is set equal to -2. [2] In one example, the BCW index is equal to 1 and w is set equal to 3. [3] In one example, the BCW index is equal to 2 and w is set equal to 4. [4] In one example, the BCW index is equal to 3 and w is set equal to 5. [5] In one example, the BCW index is equal to 4 and w is set equal to 10. i) In one example, if the local illumination compensation (LIC) flag of the merge candidate is true, the reference sample of the template is derived using the LIC method. (ii) The cost can be calculated based on the sum of absolute differences (SAD) between T and RT. a) Alternatively, the cost can be calculated based on the sum of absolute transformed differences (SATD) of T and RT. b) Alternatively, the cost can be calculated based on the sum of squared differences (SSD) of T and RT. c) Alternatively, the cost can be calculated based on weighted SAD / weighted SATD / weighted SSD. (iii) In addition to the SAD calculated in (ii), the cost may also consider the continuity (Boundary_SAD) between RT and the reconstructed samples adjacent or non-adjacently adjacent to T. For example, the reconstructed samples adjacent or non-adjacently adjacent to T on the left and / or above are considered. a) In one example, the cost may be calculated based on SAD and Boundary_SAD. [1] In one example, the cost may be calculated as (SAD + w*Boundary_SAD). w may be predefined, signaled, or derived from decoded information. 2. Whether and / or how to reorder merge candidates may depend on the category of the merge candidates. a. In one example, only adjacent spatial and temporal merging candidates may be reordered. b. In one example, only adjacent spatial merging candidates, STMVP merging candidates and temporal merging candidates may be reordered. c. In one example, only adjacent spatial merging candidates, STMVP merging candidates, temporal merging candidates and non-adjacent spatial merging candidates may be reordered. d. In one example, only adjacent spatial merging candidates, STMVP merging candidates, temporal merging candidates, non-adjacent spatial merging candidates and HMVP merging candidates may be reordered. e. In one example, only adjacent spatial merging candidates, STMVP merging candidates, temporal merging candidates, non-adjacent spatial merging candidates, HMVP merging candidates and pairwise average merging candidates may be reordered. f. In one example, only adjacent spatial merging candidates, temporal merging candidates, HMVP merging candidates and pairwise average merging candidates may be reordered. g. In one example, only adjacent spatial merging candidates, temporal merging candidates and HMVP merging candidates may be reordered. h. In one example, only adjacent spatial merging candidates may be reordered. i. In one example, only the first subgroup can be reordered. j. In one example, the last subgroup cannot be reordered. k. In one example, only the top N merge candidates may be reordered. i. In one example, N is set equal to 5. 1. In one example, for candidates that are not reordered, they will be arranged in the merge candidate list according to the original order. m. In one example, candidates that are not to be reordered may be placed after candidates to be reordered. n. In one example, candidates that are not to be reordered may be placed before candidates to be reordered. o. In one example, some combinations of the above items (a-k) can be reordered. p. Different subgroups can be reordered individually. q. Two candidates in different subgroups cannot be compared and / or reordered. r. If the first subgroup is before the second subgroup, then after reordering, the first candidate in the first subgroup needs to be placed before the second candidate in the second subgroup. 3. Whether and / or how to reorder merge candidates may depend on the codec mode. a. In one example, if the coding mode is the normal merge mode, the merge candidates may be reordered. b. In one example, if the codec mode is MMVD, the merge candidates from which the base merge candidate is derived are not reordered. i. Alternatively, the reordering method may be different for MMVD mode and other merging modes. c. In one example, if the coding mode is CIIP, the merge candidate for combining with intra prediction is based on the reordered merge candidate. i. Alternatively, the reordering method may be different for CIIP mode and other merge modes. d. In one example, if the coding mode is GPM, the merge candidates used to derive the unidirectional prediction candidate list are not reordered. i. Alternatively, the reordering method may be different for GPM mode and other merge modes. e. In one example, if the coding mode is the triangulation mode, the merge candidates used to derive the unidirectional prediction candidate list are not reordered. i. Alternatively, the reordering method may be different for triangle mode and other merging modes. f. In one example, if the coding mode is a sub-block based merge mode, merge candidates based on part or all of the sub-blocks are reordered. i. Alternatively, the reordering method may be different for the sub-block based merge mode and other merge modes. ii. In one example, merging candidates based on unidirectional predicted sub-blocks are not reordered. iii. In one example, SbTMVP candidates are not reordered. iv. In one example, the constructed affine candidates are not reordered. v. In one example, zero-filled affine candidates are not reordered. 4. Whether and / or how to reorder merging candidates may depend on the available number of adjacent spatial and / or STMVP and / or temporal merging candidates. 5. Whether merge candidates need to be reordered may depend on decoded information (eg, the width and / or height of the CU). a. In one example, if the height is greater than or equal to M, the width is greater than or equal to N, and the width*height is greater than or equal to R, then the merge candidates may be reordered. i. In one example, M, N, and R are set equal to 8, 8, and 128. ii. In one example, M, N, and R are set equal to 16, 16, and 512. b. In one example, if the height is greater than or equal to M and the width is greater than or equal to N, the merge candidates may be reordered. i. In one example, M and N are set equal to 8 and 8. ii. In one example, M and N are set equal to 16 and 16. 6. Subgroup size can be adaptive. a. In one example, the subgroup size is determined based on the available number of adjacent spatial and / or STMVP and / or temporal merging candidates denoted as N. i. In one example, if N is less than M and greater than Q, the subgroup size is set to N; ii. In one example, if N is less than or equal to Q, no reordering is performed; iii. In one example, if N is greater than or equal to M, the subgroup size is set to M. iv. In one example, M and Q are set equal to 5 and 1 respectively. (i) Alternatively, M and / or Q may be predefined, signaled, or derived from decoded information. b. In one example, the subgroup size is decided according to the available number of adjacent spatial and temporal merging candidates denoted as N. i. In one example, if N is less than M and greater than Q, the subgroup size is set to N; ii. In one example, if N is less than or equal to Q, no reordering is performed; iii. In one example, if N is greater than or equal to M, the subgroup size is set to M. iv. In one example, M and Q are set equal to 5 and 1 respectively. 7. The template shape can be adaptive. a. In one example, the template may include only neighboring samples to the left of the current block. b. In one example, the template may only include neighboring samples above the current block. c. In one example, the template shape is selected according to the CU shape. d. In one example, the width of the left template is selected according to the CU height. i. For example, if H<=M, the size of the left template is w1xH; otherwise, the size of the left template is w2xH. e. In one example, M, w1, and w2 are set equal to 8, 1, and 2, respectively. f. In one example, the height of the upper template is selected according to the CU width. i. For example, if W<=N, the size of the upper template is Wxh1; otherwise, the size of the upper template is Wxh2. (i) In one example, N, h1, and h2 are set equal to 8, 1, and 2, respectively. g. In one example, the width of the left template is selected according to the CU width. i. For example, if W<=N, the size of the left template is w1xH; otherwise, the size of the left template is w2xH. (i) In one example, N, w1, and w2 are set equal to 8, 1, and 2, respectively. h. In one example, the height of the upper template is selected according to the CU height. i. For example, if H<=M, the size of the upper template is Wxh1; otherwise, the size of the upper template is Wxh2. (i) In one example, M, h1, and h2 are set equal to 8, 1, and 2, respectively. i. In one example, samples of the template and reference samples of the template samples may be downsampled or down-sampled before being used to calculate the cost. i. Whether and / or how to downsample may depend on the CU size. ii. In one example, no downsampling is performed on the short edge of the CU. 8. In the above examples, the merge candidate is one that is included in the final merge candidate list (eg, after pruning). a. Alternatively, the merge candidate is one derived from a given spatial or temporal block or HMVP list, or derived in other ways, even though it may not be included in the final merge candidate list. 9. Templates can include samples of specific color components. a. In one example, the template includes samples of the luma component only. 10. Whether to apply adaptive merge candidate list reordering may depend on a message transmitted by signal in VPS / SPS / PPS / sequence header / picture header / slice header / CTU / CU / TU / PU. It may also be based on the region of the signal. For example, the picture is evenly or unevenly divided into groups of CTU / CU, and a flag is encoded for each group to indicate whether the merge candidate list reordering is applied. 2.11. Adaptive Motion Candidate List 1. The motion candidates in the motion candidate list of the block may be adaptively rearranged according to one or some criteria to derive a reordered motion candidate list, and the block is encoded / decoded according to the reordered motion candidate list. a. The motion candidates in the motion candidate list of a block may be adaptively rearranged according to one or some criteria to derive a reordered motion candidate list, which is not a conventional merge candidate list. b. In one example, whether and / or how to reorder motion candidates may depend on the codec mode (e.g., affine merge, affine AMVP, regular merge, regular AMVP, GPM, TPM, MMVD, TM merge, CIIP, GMVD, affine MMVD). c. In one example, whether and / or how to reorder motion candidates may depend on the category of the motion candidates (eg, spatial, temporal, STMVP, HMVP, pairwise, SbTMVP, constructed affine, inherited affine). d. In one example, the motion candidate list may be an AMVP candidate list. e. In one example, the motion candidate list may be a merge candidate list. f. In one example, the motion candidate list may be an affine merge candidate list. g. In one example, the motion candidate list may be a sub-block based merge candidate list. h. In one example, the motion candidate list may be a GPM merge candidate list. i. In one example, the motion candidate list may be a TPM merge candidate list. j. In one example, the motion candidate list may be a TM merge candidate list. k. In one example, the motion candidate list may be a candidate list of an MMVD codec block. 1. In one example, the motion candidate list may be a candidate list of a DMVR codec block. 2. How to adaptively rearrange the motion candidates in the motion candidate list may depend on decoded information, for example, the category of the motion candidate, the category of the motion candidate list, and the encoding and decoding tool. a. In one example, for different motion candidate lists, different criteria may be used to re-rank the motion candidate lists. i. In one example, the criteria may include how to select a template. ii. In one example, the criteria may include how to calculate the template cost. iii. In one example, the criteria may include how many candidates in the candidate list and / or how many subgroups need to be reordered. b. In one example, motion candidates in a motion candidate list are first adaptively rearranged to construct a fully rearranged candidate list or a partially rearranged candidate list, and then at least one motion candidate indicated by at least one index is retrieved from the rearranged candidate list to derive final motion information to be used by the current block. c. In one example, the motion candidates before refinement (e.g., using TM for TM-coded blocks; adding MVD for MMVD-coded blocks) are first adaptively rearranged to construct a fully rearranged candidate list or a partially rearranged candidate list. Then, at least one motion candidate indicated by at least one index is retrieved from the rearranged candidate list, and refinement (e.g., using TM for TM-coded blocks; adding MVD for MMVD-coded blocks) is applied to the retrieved motion candidate to derive final motion information of the current block. d. In one example, refinement is applied to at least one motion candidate in the motion candidate list (e.g., using TM for TM codec blocks; adding MVD for MMVD codec blocks), and then they are adaptively rearranged to construct a fully rearranged candidate list or a partially rearranged candidate list, and then at least one motion candidate indicated by at least one index is retrieved from the rearranged candidate list to derive final motion information without any further refinement for the current block. 3. In one example, new MERGE / AMVP motion candidates may be generated based on the candidate re-ranking. i. For example, the candidate L0 motion and L1 motion can be reordered separately. ii. For example, a new bi-prediction merging candidate may be constructed by combining one from the reordered L0 motion and another from the reordered L1 motion. iii. For example, new uni-directional prediction merging candidates can be generated by reordering L0 or L1 motion. 2.12. Adaptive Motion Candidate List For sub-block motion prediction, if the sub-block size is Wsub*Hsub, the height of the upper template is Ht, and the width of the left template is Wt, the upper template can be regarded as a composition of several sub-templates with a size of Wsub*Ht, and the left template can be regarded as a composition of several sub-templates with a size of Wt*Hsub. After deriving the reference sample of each sub-template in a similar manner as described above, the reference sample of the template is derived. Fig.24 An example diagram 2400 showing a template for a block with sub-block motion using motion information of a sub-block of a current block and reference samples of the template is shown. Fig.25An example diagram 2500 showing templates and reference samples of the templates for blocks with sub-block motion using motion information of each sub-template is shown. It should be noted that the terms mentioned below are not limited to specific terms defined in existing standards. Any changes in codecs also apply. For example, the term "GPM" is used to represent any codec that derives two sets of motion information and uses the derived information and the segmentation pattern to obtain the final prediction. For example, TPM is also considered a GPM. Note that the proposed method can be applied to the merge candidate list construction process of inter-coded blocks (e.g., translational motion), affine-coded blocks, or IBC-coded blocks; or other motion candidate list construction processes (e.g., ordinary AMVP list; affine AMVP list; IBC AMVP list). W and H are the width and height of the current block (eg, luma block). 1. In one example, if the coding mode is TM merging, some or all TM merging candidates may be reordered. a. In one example, if the coding mode is TM merging, part or all of the original TM merging candidates may be reordered before the TM refinement process. b. Alternatively, if the coding mode is TM merging, some or all of the refined TM merging candidates may be reordered after the TM refinement process. c. Alternatively, if the coding mode is TM merging, the TM merging candidates may not be reordered. d. Alternatively, the reordering method may be different for TM merge mode and other merge modes. 2. In one example, if the coding mode is a sub-block based merge mode, merge candidates based on part or all of the sub-blocks may be reordered. a. Alternatively, the reordering method may be different for the sub-block based merge mode and other merge modes. b. In one example, a template can be divided into sub-templates. Each sub-template can have individual motion information. i. In one example, the cost for the reorder candidate can be derived based on the cost of each sub-template. For example, the cost for the reorder candidate can be calculated as the sum of the costs of all sub-templates. For example, the cost of the sub-template can be calculated as SAD, SATD, SSD, or any other distortion measure between the sub-template and its corresponding reference sub-template. c. In one example, to derive reference samples of the sub-template, the motion information of the sub-blocks in the first row and first column of the current block may be used. i. In one example, the motion information of a sub-template can be derived (eg, copied) from its neighboring sub-block in the current block. Fig.24 shown. d. In one example, in order to derive a reference sample of a sub-template, the motion information of the sub-template may be derived without referring to the motion information of the sub-block in the current block. Fig.25 shown. i. In one example, the motion information of each sub-template is calculated based on the affine model of the current block. (i) In one example, the motion vector of the center sample of each sub-block containing the sub-template calculated according to the affine model of the current block is regarded as the motion vector of the sub-template. (ii) In one example, the motion vector of the center sample of each sub-template calculated according to the affine model of the current block is regarded as the motion vector of the sub-template. (iii) For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as: (iv) For the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as: (v) For (iii) and (iv), the coordinates of the upper left corner, upper right corner, and lower left corner of the current block are (0, 0), (W, 0), and (0, H), and the motion vectors of the upper left corner, upper right corner, and lower left corner of the current block are (mv 0x , mv 0y )、(mv 1x , mv 1y ) and (mv 2x , mv 2y ). (vi) In one example, the coordinates (x, y) in the above equation can be set equal to the position in the template or the position of the sub-template. For example, the coordinates (x, y) can be set equal to the center position of the sub-template. e. In one example, this scheme can be applied to affine merge candidates. f. In one example, this scheme can be applied to affine AMVP candidates. g. In one example, this scheme can be applied to SbTMVP merge candidates. h. In one example, this scheme can be applied to GPM merge candidates. i. In one example, the scheme can be applied to TPM merge candidates. j. In one example, this solution can be applied to TM to refine merge candidates. k. In one example, this solution can be applied to DMVR to refine merge candidates. 1. In one example, this solution can be applied to MULTI_PASS_DMVR to refine the merge candidates. 3. In one example, if the codec mode is MMVD, the merge candidates from which the base merge candidate is derived may be reordered. a. In one example, a reordering process may be applied to merge candidates before they are refined by a signaled or derived MVD. b. For example, the reordering method may be different for MMVD mode and other merge modes. 4. In one example, if the coding mode is MMVD, the merge candidates after MMVD refinement may be reordered. a. In one example, the reordering process can be applied to the merge candidates after they are refined by a signaled or derived MVD. b. For example, the reordering method may be different for MMVD mode and other merge modes. 5. In one example, if the coding mode is affine MMVD, the merge candidates from which the base merge candidate is derived may be reordered. a. In one example, a reordering process may be applied to affine merge candidates before they are refined by a signaled or derived MVD. b. For example, the reordering method can be different for the affine MMVD mode and other merge modes. 6. In one example, if the coding mode is affine MMVD, the merge candidates after affine MMVD refinement may be reordered. a. In one example, the reordering process can be applied to affine merge candidates after they are refined by a signaled or derived MVD. b. For example, the reordering method can be different for the affine MMVD mode and other merge modes. 7. In one example, if the codec mode is GMVD, the merge candidates from which the base merge candidate is derived may be reordered. a. In one example, a reordering process may be applied to merge candidates before they are refined by a signaled or derived MVD. b. For example, the reordering method may be different for GMVD mode and other merge modes. 8. In one example, if the codec mode is GMVD, the merge candidates after GMVD refinement may be reordered. a. In one example, the reordering process can be applied to the merge candidates after they are refined by a signaled or derived MVD. b. For example, the reordering method may be different for GMVD mode and other merge modes. 9. In one example, if the codec mode is GPM, the merge candidates may be reordered. a. In one example, a reordering process may be applied to the original merge candidates before they are used to derive the per-partition GPM candidate list (also referred to as the uni-directional prediction candidate list of the GPM). b. In one example, if the coding mode is GPM, the merge candidates in the unidirectional prediction candidate list may be reordered. c. In one example, the GPM unidirectional prediction candidate list can be constructed based on the reordering. i. In one example, a candidate with bi-directional prediction (also referred to as a bi-directional prediction candidate) may be separated into two uni-directional prediction candidates. (i) If the number of original merging candidates is M, at most 2M unidirectional prediction candidates can be separated from them. ii. In one example, unidirectional prediction candidates separated from bidirectional prediction candidates may be placed into an initial unidirectional prediction candidate list. iii. In one example, candidates in the initial unidirectional prediction candidate list may be reordered using template matching cost. iv. In one example, the first N unidirectional prediction candidates with smaller template matching costs can be used as the final GPM unidirectional prediction candidates. For example, N is equal to M. d. In one example, after deriving the GPM unidirectional prediction candidate list, a bidirectional prediction list for the combination of partition 0 and partition 1 is constructed, and then the bidirectional prediction list is reordered. i. In one example, if the number of GPM unidirectional prediction candidates is M, the number of combined bidirectional prediction candidates is M*(M-1). e. Alternatively, the reordering method may be different for GPM mode and other merge modes. 2.13. Adaptive Motion Candidate List It should be noted that the terms mentioned below are not limited to specific terms defined in existing standards. Any changes in codecs also apply. For example, the term "GPM" is used to represent any codec that derives two sets of motion information and uses the derived information and the partitioning pattern to obtain the final prediction. For example, TPM is also considered a GPM. Note that the proposed method can be applied to the merge candidate list construction process of inter-coded blocks (e.g., translational motion), affine-coded blocks, or IBC-coded blocks; or other motion candidate list construction processes (e.g., ordinary AMVP list; affine AMVP list; IBC AMVP list). W and H are the width and height of the current block (eg, luma block). 1. The reference sample (RT) of the template or sub-template used for bidirectional prediction is derived by taking an equally weighted average of the reference sample (RT0) of the template or sub-template in reference list 0 and the reference sample (RT1) of the template or sub-template in reference list 1. An example is as follows: RT(x,y)=(RT0(x,y)+RT1(x,y)+1)>>1 2. The reference sample (RT) of the template or sub-template used for bidirectional prediction is derived by weighted averaging the reference sample (RT0) of the template or sub-template in reference list 0 and the reference sample (RT1) of the template or sub-template in reference list 1. a. An example is as follows: RT(x,y)=((2 N -w)*RT0(x,y)+w*RT1(x,y)+2 N-1 )>>N, for example, N=3. b. The weights may be determined by the BCW index, or may be derived on the fly or predefined or determined by the weights used in weighted prediction. c. In one example, the weight of the reference template in reference list 0 such as (8-w) and the weight of the reference template in reference list 1 such as (w) may be determined by the BCW index of the merge candidate. i. In one example, the BCW index is equal to 0 and w is set to -2. ii. In one example, the BCW index is equal to 1 and w is set to 3. iii. In one example, the BCW index is equal to 2 and w is set to 4. iv. In one example, BCW index equals 3 and w is set to 5. v. In one example, BCW index equals 4 and w is set to 10. 3. Reference samples of the proposed template can be derived using the LIC method. a. In one example, the LIC parameters of both the left template and the upper template are the same as the LIC parameters of the current block. b. In one example, the LIC parameters of the left template are derived as the LIC parameters of the current block, and the current block uses its original motion vector plus the motion vector offset (-Wt, 0) as the motion vector of the current block. c. In one example, the LIC parameters of the upper template are derived as the LIC parameters of the current block, and the current block uses its original motion vector plus a motion vector offset (0, -Ht) as the motion vector of the current block. d. Alternatively or additionally, if the local illumination compensation (LIC) flag of the merge candidate is true, the above items may be applied. 4. It is proposed that reference samples of templates or sub-templates can be derived using the OBMC method. In the following discussion, "template" can refer to a template or a sub-template. Fig.26 An example diagram 2600 showing templates and reference samples of the templates for a block with OBMC is shown. a. In one example, in order to derive the reference samples of the upper template, the motion information of the sub-block in the first row of the current block and its adjacent sub-blocks above it is used. And the reference samples of all sub-templates constitute the reference samples of the upper template. An example is Fig.26 shown. b. In one example, to derive the reference samples of the left template, the motion information of the sub-block in the first column of the current block and its adjacent sub-blocks to the left is used. And the reference samples of all sub-templates constitute the reference samples of the left template. An example is Fig.26 shown. c. In one example, the sub-block size is 4x4. d. The reference sample of the sub-template based on the motion vector of the neighboring sub-block is denoted as P N , where N represents the index of the adjacent upper and left sub-blocks, and the reference sample of the sub-template based on the motion vector of the sub-block of the current block is represented as P C For P generated based on the motion vectors of vertically (horizontally) adjacent sub-blocks N , P N The samples in the same row (column) of P are added to P with the same weighting factor. C . i. The reference sample (P) of the sub-template can be derived as P = W N *P N +W C *P C . ii. In one example, if the height of the upper template or the width of the left template is greater than or equal to 4, then the weight factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} are used for P NThe {first, second, third, fourth} rows (columns) of P, and weight factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are used for P C The {first, second, third, fourth} rows (columns) of . iii. In one example, if the height of the upper template or the width of the left template is greater than or equal to 2, the weighting factors {1 / 4, 1 / 8} are used for P N The {first, second} rows (columns) of P, and the weighting factors {3 / 4, 7 / 8} are used for P C The {first, second} row (column) of . iv. In one example, if the height of the upper template or the width of the left template is greater than or equal to 1, the weighting factor {1 / 4} is used for P N The first row (column) of P is used, and the weighting factor {3 / 4} is used for P C The first row (column) of . e. If merge candidates are allocated with OBMC enabled, the above items may be applied. 5. In one example, if the merged candidate uses multi-hypothesis prediction, the reference sample of the template can be derived using the multi-hypothesis prediction method. 6. A template may include samples of (multiple) specific color components. a. In one example, the template includes samples of the luma component only. b. Alternatively, the template only includes samples of any component, such as Cb / Cr / R / G / B. 7. Whether and / or how to reorder motion candidates may depend on the category of the motion candidates. a. In one example, only adjacent spatial and temporal motion candidates can be reordered. b. In one example, only adjacent spatial, STMVP and temporal motion candidates may be reordered. c. In one example, only adjacent spatial, STMVP, temporal motion candidates and non-adjacent spatial motion candidates may be reordered. d. In one example, only adjacent spatial, STMVP, temporal motion candidates, non-adjacent spatial and HMVP motion candidates may be reordered. e. In one example, only adjacent spatial, STMVP, temporal motion candidates, non-adjacent spatial, HMVP, and pairwise average motion candidates may be reordered. f. In one example, only neighboring spatial, temporal, HMVP and pairwise average motion candidates may be reordered. g. In one example, only adjacent spatial, temporal and HMVP motion candidates may be reordered. h. In one example, only adjacent spatial motion candidates may be reordered. i. In one example, motion candidates based on unidirectional prediction sub-blocks are not reordered. j. In one example, SbTMVP candidates are not reordered. k. In one example, inherited affine motion candidates are not reordered. 1. In one example, the reconstructed affine motion candidates are not reordered. m. In one example, zero-filled affine motion candidates are not reordered. n. In one example, only the top N motion candidates may be reordered. i. In one example, N is set equal to 5. 8. In one example, the motion candidates may be divided into multiple subgroups. Whether and / or how to reorder the motion candidates may depend on the subgroups of the motion candidates. a. In one example, only the first subgroup can be reordered. b. In one example, the last subgroup may not be reordered. c. In one example, the last subgroup may not be reordered. However, if the last subgroup is also the first subgroup, it may be reordered. d. Different subgroups can be reordered individually. e. Two candidates in different subgroups cannot be compared and / or re-ranked. f. If the first subgroup precedes the second subgroup, then after reordering the first candidate in the first subgroup must be placed before the second candidate in the second subgroup. 9. In one example, motion candidates not included in the reordering process may be processed in a specified manner. a. In one example, for candidates that are not reordered, they will be arranged in the merge candidate list according to the original order. b. In one example, candidates that are not to be reordered may be placed after candidates to be reordered. c. In one example, candidates that are not to be reordered may be placed before candidates to be reordered. 10. Whether to apply adaptive merge candidate list reordering may depend on a message transmitted by signal in VPS / SPS / PPS / sequence header / picture header / slice header / CTU / CU / TU / PU. It may also be based on the region of the signal. For example, the picture is evenly or unevenly divided into groups of CTU / CU, and a flag is encoded for each group to indicate whether the merge candidate list reordering is applied. 2.14. Cost Functions Utilized in Codec Data Refinement in Image / Video Codecs The term "block" may refer to a codec tree block (CTB), a codec tree unit (CTU), a codec block (CB), a CU, a PU, a TU, a PB, a TB, or a video processing unit including a plurality of samples / pixels. A block may be rectangular or non-rectangular. In the present disclosure, the phrase "motion candidate" may refer to a merged motion candidate in a regular / extended merge list indicated by a merge candidate index, or an AMVP motion candidate in a regular / extended AMVP list indicated by an AMVP candidate index, or an AMVP motion candidate, or a merged motion candidate. In this disclosure, a motion candidate is referred to as "refined" if its motion information is modified based on information transmitted by signal from the encoder or derived at the decoder. For example, the motion vector can be refined by DMVR, FRUC, TM Merge, TM AMVP, TM GPM, TM CIIP, TM Affine, MMVD, GMVD, Affine MMVD, BDOF, etc. In the present disclosure, the phrase "codec data refinement" may refer to a refinement process to derive or refine a signaled / decoded / derived prediction mode, prediction direction, or signaled / decoded / derived motion information, prediction, and / or reconstructed samples for a block. In one example, the refinement process may include motion candidate reordering. In the following discussion, a "template-based" block may refer to a block that uses a template matching-based method to derive or refine the coded information during the encoding / decoding process, such as template matching-based motion derivation, template matching-based motion list reconstruction, LIC, symbol prediction, template matching-based block vector (e.g., used in IBC mode) derivation, DIMD, template matching-based non-inter (e.g., intra) prediction, etc. The template-based coding method may be combined with any other coding tools, such as MMVD, CIIP, GPM, FRUC, Affine, BDOF, DMVR, OBMC, etc. In another example, a "template-based" block may also refer to a block that uses neighboring reconstructed samples (adjacent or non-adjacent) to derive or refine its decoded information based on specific rules, such as the DIMD method in Section 2.27 and the TIMD method in Section 2.29). In the following discussion, a "bilateral codec-based" block may refer to a block that uses a bilateral matching-based method to derive or refine the coded information during the encoding / decoding process, such as bilateral matching-based motion derivation, bilateral matching-based motion list reconstruction, etc. The bilateral codec-based method can be combined with any other codec tools, such as MMVD, CIIP, GPM, FRUC, Affine, DMVR, etc. W and H are the width and height of the current block (eg, luma block). W*H is the size of the current block (eg, luma block). In the following discussion, Shift(x,s) is defined as Shift(x,s)=(x+offset)>>s, where offset is an integer, such as offset=0 or offset=1<<(s-1) or offset=(1<<(s-1))-1. In another example, offset depends on x. For example, offset = (x<0?(1<<(s-1)):((1<<(s-1)-1). 1. In addition to error measurement, it is proposed to add conventional items to the cost calculation process. a) In one example, the cost is defined as: E+W*RI, where E represents the output of the error function and W is the weight applied to the regular term represented by RI. i. In one example, to process a template-based codec block / a bilateral-based codec block, the cost function is set to: E+W*RI, where E can be SAD / MRSAD / SATD Or otherwise, RI is the estimated bit for the motion vector / motion vector difference, and W is the weight, which may depend on QP / temporal layer, etc., for example. ii. Alternatively, the cost may be defined as: w0*E+W1*RI, where E represents the output of the error function, W1 is the weight applied to the regular term represented by RI, and w0 is the weight applied to the output of the error function. (i) Alternatively or additionally, W1 may be set to 0. b) In one example, the regular term may be multiplied by a weighting rate. i. In one example, weights are derived on the fly. ii. In one example, the weight is set to the lambda used in the full RDO process. iii. In one example, the weights are set to the square root of the lambda used in the full RDO process. c) In one example, the cost is calculated as E+Shift(W*RI,s), where s and W are integers. i. Alternatively, the cost is calculated as Shift((E << s) + W * RI, s), where s and W are integers. 2. An error function different from SAD / MR - SAD (Sum of Absolute Differences with Mean Removed) is proposed to process template - based encoding / decoding blocks / bilateral - based encoding / decoding blocks. a) In one example, the error function can be: i. SATD, ii. MR - SATD, iii. Gradient information, iv. SSE / SSD, v. MR - SSE / MR - SSD, vi. Weighted SAD / Weighted MR - SAD, vii. Weighted SATD / Weighted MR - SATD, viii. Weighted SSD / Weighted MR - SSD, ix. Weighted SSE / Weighted MR - SSE. b) Alternatively or additionally, an adaptive selection of the error function among different cost functions such as the above - mentioned error functions and SAD / MR - SAD is proposed. i. The selection can be determined instantaneously. 3. When using an error function based on MR - X (e.g., X is SATD, SAD, SSE) (such as MR - SAD / MR - SATD, etc.), the following can be further applied: a) In one example, the mean can be calculated taking into account all samples in the block to be compared. b) In one example, the mean can be calculated taking into account a part of the samples in the block to be compared. c) In one example, the mean and the X function can depend on the same samples in the block. i. In one example, the mean and the X function can be calculated using all samples in the block. ii. In one example, the mean and the X function can be calculated using a part of the samples in the block. d) In one example, the mean and the X function can depend on at least one different sample in the block. i. In one example, the mean can be calculated using all samples, while the X function can depend on a part of the samples in the block. ii. In one example, the mean can be calculated using a part of the samples, while the X function can depend on all samples in the block. 4. The template / bilateral matching cost can be calculated by applying a cost factor to the error cost function. a) In one example, it is proposed to bias motion candidates towards the front during the template / bilateral matching based re-ranking process. i. In one example, the motion candidate in the i-th position is assigned a smaller cost factor than the cost factor of the motion candidate in the (i+1)-th position. ii. In one example, motion candidates in the i-th group (eg, involving M motion candidates) are assigned a smaller cost factor than a cost factor of motion candidates in the (i+1)-th group (eg, involving N motion candidates). (i) In one example, M may be equal to N. For example, M=N=2. (ii) In one example, M may not be equal to N. For example, M=2, N=3. b) In one example, it is proposed that the refinement process based on template / bilateral matching tends to be closer to the original Search MV for MV. i. In one example, each search region is assigned a cost factor, which may be determined by the distance (eg, integer pixel precision delta MV) between each search MV in the search region and the start MV. ii. In one example, each search region is assigned a cost factor, which may be determined by the distance between a center search MV and a start MV in the search region (eg, a delta MV in integer pixel precision). iii. In one example, each search MV is assigned a cost factor, which may be determined by the distance between each search MV and the start MV (eg, a delta MV in integer pixel precision). 5. The above method can be applied to any codec data refinement process, for example, for blocks based on template codecs, for blocks based on bilateral codecs (eg, DMVR in VVC). 6. For different template matching refinement methods, the template matching cost measure can be different. a. In one example, the template matching refinement method may be a motion candidate reordering based on template matching. b. In one example, the template matching refinement method may be a motion derivation based on template matching. i. In one example, the refinement method may be TM AMVP, TM Merge, and / or FRUC. c. In one example, the template matching refinement method may be template matching based motion refinement. ii. In one example, the refinement method may be TM GPM, TM CIIP and / or TM Affine. d. In one example, the template matching refinement method may be a block vector derivation based on template matching. e. In one example, the template matching refinement method may be intra-mode derivation based on template matching. iii. In one example, the thinning method may be DIMD and / or TIMD. f. In one example, the template matching cost measure may be calculated based on the sum of absolute differences (SAD) between the current template and the reference template. g. In one example, the template matching cost measure may be calculated based on the mean-subtracted SAD between the current template and the reference template. h. In one example, SAD and mean-removed SAD (MR-SAD) can be selectively utilized according to the size of the current block. i. In one example, mean removed SAD is used for blocks of size greater than M, and SAD is used for blocks of size less than or equal to M. i. (i) In one example, M is 64. i. In one example, SAD and mean-removed SAD (MR-SAD) can be selectively utilized according to the LIC flag of the current block. i. In one example, if the LIC flag of the current block is false, the template matching cost measure may be SAD. ii. In one example, if the LIC flag of the current block is true, the template matching cost measure may be MR-SAD. j. In one example, the template matching cost measure may be calculated based on the sum of absolute transformed differences (SATD) between the current template and the reference template. k. In one example, the template matching cost measure may be calculated based on the mean-removed SATD between the current template and the reference template. 1. In one example, SATD and mean-removed SATD (MR-SATD) can be selectively utilized according to the size of the current block. i. In one example, SATD with mean removal is used for blocks of size greater than M, and SATD is used for blocks of size less than or equal to M. (i) In one example, M is 64. m. In one example, SATD and mean-removed SATD (MR-SATD) can be selectively utilized according to the LIC flag of the current block. i. In one example, if the LIC flag of the current block is false, the template matching cost measure can be SATD. ii. In one example, if the LIC flag of the current block is true, the template matching cost measure may be MR-SATD. n. In one example, the template matching cost measure may be calculated based on the sum of squared differences (SSD) between the current template and the reference template. o. In one example, the template matching cost measure may be calculated based on a mean-subtracted SSD between the current template and a reference template. p. In one example, SSD and mean removed SSD (MR-SSD) can be selectively utilized according to the size of the current block. i. In one example, mean removed SSD is used for blocks of size greater than M, and SSD is used for blocks of size less than or equal to M. (i) In one example, M is 64. q. In one example, the template matching cost measurement can be weighted SAD / weighted MR-SAD / selective weighted MR-SAD and SAD / weighted SATD / weighted MR-SATD / selective weighted MR-SATD and SATD / weighted SSD / weighted MR-SSD / selective weighted MR-SSD and SSD between the current template and the reference template. i. In one example, weighting means applying different weights to each sample based on its row and column index in the template block when calculating the distortion between the current template and the reference template. ii. In one example, weighting means applying different weights to each sample based on its position in the template block when calculating the distortion between the current template and the reference template. iii. In one example, weighting means applying different weights to each sample based on its distance to the current block when calculating the distortion between the current template and the reference template. r. In one example, the template matching cost may be calculated as follows: tplCost = w1*mvDistanceCost + w2*distortionCost. i. In one example, distortionCost may be the weighted SAD / weighted MR-SAD / weighted SATD / weighted MR-SATD / weighted SSD / weighted MR-SSD / SAD / MR-SAD / SATD / MR-SATD / SSD / MR-SSD between the current template and the reference template. ii. In one example, mvDistanceCost may be the sum of the absolute mv differences between the search point and the start point in the horizontal and vertical directions. iii. In one example, w1 and w2 may be predetermined, or signaled, or derived from decoded information. (i) In one example, w1 is a weighting factor set to 4 and w2 is a weighting factor set to 1 s. In addition to the SAD calculated in (f), the cost can also consider the continuity between the reference template and the reconstructed samples adjacent or non-adjacent to the current template (Boundary_SAD). For example, on the left and And / or reconstructed samples that are adjacent to or non-adjacently adjacent to the current sample above are considered. i. In one example, the cost can be calculated based on SAD and Boundary_SAD (i) In one example, the cost may be calculated as (SAD + w*Boundary_SAD). w may be predetermined, signaled, or derived from decoded information. 7. The bilateral matching cost measure may be different for different bilateral matching refinement methods. a) In one example, the bilateral matching refinement method may be a motion candidate re-ranking based on bilateral matching. b) In one example, the bilateral matching refinement method may be motion derivation based on bilateral matching. i. In one example, the refinement method may be BM merging and / or FRUC. c) In one example, the bilateral matching refinement method may be motion refinement based on bilateral matching. i. In one example, the refinement method may be BM GPM, BM CIIP and / or BM Affine. d) In one example, the bilateral matching refinement method may be a block vector derivation based on bilateral matching. e) In one example, the bilateral matching refinement method may be an intra-mode derivation based on bilateral matching. f) In one example, the bilateral matching cost measure may be calculated based on the sum of absolute differences (SAD) between two reference blocks / sub-blocks. g) In one example, the bilateral matching cost measure may be calculated based on the mean-removed SAD between two reference blocks / sub-blocks. h) In one example, SAD and mean-removed SAD (MR-SAD) can be selectively utilized according to the size of the current block / sub-block. i. In one example, mean removed SAD is used for blocks / sub-blocks of size greater than M, and SAD is used for blocks / sub-blocks of size less than or equal to M. i. (i) In one example, M is 64. i) In one example, SAD and mean-removed SAD (MR-SAD) can be selectively utilized according to the LIC flag of the current block. i. In one example, if the LIC flag of the current block is false, the bilateral matching cost measure can be SAD. ii. In one example, if the LIC flag of the current block is true, the bilateral matching cost measure may be MR-SAD. j) In one example, the bilateral matching cost measure may be calculated based on the sum of absolute transformed differences (SATD) between two reference blocks / sub-blocks. k) In one example, the bilateral matching cost measure may be calculated based on the mean-removed SATD between two reference blocks / sub-blocks. l) In one example, SATD and mean-removed SATD (MR-SATD) can be selectively utilized according to the size of the current block / sub-block. i. In one example, SATD with mean removal is used for blocks / sub-blocks with size greater than M, and SATD is used for blocks / sub-blocks with size less than or equal to M. (i) In one example, M is 64. m) In one example, SATD and mean-removed SATD (MR-SATD) can be selectively utilized according to the LIC flag of the current block. i. In one example, if the LIC flag of the current block is false, the bilateral matching cost measure can be SATD. ii. In one example, if the LIC flag of the current block is true, the bilateral matching cost measure can be MR-SATD. n) In one example, the bilateral matching cost measure may be calculated based on the sum of squared differences (SSD) between two reference blocks / sub-blocks. o) In one example, the bilateral matching cost measure may be calculated based on the mean-removed SSD between two reference blocks / sub-blocks. p) In one example, SSD and mean removed SSD (MR-SSD) may be selectively utilized according to the size of the current block / sub-block. i. In one example, mean removed SSD is used for blocks / sub-blocks of size greater than M, and SSD is used for blocks / sub-blocks of size less than or equal to M. (i) In one example, M is 64. q) In one example, SSD and mean removed SSD (MR-SSD) can be selectively utilized according to the LIC flag of the current block. i. In one example, if the LIC flag of the current block is false, the bilateral matching cost measure can be SSD. ii. In one example, if the LIC flag of the current block is true, the bilateral matching cost measure can be MR-SSD. r) In one example, the bilateral matching cost measure can be weighted SAD / weighted MR-SAD / selective weighted MR-SAD and SAD / weighted SATD / weighted MR-SATD / selective weighted MR-SATD and SATD / weighted SSD / weighted MR-SSD / selective weighted MR-SSD and SSD between two reference blocks / sub-blocks. i. In one example, weighting means applying different weights to each sample based on its row and column index in the reference block / sub-block when calculating the distortion between two reference blocks / sub-blocks. ii. In one example, weighting means that when calculating the distortion between two reference blocks / sub-blocks, different weights are applied to each sample based on its position in the reference block / sub-block. iii. In one example, weighting means that when calculating the distortion between two reference blocks / sub-blocks, different weights are applied to each sample based on the distance of each sample to the center position of the reference block / sub-block. s) In one example, if MR-SAD / MR-SATD / MR-SSD is used for bilateral matching cost measurement, LIC may not be used when deriving reference blocks / sub-blocks. t) In one example, the bilateral matching cost can be calculated as follows: bilCost = w1*mvDistanceCost+w2*distortionCost. i. In one example, distortionCost can be the weighted SAD / weighted MR-SAD / weighted SATD / weighted MR-SATD / weighted SSD / weighted MR-SSD / SAD / MR-SAD / SATD / MR-SATD / SSD / MR-SSD between two reference blocks / sub-blocks. ii. In one example, mvDistanceCost may be the sum of the absolute mv differences between the search point and the start point in the horizontal and vertical directions. iii. In one example, w1 and w2 may be predetermined, or transmitted by a signal, or derived from decoded information. (i) In one example, w1 is a weighting factor set to 4, and w2 is a weighting factor set to 1. 8. The bilateral matching cost or template matching cost can be calculated based on the prediction / reference samples modified by the function. a) In one example, the prediction / reference samples may be filtered before being used to calculate the bilateral matching cost or the template matching cost. b) In one example, the prediction / reference sample S may be modified to a*S+b before being used to calculate the bilateral matching cost or the template matching cost. c) In one example, the modification may depend on the codec mode of the block, such as whether the block is LIC coded or BCW coded. 2.15. Use of multiple cost functions in codec data refinement in image / video codecs The term "block" may refer to a codec tree block (CTB), a codec tree unit (CTU), a codec block (CB), a CU, a PU, a TU, a PB, a TB, or a video processing unit including a plurality of samples / pixels. A block may be rectangular or non-rectangular. In the present disclosure, the phrase "motion candidate" may refer to a merged motion candidate in a regular / extended merge list indicated by a merge candidate index, or an AMVP motion candidate in a regular / extended AMVP list indicated by an AMVP candidate index, or an AMVP motion candidate, or a merged motion candidate. In this disclosure, a motion candidate is referred to as "refined" if its motion information is modified based on information transmitted by signal from the encoder or derived at the decoder. For example, the motion vector can be refined by DMVR, FRUC, TM Merge, TM AMVP, TM GPM, TM CIIP, TM Affine, MMVD, GMVD, Affine MMVD, BDOF, etc. In the present disclosure, the phrase "codec data refinement" may refer to a refinement process to derive or refine a signaled / decoded / derived prediction mode, prediction direction, or signaled / decoded / derived motion information, prediction and / or reconstructed samples for a block. In one example, the refinement process may include motion candidate reordering. In the following discussion, a "template-based" block may refer to a block that uses a template matching-based method to derive or refine the coded information during the encoding / decoding process, such as template matching-based motion derivation, template matching-based motion list reconstruction, LIC, symbol prediction, template matching-based block vector (e.g., used in IBC mode) derivation, DIMD, template matching-based non-inter (e.g., intra) prediction, etc. The template matching-based method may be combined with any other codec tools, such as MMVD, CIIP, GMP, FRUC, Affine, BDOF, DMVR, OBMC, etc. In another example, a "template-based" block may also refer to a block that uses (adjacent or non-adjacent) neighboring reconstructed samples based on specific rules to derive or refine its decoded information, such as the DIMD method in Section 2.27 and the TIMD method in Section 2.29. In the following discussion, a "bilateral codec-based" block may refer to a block that uses a bilateral matching-based method to derive or refine the coded information during the encoding / decoding process, such as bilateral matching-based motion derivation, bilateral matching-based motion list reconstruction, etc. The bilateral codec-based method can be combined with any other codec tools, such as MMVD, CIIP, GPM, FRUC, Affine, DMVR, etc. W and H are the width and height of the current block (eg, luma block). W*H is the size of the current block (eg, luma block). 1. The cost definition may rely on the output of multiple error functions (eg, distortion measurement methods) regarding the error / difference between two samples / blocks to be evaluated in a codec data refinement process of the current block. a) In one example, the error function may be: i.SAD, ii.SATD, iii.MR-SAD, iv.MR-SATD, v. Gradient information, vi.SSE / SSD, vii.MR-SSE / MR-SSD, viii. Weighted SAD / Weighted MR-SAD, ix. Weighted SATD / weighted MR-SATD, x. Weighted SSD / weighted MR-SSD, xi. Weighted SSE / Weighted MR-SSE. b) In one example, the error function may be performed at a block level or a sub-block level. i. Alternatively, additionally, the error function may be different for the two sub-blocks. ii. Alternatively, additionally, the final output of the estimated error for the block may be based on the outputs of the sub-blocks, for example, the sum of the outputs of the error function applied to each sub-block. 2. When the cost definition depends on the output of multiple functions, the following items may further apply: a) In one example, the cost function may rely on a linear weighted sum of multiple error functions. b) In one example, the cost function may rely on a non-linear weighted sum of multiple error functions. c) In one example, the cost function may further depend on estimated bits for the side information. d) In one example, the cost function may be defined as: Where R represents the estimated bits for the side information, W i and E i denote the weight applied to the i-th error function and the output of the i-th error function, respectively. 3. Multiple refinement processes may be applied to one block, wherein at least more than two different cost functions are applied to at least two refinement processes. a) In one example, a first refinement process may be invoked using a first cost function. Based on the output of the first refinement process, a second cost function is further applied to a second refinement process. b) The above method can be applied to blocks based on template coding. 4. Whether to use multiple refinement processes and / or how to select one or more error functions and / or how to define the cost function and / or which samples to involve in the error function may depend on the decoding information of the current block and / or the decoding information of neighboring (adjacent or non-adjacent) blocks of the current block. a) In one example, how to select one or more error functions and / or how to define the cost function may depend on the codec tools applied to the current block and / or neighboring blocks of the current block. i. In one example, the codec is LIC. (i) In one example, SSD and mean removed SSD (MR-SSD) can be selectively utilized according to the LIC flag of the current block. a) In one example, if the LIC flag of the current block is false, the template matching cost measure can be SSD. b) In one example, if the LIC flag of the current block is true, the template matching cost measure can be MR-SSD. (ii) In one example, if MR-SAD / MR-SATD / MR-SSD is used for the template matching cost measurement, the linear function used in the LIC process when deriving the reference template may not be used. (iii) In one example, if MR-SAD / MR-SATD / MR-SSD is used for the bilateral matching cost measurement, the linear function used in the LIC process when deriving the reference block may not be used. b) In one example, it may depend on block size, temporal layer, low latency check flag, etc. c) In one example, it may depend on whether the motion information of the current block is similar / identical to the motion information of the neighboring blocks. d) In one example, it may depend on a reference picture list and / or reference picture information. i. In one example, for list X, a first error function (eg, SAD / SSE) may be used, and for list Y (Y=1-X), a second error function (eg, MR-SAD / MR-SSE) may be used. ii. Alternatively, additionally, the final cost may be based on the cost of each reference picture list. 5. The above method can be applied to any codec data refinement process, for example, for blocks based on template codecs, for blocks based on bilateral codecs (eg, DMVR in VVC). 2.16. Samples used in codec data refinement for image / video codecs The term "block" may refer to a codec tree block (CTB), a codec tree unit (CTU), a codec block (CB), a CU, a PU, a TU, a PB, a TB, or a video processing unit including a plurality of samples / pixels. A block may be rectangular or non-rectangular. In the present disclosure, the phrase "motion candidate" may refer to a merged motion candidate in a regular / extended merge list indicated by a merge candidate index, or an AMVP motion candidate in a regular / extended AMVP list indicated by an AMVP candidate index, or an AMVP motion candidate, or a merged motion candidate. In this disclosure, a motion candidate is referred to as "refined" if its motion information is modified according to information signaled from an encoder or derived at a decoder. For example, a motion vector may be refined by DMVR, FRUC, TM Merge, TM AMVP, TM GPM, TM CIIP, TM Affine, MMVD, GMVD, Affine MMVD, BDOF, etc. In the present disclosure, the phrase "codec data refinement" may refer to a refinement process to derive or refine a signaled / decoded / derived prediction mode, prediction direction, or signaled / decoded / derived motion information, prediction, and / or reconstructed samples for a block. In one example, the refinement process may include motion candidate reordering. In the following discussion, a "template-based" block may refer to a block that uses a template matching-based method to derive or refine the coded information during the encoding / decoding process, such as template matching-based motion derivation, template matching-based motion list reconstruction, LIC, symbol prediction, template matching-based block vector (e.g., used in IBC mode) derivation, DIMD, template matching-based non-inter (e.g., intra) prediction, etc. The template matching-based method may be combined with any other codec tools, such as MMVD, CIIP, GMP, FRUC, Affine, BDOF, DMVR, OBMC, etc. In another example, a "template-based" block may also refer to a block that uses (adjacent or non-adjacent) neighboring reconstructed samples based on specific rules to derive or refine its decoded information, such as the DIMD method in 2.27 and the TIMD method in 2.29. In the following discussion, a "bilateral codec-based" block may refer to a block that uses a bilateral matching-based method to derive or refine the coded information during the encoding / decoding process, such as bilateral matching-based motion derivation, bilateral matching-based motion list reconstruction, etc. The bilateral codec-based method can be combined with any other codec tools, such as MMVD, CIIP, GPM, FRUC, Affine, DMVR, etc. W and H are the width and height of the current block (eg, luma block). W*H is the size of the current block (eg, luma block). 1. The error / cost estimation in the codec data refinement process may depend on both the reference samples corresponding to the current block (eg, the reference block used in bilateral matching) and the reference samples corresponding to the template of the current block. a) Alternatively, it may depend on both the reference samples corresponding to the current block and the samples in the template of the current block. b) In one example, the template may be neighboring samples (adjacent or non-adjacent) of the current block. 2. Multiple refinement processes may be applied to one block with different templates applied to at least two refinement processes. a) In one example, a first refinement process may involve a first template. Based on the output of the first refinement process, a second template is further utilized in a second refinement process. b) In one example, the first template may contain more samples than the second template. c) In one example, the first template and the second template may contain at least one different sample. d) In one example, the first refinement process and the second refinement process may use different cost / error functions. 3. Whether to use multiple refinement processes, and / or how to select one or more error functions and / or how to define the cost function and / or which samples are to be involved in the error function may depend on the decoding information of the current block and / or the decoding information of neighboring blocks (adjacent or non-adjacent). a) In one example, how to select one or more error functions and / or how to define the cost function may depend on the codec tools applied to the current block and / or neighboring blocks. i. In one example, the codec is LIC. (i) In one example, SSD and mean removed SSD (MR-SSD) may be selectively used according to the LIC flag of the current block. a) In one example, if the LIC flag of the current block is false, the template matching cost measure may be SSD. b) In one example, if the LIC flag of the current block is true, the template matching cost measure may be MR-SSD. (ii) In one example, if MR-SAD / MR-SATD / MR-SSD is used for the template matching cost measurement, the linear function used in the LIC process when deriving the reference template may not be used. (iii) In one example, if MR-SAD / MR-SATD / MR-SSD is used for the bilateral matching cost measurement, the linear function used in the LIC process when deriving the reference template may not be used. b) In one example, it may depend on block size (e.g., W, H), temporal layer, low latency check flag, etc. c) In one example, it may depend on whether the motion information of the current block and the neighboring blocks are similar / identical. d) In one example, it may depend on a reference picture list and / or reference picture information. i. In one example, for list X, a first error function (e.g., SAD / SSE) may be used, and for list Y (Y=1-X), a second error function (e.g., MR-SAD / MR-SSE) Can be used. ii. Alternatively or additionally, the final cost may be based on the cost of each reference picture list. 4. In one example, LIC may be enabled for reference list X and disabled for reference list Y. a) In one example, the final prediction of the current block may be a weighted average of the LIC prediction from reference list X and the regular prediction from reference list Y. 5. The above method can be applied to any codec data refinement process, for example, for blocks based on template codecs, for blocks based on bilateral codecs (eg, DMVR in VVC). 2.17. Adaptive Motion Candidate List It should be noted that the terms mentioned below are not limited to the specific terms defined in existing standards. Any variation of the codec tool is also applicable. For example, the term "GPM" is used to represent any codec tool that derives two sets of motion information and uses the derived information and the partitioning mode to obtain the final prediction. For example, TPM is also considered to be GPM. Note that the proposed method can be applied to the merge candidate list construction process for inter-frame codec blocks (e.g., transform motion), affine codec blocks, TM codec blocks, or IBC codec blocks, or other motion candidate list construction processes (e.g., ordinary AMVP list, affine AMVP list, IBC AMVP list, HMVP table). Cost functions other than template matching cost are also applicable for motion candidate re-ranking. W and H are the width and height of the current block (eg, luma block). 1. The template / bilateral matching cost C can be calculated as f(C) before being used to compare with another template matching cost. a. In one example, f(C) = w*C, where w is represented as the cost factor. b. In one example, f(C) = w*C + u. c. In one example, f(C)=Shift((w*C),s). d. In one example, w and / or u and / or s are integers. e. In one example, a first template matching cost for a first motion candidate may be multiplied by a cost factor before being compared to a second template matching cost for a second motion candidate. f. In one example, it is proposed that the cost factor for a motion candidate may depend on the position of the candidate before re-ranking. i. In one example, a motion candidate at the i-th position may be assigned a smaller cost factor than a motion candidate at the j-th position, where j>i, eg, j=i+1. (i) In one example, the cost factor of the motion candidate at the i-th position is 4, and the cost factor of the motion candidate at the j-th position is 5. (ii) In one example, the cost factor of the motion candidate at the i-th position is 1, and the cost factor of the motion candidate at the j-th position is 5. ii. In one example, a motion candidate at the i-th position may be assigned a cost factor greater than a cost factor of a motion candidate at the j-th position, where j>i, eg, j=i+1. iii. In one example, motion candidates in the pth group (eg, including M motion candidates) may be assigned a smaller cost factor than a cost factor of motion candidates in the qth group (eg, including N motion candidates), where q>p, eg, q=p+1. (i) Alternatively, motion candidates in the pth group (eg, including M motion candidates) may be assigned a cost factor greater than a cost factor of motion candidates in the qth group (eg, including N motion candidates), where q>p, eg, q=p+1. (ii) In one example, M may be equal to N. For example, M=N=2. (iii) In one example, M may not be equal to N. For example, M=2, N=3. (iv) In one example, the cost factor of the motion candidates of the pth group is 4, and the cost factor of the motion candidates of the qth group is 5. (v) In one example, the cost factor of the motion candidates of the pth group is 1, and the cost factor of the motion candidates of the qth group is 5. iv. In one example, the cost factor may not be applied to sub-block motion candidates. v. In one example, the cost factor may not be applied to affine motion candidates. vi. In one example, the cost factor may not be applied to SbTMVP motion candidates. g. In one example, the cost factor for motion candidates in one group / position may be adaptive. i. In one example, the cost factor of a motion candidate in a group / position may depend on the codec mode of the neighboring codec blocks. (i) In one example, the cost factor of the SbTMVP merge candidate may depend on the number of neighboring affine codec blocks. (ii) In one example, the adjacent codec block may include five spatial adjacent blocks (e.g. Figure 4 ) and / or (multiple) temporal neighboring blocks (such as Figure 7 at least one of the above (shown). (iii) In one example, when the spatially adjacent affine codec block (e.g. Figure 4 When the number of affine codec blocks (such as Figure 4) is 1, the cost factor of the SbTMVP merge candidate can be 0.5; when the spatial domain adjacent affine codec block (such as Figure 4 ) is 2, the cost factor of the SbTMVP merge candidate may be 0.8; otherwise, the cost factor of the SbTMVP merge candidate may be 1 (which means remains unchanged). (iv) In one example, when the spatially adjacent affine codec block ( Figure 4 When the number of affine codec blocks (shown in FIG. 1 ) is 0, the cost factor of the SbTMVP merge candidate can be 0.2; when the number of spatially adjacent affine codec blocks ( Figure 4 When the number of affine codec blocks (shown) is 1, the cost factor of the SbTMVP merge candidate can be 0.5; when the spatial domain adjacent affine codec block ( Figure 4 When the number of SbTMVP merge candidates is greater than or equal to 2, the cost factor of the SbTMVP merge candidate can be 0.8. (v) In one example, when the spatially adjacent affine codec block (e.g. Figure 4 When the number of affine codec blocks (such as Figure 4 ) is 1, the cost factor of the SbTMVP merge candidate can be 5; when the spatial domain adjacent affine codec block (such as Figure 4 ) is 2, the cost factor of the SbTMVP merge candidate may be 8; otherwise, the cost factor of the SbTMVP merge candidate may be 10 (wherein the cost factor of the affine merge candidate is 10). (vi) In one example, when the spatially adjacent affine codec block (e.g. Figure 4 When the number of affine codec blocks (such as Figure 4 ) is 1, the cost factor of the SbTMVP merge candidate can be 5; when the spatial domain adjacent affine codec block (such as Figure 4 When the number of merging candidates (as shown) is greater than or equal to 2, the cost factor of the SbTMVP merge candidate can be 8 (where the cost factor of the affine merge candidate is 10). 2. The subgroup sizes of different codec modes can be different. a. The coding mode may include normal / sub-block / TM merge mode. i. For normal merge mode, the subgroup size may be K1 (eg, K1=5). ii. For the sub-block merging mode, the sub-group size may be K2 (eg, K2=3). iii. For TM merging mode, the subgroup size may be K3 (eg, K3=3). b. The subgroup size may be greater than or equal to the maximum number of subblock merging candidates defined in the sps / picture / slice header for the subblock merging mode (which means reordering the whole list together). c. The subgroup size may be greater than or equal to the maximum number of TM merge candidates defined in the sps / picture / slice header for TM merge mode (which means reordering the whole list together). d. The subgroup size for a codec mode may depend on the maximum number of motion candidates in the codec mode. e. The subgroup size for the sub-block merging mode can be adaptively dependent on the number of neighboring affine coded blocks. i. In one example, the adjacent codec block may include five spatial adjacent blocks (such as Figure 4 ) and / or (multiple) temporal neighboring blocks (such as Figure 7 at least one of the above (shown). ii. In one example, when the spatial domain is adjacent to the affine codec block (such as Figure 4 ) is 0 or 1, the subgroup size for the subblock merging mode can be 3; when the spatial domain is adjacent to the affine codec block (such as Figure 4 When the number of sub-blocks (as shown) is greater than 1, the sub-group size of the sub-block merging mode may be 5; 3. The template size can be different for different codec modes. a. The coding mode may include normal / sub-block / TM merge mode. i. For the normal merge mode, the template size may be K1 (eg, K1=1). ii. For the sub-block merging mode, the template size may be K2 (eg, K2=1, 2, or 4). iii. For TM merge mode, the template size may be K3 (eg, K3=1). 4. Whether and / or how to reorder motion candidates may depend on the coding modes of neighboring coded blocks. a. In one example, the adjacent coded blocks may include five spatially adjacent blocks (eg Figure 4 ) and / or (multiple) temporal neighboring blocks (such as Figure 7 at least one of the above (shown). b. When using the spatially adjacent codec blocks of the conventional merge mode (such as Figure 4 When the number of merging candidates (as shown) is greater than or equal to K (eg, K=1), the conventional merge candidates may be reordered. c. When using the spatial domain adjacent codec block of the sub-block merging mode (such as Figure 4 ), when the number of sub-block merging candidates is greater than or equal to K (eg, K=1), the sub-block merging candidates may be reordered. d. When using the spatially adjacent codec blocks of the affine merge mode (such as Figure 4 ) is greater than or equal to K (eg, K=1), the affine merge candidates may be reordered. e. When using the affine merge mode, the spatial domain of the adjacent codec blocks (such as Figure 4 When the number of SbTMVP merge candidates (as shown) is greater than or equal to K (eg, K=1, 2, or 3), the SbTMVP merge candidates may be reordered. f. When using the spatial domain adjacent codec block of the TM merge mode (such as Figure 4 When the number of TM merge candidates (as shown) is greater than or equal to K (eg, K=1), the TM merge candidates may be reordered. 5. HMVP motion candidates in the HMVP table can be reordered based on template / bilateral matching, etc. a. In one example, costs are assigned to HMVP motion candidates, and the HMVP candidates are adaptively reordered in descending order of HMVP candidate cost. i. In one example, the cost of the HMVP candidate may be a template matching cost. b. In one example, HMVP motion candidates can be reordered before encoding and decoding blocks. i. In one example, HMVP motion candidates may be reordered before encoding inter-coded blocks. c. In one example, HMVP motion candidates may be reordered in different ways according to the coding information of the current block and / or neighboring blocks. General requirements 6. Whether and / or how to apply the method disclosed above can be transmitted by a signal at the sequence level / picture group level / picture level / slice level / tile group level, such as in a sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / slice header / tile group header. 7. Whether and / or how to apply the methods disclosed above may be signaled at PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU row / slice / tile / sub-picture / other kind of region containing more than one sample or pixel. 8. Whether and / or how to apply the methods disclosed above may depend on the coded information, such as codec mode, block size, color format, single / dual tree partitioning, color component, slice / picture type. 2.18. Adaptive GPM Candidate List The term "block" may refer to a codec tree block (CTB), a codec tree unit (CTU), a codec block (CB), a CU, a PU, a TU, a PB, a TB, or a video processing unit including a plurality of samples / pixels. A block may be rectangular or non-rectangular. It should be noted that the terms mentioned below are not limited to specific terms defined in existing standards. Any changes in codecs also apply. For example, the term "GPM" is used to represent any codec that derives two or more sets of motion information and uses the derived motion information and partitioning patterns / weighted masks to obtain the final prediction. For example, TPM is also considered a GPM. Note that the proposed method can be applied to the merge candidate list construction process for inter-frame codec blocks (e.g., transform motion), affine codec blocks, TM codec blocks, GPM codec blocks or IBC codec blocks, or other motion candidate list construction processes (e.g., ordinary AMVP list; affine AMVP list; IBC AMVP list; HMVP table). Cost functions other than template matching cost are also applicable for motion candidate re-ranking. In the following, a template is a set of adjacent or non-adjacent reconstructed / predicted samples of the current block. The reference samples of the template (i.e., the reference template) are the mapping of the template in the reference picture, which depends on the motion information of the current block. "Template above" means a template constructed by a set of adjacent or non-adjacent reconstructed / predicted samples above the current block and its reference template. "Template on the left" means a template constructed by a set of adjacent or non-adjacent reconstructed / predicted samples on the left side of the current block and its reference template. "Template above and on the left" includes the template above and the template on the left. In the following, in one example, a GPM candidate list is referred to as OGPMList, where the GPM candidates are directly derived from a rule merge list (before or without motion refinement based on template matching); a refined GPM candidate list is referred to as AGPMList, where the GPM candidates are refined by a first refinement method (e.g., template matching using an upper template); a refined GPM candidate list is referred to as LGPMList, where the GPM candidates are refined by a second refinement method (e.g., template matching using a left template); and a refined GPM candidate list is referred to as LAGPMList, where the GPM candidates are refined by a third refinement method (e.g., template matching using left and upper templates). W and H are the width and height of the current block (eg, luma block). 1. It is proposed that for a GPM-coded block, a decoded candidate index may correspond to a candidate with a different candidate index in a candidate list for the GPM-coded block. a. Alternatively, additionally, the candidate list constructed for the GPM-coded block may be reordered before being used, and the coded index corresponds to the reordered candidate list. b. Alternatively, additionally, for the first type of GPM-coded blocks, the candidate list may be reordered, while for the second type of GPM-coded blocks, the candidate list may not be reordered. i. In one example, the first type is a template-based GPM-encoded block. ii. In one example, the second type is a GPM-coded block based on MMVD (eg, GMVD). c. Alternatively, additionally, for a first type of GPM-coded blocks, the candidate list may be reordered using a first rule, and for a second type of GPM-coded blocks, the candidate list may be reordered using a second rule. d. In one example, the reordering method for GPM-encoded blocks may be the same as the reordering method for non-GPM-encoded blocks. i. The reordering method for GPM-coded blocks may be different from the reordering method for non-GPM-coded blocks. 2. It is proposed that for a GPM-coded block, the decoded candidate index may correspond to a candidate from a refined candidate list for the GPM-coded block. a. Alternatively, additionally, the candidate list constructed for the GPM-coded block may first be refined before being used, and the decoded index corresponds to the refined candidate list. b. Alternatively, additionally, for the first type of GPM-coded blocks, the candidate list may be refined, while for the second type of GPM-coded blocks, the candidate list may not be refined. i. In one example, the first type is a template-based GPM-encoded block. ii. In one example, the second type is a GPM-coded block based on MMVD (eg, GMVD). c. Alternatively, additionally, for a first type of GPM-coded blocks, the candidate list may be refined with a first rule, and for a second type of GPM-coded blocks, the candidate list may be refined with a second rule. d. In one example, the refinement method for a block decoded with GPM can be the same as that for a block not decoded with GPM. The block refinement method for GPM codec is the same. i. The refinement method for GPM-coded blocks may be different from the refinement method for non-GPM-coded blocks. 3. In one example, the GPM candidates may be divided into sub-groups, and whether and / or how to reorder the GPM candidates may depend on the sub-groups of the GPM candidates. a. In one example, only the first subgroup can be reordered. b. In one example, the last subgroup cannot be reordered. c. In one example, the last subgroup cannot be reordered. However, if the last subgroup is also the first subgroup, it can be reordered. d. Different subgroups can be reordered individually. e. Two candidates in different subgroups cannot be compared and / or reordered. f. If the first subgroup precedes the second subgroup, then after reordering, the first candidate in the first subgroup must be placed before the second candidate in the second subgroup. 4. In one example, GPM candidates that are not included in the reordering process may be processed in a specified manner. a. In one example, for candidates that do not need to be reordered, they are arranged in the merge candidate list according to the original order. b. In one example, the candidates that are not to be reordered may be placed after the candidates that are to be reordered. c. In one example, candidates that are not to be reordered may be placed before candidates that are to be reordered. 5. The GPM candidate list to be reordered can be referenced Case 1: The first candidate lists prepared for two GPM partitions are used to derive separate GPM candidate lists for each GPM partition. Case 2: Second GPM candidate list used for each GPM partition. Usually the second GPM candidates are derived from the first candidate list. a. In one example, in case 1, the reordering method may be the same as the reordering method used for a conventional merge candidate list. b. In one example, in case 1, if the corresponding candidate is bidirectionally predicted, the template matching method in the reordering method can be performed in a bidirectionally predicted manner. c. In one example, in case 2, the template matching method in the reordering method cannot be performed in a bidirectional predictive manner. d. In one example, in case 2, the reordering method can be the same for all GPM partitions. e. In one example, in case 2, the reordering method may be different for different GPM partitions. 6. In the above example, the GPM-encoded block may be a GPM-encoded block with a merge mode or a GPM-encoded block with an AMVP mode. a. Alternatively, additionally, the above merge candidates may be replaced by AMVP candidates. General requirements 7. Whether and / or how to apply the method disclosed above can be transmitted by a signal at the sequence level / picture group level / picture level / slice level / tile group level, such as in a sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / slice header / tile group header. 8. Whether and / or how to apply the methods disclosed above can be transmitted by signals in PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU row / slice / tile / sub-picture / other kinds of regions containing more than one sample or pixel. 9. Whether and / or how to apply the method disclosed above may depend on the coded information, such as codec mode, block size, GPM partition information, color format, single / dual tree partition, color component, slice / picture type. 2.19. Adaptive GPM Candidate List The term "block" may refer to a codec tree block (CTB), a codec tree unit (CTU), a codec block (CB), a CU, a PU, a TU, a PB, a TB, or a video processing unit including a plurality of samples / pixels. A block may be rectangular or non-rectangular. It should be noted that the terms mentioned below are not limited to specific terms defined in existing standards. Any changes in codecs also apply. For example, the term "GPM" is used to represent any codec that derives two or more sets of motion information and uses the derived motion information and partitioning mode / weighting mask to obtain the final prediction. For example, TPM is also considered a GPM. Note that the proposed method can be applied to the merge candidate list construction process for inter-coded blocks (e.g., translational motion), affine-coded blocks, TM-coded blocks, GPM-coded blocks, or IBC-coded blocks; or other motion candidate list construction processes (e.g., ordinary AMVP list; affine AMVP list; IBC AMVP list; HMVP table). Cost functions other than template matching cost are also applicable for motion candidate re-ranking. In the following, a template is a set of adjacent or non-adjacent reconstructed / predicted samples adjacent to the current block. The reference samples of the template (i.e., the reference template) are the mapping of the template in the reference picture, which depends on the motion information of the current block. "Template above" means a template constructed by a set of adjacent or non-adjacent reconstructed / predicted samples adjacent to the current block and its reference template above. "Template on the left" means a template constructed by a set of adjacent or non-adjacent reconstructed / predicted samples adjacent to the left of the current block and its reference template. "Template above and left" includes the template above and the template on the left. In the following, in one example, a GPM candidate list is referred to as OGPMList, where the GPM candidates are directly derived from a rule merge list (before or without motion refinement based on template matching); a refined GPM candidate list is referred to as AGPMList, where the GPM candidates are refined by a first refinement method (e.g., template matching using an upper template); a refined GPM candidate list is referred to as LGPMList, where the GPM candidates are refined by a second refinement method (e.g., template matching using a left template); and a refined GPM candidate list is referred to as LAGPMList, where the GPM candidates are refined by a third refinement method (e.g., template matching using left and upper templates). Regarding the types of GPM candidates in the original GPM candidate list, the GPM candidates derived in the first step of the GPM candidate list construction process in Section 2.29 are called GPM parity-based candidates; the GPM candidates derived in the second step of the GPM candidate list construction process in Section 2.29 are called GPM anti-parity-based candidates; and the GPM candidates derived in the third step of the GPM candidate list construction process in Section 2.29 are called filled GPM candidates. W and H are the width and height of the current block (eg, luma block). 1. In one example, if the codec mode is GPM, the merge candidates may be reordered. a. In one example, merge candidates in OGPMList may be reordered. i. In one example, at least two merge candidates in the OGPMList may be reordered. ii. In one example, at least one type of template may be used for OGPMList reordering. iii. Alternatively, merge candidates in OGPMList cannot be reordered. iv. In one example, the first type of template may include only left neighboring samples of the current block. v. In one example, the second type of template may include only the upper neighboring samples of the current block. vi. In one example, the third type of template may include neighboring samples above and to the left of the current block. vii. The reordering process may be called after the parsing process but before the MV reconstruction process. b. In one example, the merge candidates in the AGPMList may be reordered. i. In one example, at least two merge candidates in the AGPMList may be reordered. ii. In one example, at least one type of template may be used for AGPMList reordering. iii. In one example, the first type of template may include only neighboring samples above the current block. iv. In one example, the second type of template may include neighboring samples to the left and above the current block. c. In one example, the merge candidates in LGPMList may be reordered. i. In one example, at least two merge candidates in the LGPMList may be reordered. ii. In one example, at least one type of template may be used for LGPMList reordering. iii. In one example, the first type of template may include only neighboring samples to the left of the current block. iv. In one example, the second type of template may include neighboring samples to the left and above the current block. d. In one example, the merge candidates in LAGPMList may be reordered. i. In one example, at least two merge candidates in the LAGPMList may be reordered. ii. In one example, at least one type of template may be used for LAGPMList reordering. iii. In one example, the first type of template may include only neighboring samples to the left of the current block. iv. In one example, the second type of template may include only neighboring samples above the current block. v. In one example, the third type of template may include neighboring samples to the left and above the current block. e. In one example, whether and / or how to reorder merge candidates in the GPM list may depend on codec information. i. In one example, whether to reorder merge candidates in the GPM list may depend on whether template matching based motion refinement is applied to one GPM partition or two GPM partitions (ie, GPM-encoded CUs). (i) For example, if the motion of one GPM partition or two GPM partitions (ie, GPM-encoded CUs) is not refined based on template matching (eg, template matching flag equals false), the corresponding GPM lists may not be reordered. a) For example, if the GPM partition is encoded using the merge candidates in OGPMList (eg, no motion refinement is applied), the merge candidates in OGPMList may not be reordered. (ii) For example, if the motion of one or both GPM partitions (ie, GPM-encoded CUs) is refined based on template matching (eg, template matching flag equals true), the corresponding GPM lists may be reordered. a) For example, if the GPM partition is encoded using the merge candidates in the AGPMList (eg, a template matching motion refinement method using an upper template is applied), the merge candidates in the AGPMList may be reordered. b) For example, if the GPM partition is encoded using the merge candidates in the LGPMList (eg, a template matching motion refinement method using the left template is applied), the merge candidates in the LGPMList may be reordered. c) For example, if the GPM partition is encoded using the merge candidates in the LAGPMList (eg, a template matching motion refinement method using the left template and the upper template is applied), the merge candidates in the LAGPMList may be reordered. ii. In one example, how to reorder the merge candidates in the GPM list may depend on the GPM partition information (eg, partition mode, partition angle, partition distance, etc.). (i) For example, the above template may be used for merging candidate reordering when the current GPM partition is partitioned by a first partition angle (or partition mode, or partition distance, etc.). (ii) For example, the left template may be used for merging candidate reordering when the current GPM partition is partitioned by a second partition angle (or partition mode, or partition distance, etc.). (iii) For example, the left template and the upper template may be used for merging candidate reordering when the current GPM partition is partitioned by a third partition angle (or partition mode, or partition distance, etc.). (iv) For example, a template type corresponding to the first / second / third division angle (or division mode, or division distance, etc.) may be specified. (v) For example, at least one lookup table (i.e., mapping table) is used to map what specified division angle (or division mode, or division distance, etc.) corresponds to what type of template (e.g., upper template, left template, or upper and left templates). f. In one example, merge candidates in OGPMList may not be reordered, and merge candidates in AGPMList and / or LGPMList and / or LAGPMList may be reordered. 2. Merge candidates can be adaptively rearranged in the final GPM candidate list according to one or some criteria. a. In one example, the GPM candidate list can be i.OGPMList, ii.AGPMList, iii.LGPMLIst, iv.LAGPMList. b. GPM candidates can be divided into several subgroups. i. For example, the number of GPM candidates in a sub-group (such as X=3 or 5 or any other integer value) may be predefined. c. In one example, part or all of the current GPM candidate list construction process is called first, followed by a reordering of the candidates in the GPM list. i. Alternatively, the candidates in the first subgroup may be reordered and they should be added before those in the second subgroup, where the first subgroup is added before the second subgroup. ii. The construction process may include pruning methods. d. In one example, merge candidates may be adaptively rearranged before retrieval. i. In one example, the process of ranking merge candidates can be adaptively processed before obtaining merge candidates to be used in the motion compensation process. e. The criterion can be based on template matching cost. i. In one example, the cost function between the current template and the reference template can be (i) SAD / MR-SAD, (ii) SATD / MR-SATD, (iii) SSD / MR-SSD, (iv) SSE / MR-SSE, (v) Weighted SAD / Weighted MR-SAD, (vi) Weighted SATD / weighted MR-SATD, (vii) Weighted SSD / Weighted MR-SSD, (viii) Weighted SSE / weighted MR-SSE, (ix) Gradient information. 3. When deriving two motions of two geometric partitions, the process can be: a. In one example, if TM is not applied to a partition, the motion may be derived based on the signaled merge index from OGPMList / reordered OGPMList. b. In one example, if TM is applied to one partition, motion can be derived based on the signaled merge index from AGPMList / reordered AGPMList or LGPMList / reordered LGPMLIst or LAGPMList / reordered LAGPMLIst, depending on the partition angle and partition index. i. In one example, if the partition angle is X (eg, 0), for the first partition, AGPMList / reordered AGPMList will be used; for the second partition, LAGPMList / reordered LAGPMLIst will be used. c. In one example, if TM is applied to one partition, the motion can be derived based on the signaled merge index from the AGPMList / reordered AGPMList. d. In one example, if TM is applied to a partition, the motion can be derived based on the signaled merge index from LGPMList / reordered LGPMLIst. e. In one example, if TM is applied to a partition, the motion can be derived based on the signaled merge index from LAGPMList / reordered LAGPMLIst. 4. Whether and / or how to reorder GPM candidates may depend on the category of the GPM candidates. a. In one example, only candidates based on GPM parity may be reordered. b. In one example, only GPM parity-based and GPM anti-parity-based candidates may be reordered. c. In one example, filled GPM candidates may not be reordered. d. In one example, two candidates in different GPM lists cannot be compared and / or reordered. e. In one example, only the top N GPM candidates may be reordered. i. In one example, N is set equal to 5. 5. In the above example, the GPM-encoded block may be a GPM-encoded block with a merge mode or a GPM-encoded block with an AMVP mode. a. Alternatively, additionally, the above merge candidates may be replaced by AMVP candidates. General requirements 6. Whether and / or how to apply the method disclosed above can be transmitted by a signal at the sequence level / picture group level / picture level / slice level / tile group level, such as in a sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / slice header / tile group header. 7. Whether and / or how to apply the methods disclosed above can be transmitted by signals in PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU row / slice / tile / sub-picture / other kinds of regions containing more than one sample or pixel. 8. Whether and / or how to apply the method disclosed above may depend on the coded information, such as codec mode, block size, GPM partition information, color format, single / dual tree partition, color component, slice / picture type. 2.20. Hash-based motion estimation for screen content encoding and decoding The VTM reference software uses hash-based motion estimation to handle the sometimes large and irregular motion in screen content. For each reference picture, a hash table corresponding to 4x4 to 64x64 block sizes is generated using a bottom-up approach as follows: – For each 2x2 block, a block hash value is calculated directly from the original sample values ​​(luminance samples are used if 4:2:0 chroma format, and both luma and chroma sample values ​​are used if 4:4:4 chroma format). A cyclic redundancy check (CRC) value is used as the hash value. – For 4x4, 8x8, 16x16, 32x32, and 64x64 blocks, the hash value of the current block is the CRC value calculated based on the CRC values ​​of its four sub-blocks. To enable efficient search for matching blocks, an inverted index structure is used, where hash values ​​are used as indexes into the table and the table entries contain all blocks with the same hash value as the corresponding table index. Blocks corresponding to a given table index are stored as a linked list. Two CRC values ​​are calculated for each block, one 16-bit hash and the other 24-bit hash. The two hash values ​​are calculated in a similar manner but using different CRC truncation polynomials. The first 16-bit CRC value is used as an inverted index. In the event that more than one matching block is found, the second 24-bit hash value is stored with the block to resolve hash conflicts. In order to reduce the length of the hash table, the hash values ​​of all "simple" blocks (defined as blocks with only one sample value in each row or column) are excluded from the hash table. In motion estimation, if the current block is a square block (except for 128x128 blocks), its hash value is calculated. Then, the encoder queries the corresponding hash table. If a hash match is found, the matching block is used as a reference. If the current block is a rectangular block of size NxM (and without loss of generality assume M>N), it will be divided into several non-overlapping square sub-blocks of size NxN. Fig. 27 An example diagram 2700 showing motion estimation of a rectangular block with hash values ​​of square sub-blocks is shown. Fig. 27 The encoder will find the first non-simple square sub-block and calculate its hash value. The encoder looks up the hash value of this NxN square sub-block on the hash table corresponding to the NxN block size. One or more matching reference blocks are considered as reference block candidates. For each matching reference block candidate, the encoder will continue to check the remaining square sub-blocks (i.e., in Fig. 27The white area after the first non-simple square sub-block depicted in is equal to the hash value of the square sub-blocks adjacent to the reference block candidate. If the hash values ​​of all square sub-blocks match, the reference block candidate is considered a valid reference block. For inter codecs, a hash-based motion search is performed before testing all codec modes. Additionally, the encoder will reuse the MV of the hash mode as a starting point candidate in the normal motion estimation process. If there is a hash-based motion vector that indicates that the block most likely contains screen content, then fractional motion estimation is skipped. To speed up the encoder, codec modes other than the skip and merge parts of ETM_MERGE_SKIP, ETM_AFFINE, and ETM_MERGE_GPM modes and finer-grained block partitioning are skipped if all of the following conditions are met: – Current block sizes are 64x64, 128x64, or 64x128. – The same reference block is found in the reference picture. – The QP of the reference picture is not greater than the QP of the current picture. 2.21. Luminance Mapping with Chroma Scaling (LMCS) In VVC, a codec tool called "luminance mapping with chroma scaling (LMCS)" is added as a new processing block before loop filtering. LMCS has two main components: 1) in-loop mapping of luma components based on an adaptive piecewise linear model; 2) for chroma components, luma-dependent chroma residual scaling is applied. Fig.28 An example diagram 2800 illustrating luma mapping with a chroma scaling architecture is shown. Fig.28 shows the LMCS architecture from the decoder perspective. Fig.28 The light blue shaded blocks in represent where processing is applied in the mapped domain; and these include inverse quantization, inverse transform, luma intra prediction, and adding the luma prediction to the luma residual. Fig.28 The unshaded blocks in indicate where processing is applied in the original (i.e., unmapped) domain; and these include loop filters such as deblocking, ALF and SAO, motion compensated prediction, chroma intra prediction, adding chroma prediction to chroma residual, and storing the decoded picture as a reference picture. Fig.28 The light yellow shaded blocks in the are the new LMCS functional blocks, including the forward and reverse mapping of the luma signal, and the luma-dependent chroma scaling process. Like most other tools in VVC, LMCS can be enabled / disabled at the sequence level using the SPS flag. 2.21.1 Brightness Mapping with a Piecewise Linear Model The in-loop mapping of the luma component adjusts the dynamic range of the input signal by redistributing codewords within the dynamic range to improve compression efficiency. Luma mapping uses a forward mapping function FwdMap and a corresponding inverse mapping function InvMap. The FwdMap function is signaled using a piecewise linear model with 16 equal segments. The InvMap function does not need to be signaled but is derived from the FwdMap function. The luminance mapping model is signaled in the adaptive parameter set (APS) syntax structure with aps_params_type set to equal 1 (LMCS_APS). A maximum of 4 LMCS APSs can be used in a coded video sequence. Only 1 LMCS APS can be used for a picture. The luminance mapping model is signaled using a piecewise linear model. The piecewise linear model divides the dynamic range of the input signal into 16 equal parts, and for each part, its linear mapping parameters are represented by the number of codewords assigned to the part. Take 10-bit input as an example. By default, each of the 16 fragments is assigned 64 codewords. The number of codewords transmitted by the signal is used to calculate the scaling factor and adjust the mapping function of the fragment accordingly. At the slice level, the LMCS enable flag is signaled to indicate if Fig.28 Whether the LMCS process shown is applied to the current slice. If LMCS is enabled for the current slice, aps_id is signaled in the slice header to identify the APS that carries the luma mapping parameters. Each i-th fragment, i=0...15 of the FwdMap piecewise linear model is defined by two input pivot points InputPivot[] and two output (mapped) pivot points MappedPivot[]. InputPivot[] and MappedPivot[] are calculated as follows (assuming 10-bit video): 1)OrgCW=64 2) For i = 0:16, InputPivot[i] = i * OrgCW 3) For i = 0:16, MappedPivot[i] is calculated as follows: MappedPivot[0] = 0; for(i=0;i<16;i++) MappedPivot[i+1]=MappedPivot[i]+SignalledCW[i] where SignalledCW[i] is the number of codewords signaled for the i-th segment. like Fig.28As shown, for the inter-frame coded block, motion compensation prediction is performed in the mapping domain. In other words, the motion compensation prediction block Y is calculated based on the reference signal in the DPB. pred Finally, the FwdMap function is applied to map the brightness prediction block in the original domain to the mapping domain, Y′ pred =FwdMap(T pred ). For intra-coded blocks, the FwdMap function is not applied because intra prediction is performed in the mapping domain. Calculate the reconstructed block Y r Finally, the InvMap function is applied to convert the reconstructed brightness value in the mapping domain back to the reconstructed brightness value in the original domain. ). The InvMap function is applicable to intra-frame and inter-frame coded luminance blocks. The brightness mapping process (forward and / or inverse mapping) can be implemented using a lookup table (LUT) or using on-the-fly calculations. If LUTs are used, then FwdMapLUT and InvMapLUT can be pre-calculated and pre-stored for use at the tile group level, and forward and inverse mapping can be simply implemented as FwdMap(Y pred )=FwdMapLUT[Y pred ] and InvMap(Y r )=InvMapLUT[Y r ]. Alternatively, immediate calculation can be used. Take the forward mapping function FwdMap as an example. To find out which fragment a brightness sample belongs to, the sample value is shifted right by 6 bits (equivalent to 16 equal fragments). Then, the linear model parameters of the fragment are retrieved and applied immediately to calculate the mapped brightness value. Let i be the fragment index, a1 and a2 be InputPivot[i] and InputPivot[i+1] respectively, and b1 and b2 be MappedPivot[i] and MappedPivot[i+1] respectively. The calculation of the FwdMap function is as follows: FwdMap(Y pred )=((b2-b1) / (a2-a1))*(Y pred -a1)+b1 The InvMap function can be calculated on the fly in a similar way. Typically, the fragments in the mapping domain are of unequal size, so the most straightforward inverse mapping process will require comparisons in order to find out which fragment the current sample value belongs to. This comparison increases the complexity of the decoder. Therefore, VVC imposes a bitstream constraint on the value of the output pivot point MappedPivot[i] as shown below. Assume that the range of the mapping domain (for 10-bit video, the range is [0,1023]) is divided into 32 equal fragments. If MappedPivot[i] is not a multiple of 32, then MappedPivot[i+1] and MappedPivot[i] cannot belong to the same fragment among the 32 fragments of the same size, that is, MappedPivot[i+1]>>(BitDepth Y -5) must not be equal to MappedPivot[i]>>(BitDepth Y Due to this bitrate constraint, the InvMap function can also be performed using a simple right shift of 5 bits (corresponding to 32 equally sized fragments) to find out which fragment the sample value belongs to. 2.21.2 Luma-Dependent Chroma Residual Scaling Chroma residual scaling is intended to compensate for the interaction between luma signals and their corresponding chroma signals. Whether chroma residual scaling is enabled is also signaled at the slice level. If luma mapping is enabled, an additional flag is emitted to indicate whether luma-dependent chroma residual scaling is enabled. When luma mapping is not used, luma-dependent chroma residual scaling is disabled. Additionally, luma-dependent chroma residual scaling is always disabled for chroma blocks with an area less than or equal to 4. The chroma residual scaling depends on the average of the neighboring luma samples reconstructed on top and / or to the left of the current VPDU. If the current CU is Inter 128x128, Inter 128x64, and Inter 64x128, the chroma residual scaling factor derived for the CU associated with the first VPDU is used for all chroma transform blocks in that CU. Denote avgYr as the average of the neighboring luma samples reconstructed (see Fig.28 ). C ScaleInv The value of is calculated as follows: 1) Based on the InvMap function, find the index Y of the piecewise linear model to which avgYr belongs Idx . 2) C ScaleInv =cScaleInv[Y Idx ], where cScaleInv[] is a 16-segment LUT pre-computed based on the value of SignalledCW[i] and the offset value for the chroma residual scaling process signaled in APS. Unlike brightness mapping performed based on samples, CScaleInv Is a constant for the entire chrominance block. ScaleInv , chroma residual scaling is applied as follows: Encoder side: C ResScale =C Res *C Scale =C Res / C ScaleInv Decoder side: C Res =C ResScale / C Scale =C ResScale *C ScaleInv 2.21.3 Encoder-side LMCS parameter estimation A non-standard reference implementation is provided in the VTM encoder to estimate the LMCS model parameters. Because the VTM anchor handles SDR, HDR PQ and HDR HLG differently, the reference algorithm in VTM13 is designed differently for SDR sequences, HDR PQ sequences and HDR HLG sequences. For SDR sequences and HDR HLG sequences, the encoder algorithm is optimized based on local brightness changes and for the PSNR metric. For HDR PQ sequences, the encoder algorithm is optimized based on brightness values ​​and for the wPSNR (weighted PSNR) metric. 2.22. Merge Mode with MVD (MMVD) In addition to the merge mode, a merge mode with motion vector difference (MMVD) is introduced in VVC, where the implicitly derived motion information is directly used for prediction sample generation of the current CU. The MMVD flag is signaled immediately after the regular merge flag is sent to specify whether the MMVD mode is used for the CU. In MMVD, after a merge candidate is selected, it is further refined by MVD information transmitted by signal. Further information includes a merge candidate flag, an index for specifying the magnitude of motion, and an index for indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the MV basis. The mmvd candidate flag is transmitted by signal to specify which one to use between the first merge candidate and the second merge candidate. The distance index specifies the motion magnitude information and indicates a predefined offset from the starting point. Fig.29 As shown, the offset is added to the horizontal component or the vertical component of the starting MV. The relationship between the distance index and the predefined offset is specified in Table 2-2. Table 2-2. Relationship between distance index and predefined offset The direction index indicates the direction of the MVD relative to the starting point. The direction index can represent four directions as shown in Table 2-3. It should be noted that the meaning of the MVD symbol may change according to the information of the starting MV. When the starting MV is a non-predicted MV or a bidirectionally predicted MV with two lists pointing to the same side of the current picture (i.e., the POCs of both references are greater than the POC of the current picture, or both are less than the POC of the current picture), the symbol in Table 2-3 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectionally predicted MV with two MVs to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), and the POC difference in list 0 is greater than the POC difference in list 1, the symbol in Table 2-3 specifies the sign of the MV offset added to the list 0 MV component of the starting MV, and the sign of the list 1 MV has an opposite value. Otherwise, if the POC difference in list 1 is greater than the POC difference in list 0, then the sign in Table 2-3 specifies the sign of the MV offset added to the list 1 MV component of the starting MV, and the sign of the list 0 MV has the opposite value. The MVD is scaled based on the difference in POC in each direction. If the POC difference in both lists is the same, no scaling is required. Otherwise, if the POC difference in list 0 is greater than the POC difference in list 1, then Figure 6 As described in , the MVD of list 1 is scaled by defining the POC difference of L0 as td and the POC difference of L1 as tb. If the POC difference of L1 is greater than the POC difference of L0, the MVD of list 0 is scaled in the same way. If the starting MV is unidirectionally predicted, the MVD is added to the available MVs. MMVD is also known as Ultimate Motion Vector Expression (UMVE). Table 3. Signs of MV offsets specified by direction index Directions to IDX 00 01 10 11 x-axis + - N / A N / A y-axis N / A N / A + - 2.23. Triangulation for Inter Prediction In VVC, triangulation partition mode (TPM) is supported for inter prediction. Triangulation partition mode is only applicable to CUs of 8x8 or larger. Triangulation partition mode is signaled using a CU level flag as a merge mode. Other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. When using this mode, use diagonal or anti-diagonal splits (such as Fig. 30A and Fig. 30B), the CU is evenly split into two triangular partitions. Each triangular partition in the CU uses its own motion for inter-frame prediction; each partition only allows unidirectional prediction, that is, each partition has one motion vector and one reference index. Unidirectional prediction motion constraints are applied to ensure that only two motion compensated predictions are required for each CU, which is the same as traditional bidirectional prediction. The unidirectional prediction motion of each partition is derived using the process described in 2.23.1. If triangulation mode is used for the current CU, a flag indicating the direction of the triangulation (diagonal or anti-diagonal) and two merge indices (one for each partition) are further signaled. The number of maximum TPM candidate sizes is explicitly signaled at the slice level, and the syntax for specifying the TMP merge indices is binary. After predicting each triangulation, a hybrid process with adaptive weights is used to adjust the sample values ​​along the diagonal or anti-diagonal edges. This is the prediction signal for the entire CU, and the transform and quantization process will be applied to the entire CU as in other prediction modes. Finally, the motion field of the CU predicted using triangulation mode is stored as in 2.23.3. The triangulation mode is not used in conjunction with SBT, i.e., when the signaled triangulation mode is equal to 1, cu_sbt_flag is inferred to be 0 if not signaled. 2.23.1 One-way prediction candidate list construction The unidirectional prediction candidate list is directly derived from the merge candidate list constructed according to the extended merge prediction process in 2.1. Denote n as the index of the unidirectional prediction motion in the triangular unidirectional prediction candidate list. The LX motion vector (x equals the parity of n) of the nth extended merge candidate is used as the nth unidirectional prediction motion vector of the triangular partitioning mode. These motion vectors are Fig.31 If there is no corresponding LX motion vector of the n-th extended merge candidate, the L(1-X) motion vector of the same candidate is used as the unidirectional prediction motion vector of the triangulation mode. 2.23.2 Blending along triangulation edges After each triangulation is predicted using its own motion, blending is applied to the two prediction signals to derive samples around diagonal or anti-diagonal edges. The following weights are used in the blending process: ·like Fig.32 As shown, {7 / 8, 6 / 8, 5 / 8, 4 / 8, 3 / 8, 2 / 8, 1 / 8} for luminance and {6 / 8, 4 / 8 and 2 / 8} for chrominance. 2.23.3 Sports Field Storage Motion vector of the triangle prediction unit ( Fig.33A , Fig.33B and Fig.33CThe Mv1 and Mv2 in the CU are stored in a 4x4 grid. For each 4x4 grid, a unidirectional prediction motion vector or a bidirectional prediction motion vector is stored depending on the position of the 4x4 grid in the CU. In FIG. 33 , a unidirectional prediction motion vector (Mv1 or Mv2) is stored for a 4×4 grid located in a non-weighted area (i.e., not located at a diagonal edge). On the other hand, a bidirectional prediction motion vector is stored for a 4x4 grid located in a weighted area. The bidirectional prediction motion vector is derived from Mv1 and Mv2 according to the following rules: 1) If Mv1 and Mv2 are from different reference picture lists (one from L0 and the other from L1), then Mv1 and Mv2 are simply combined to form a bidirectional prediction motion vector. 2) Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored. 2.23.4 Motion Vector Storage Process for Triangle Merging Mode (Alternative Description) The variables numSbX and numSbY, which specify the number of 4x4 blocks in the current codec block in the horizontal and vertical directions, are set equal to numSbX=cbWidth>>2 and numSbY=cbHeight>>2. Where cbWidth and cbHeight specify the width and height of the current codec block in luma samples. The variable minSb is set equal to min(numSbX, numSbY)-1. The variable cbRatio is derived as follows: cbRatio=(cbWidth>cbHeight)? (cbWidth / cbHeight): (cbHeight / cbWidth) For each 4x4 sub-block at sub-block index (xSbIdx, ySbIdx), xSbIdx=0...numSbX=1, and ySbIdx=0...numSbY=1, the following applies: – The variables xIdx and yIdx are derived as follows: xIdx=(cbWidth>cbHeight)? (xSbIdx / cbRatio): xSbIdx yIdx=(cbWidth>cbHeight)? ySbIdx: (ySbIdx / cbRatio) –The variable sType is exported as follows: – If triangleDir is equal to 0, the following applies: sType=(xIdx==yIdx)? 2: ((xIdx>yIdx)?0:1) – Otherwise (triangleDir is equal to 1), the following applies: sType=(xIdx+yIdx==minSb)? 2: ((xIdx+yIdx <minSb)?0:1) Where triangleDir specifies the division direction. As shown in FIG33 , sType equal to 0 corresponds to the P1 region; sType equal to 1 corresponds to the P2 region; and sType equal to 2 corresponds to the weighted region. The motion information of the P1 region is expressed as (Mv1, refIdx1); the motion information of the P2 region is expressed as (Mv2, refIdx2). – Depending on the value of sType, the following assignments are made: – If sType is equal to 0, the motion information of the 4x4 sub-block is (Mv1, refIdx1). – If sType is equal to 1, the motion information of the 4x4 sub-block is (Mv2, refIdx2). – Otherwise (sType equals 2), the following applies: 1) If Mv1 and Mv2 are from different reference picture lists (one from L0 and the other from L1). Then, simply combine Mv1 and Mv2 to form a bi-directional prediction motion vector. 2) Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored. 2.24. Geometric Partitioning Model (GPM) In VVC, geometric partitioning mode is supported for inter prediction. The CU level flag is used as a merge mode to signal the geometric partitioning mode. Other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. For each possible CU size w×h=2 m ×2 n , where m,n∈{3…6} does not include 8x64 and 64x8, and the geometric partitioning mode supports a total of 64 partitions. When this mode is used, the CU is divided into two parts by a straight line located geometrically ( Fig.34 ). The position of the partition line is mathematically derived from the angle and offset parameters of the specific partition. Each part of the geometric partition in the CU is inter-predicted using its own motion; only unidirectional prediction is allowed for each partition, i.e. each part has a motion vector and a reference index. Unidirectional prediction motion constraints are applied to ensure that only two motion compensated predictions are required for each CU, the same as traditional bidirectional prediction. The unidirectional prediction motion for each partition is derived using the process described in 2.24.1. If a geometric partitioning mode is used for the current CU, a geometric partitioning index (angle and offset) indicating the partitioning mode of the geometric partitioning and two merge indices (one for each partition) are further transmitted through a signal. The number of maximum GPM candidate sizes is explicitly transmitted through a signal in the SPS, and the syntax binary for the GPM merge indices is specified. After predicting each part of the geometric partitioning, as in 2.24.2, a hybrid process with adaptive weights is used to adjust the sample values ​​along the edges of the geometric partitioning. This is the prediction signal for the entire CU, and the transformation process and quantization process will be applied to the entire CU as in other prediction modes. Finally, the motion field of the CU predicted using the geometric partitioning mode is stored, as described in 2.24.3. 2.24.1 One-way prediction candidate list construction In 3.4.1, the unidirectional prediction candidate list is directly derived from the merge candidate list constructed according to the extended merge prediction process. Denote n as the index of the unidirectional prediction motion in the geometric unidirectional prediction candidate list. The LX motion vector (x equals parity of n) of the nth extended merge candidate is used as the nth unidirectional prediction motion vector of the geometric partitioning mode. These motion vectors are Fig.35 If the corresponding LX motion vector of the n-th extended merge candidate does not exist, the L(1-X) motion vector of the same candidate is used as the unidirectional prediction motion vector of the geometric partitioning mode. 2.24.2 Blending Along Geometric Partition Edges After predicting each part of the geometric partition using its own motion, blending is applied to the two prediction signals to derive samples around the geometric partition edges. The blending weights for each position of the CU are derived based on the distance between the independent position and the partition edge. The distance from position (x,y) to the partition edge is derived as: where i,j are the angle and offset indices of the geometric partition, which depend on the geometric partition index transmitted by the signal. x,j and ρ y,j The sign of depends on the angle index i. The weights for each part of the geometric partition are derived as follows: wIdxL(x,y)=partIdx? 32+d(x,y):32-d(x,y) (2-5) w1(x,y)=1-w0(x,y) (2-7) partIdx depends on the angle index i. An example of weight w0 is Fig.36 is shown in . 2.24.3 Motion Field Storage for Geometry Partitioning Mode Mv1 from the first part of the geometric partitioning, Mv2 from the second part of the geometric partitioning, and the combination Mv of Mv1 and Mv2 are stored in the motion field of the CU encoded in the geometric partitioning mode. The type of motion vector stored for each individual position in the motion field is determined as: sType=abs(motionIdx)<32?2:(motionIdx≤0?(1-partIdx):partIdx) (2-8) Where motionIdx is equal to d(4x+2,4y+2), which is recalculated from equation (2-1). partIdx depends on the angle index i. If sType is equal to 0 or 1, then Mv0 or Mv1 is stored in the corresponding motion field, otherwise, if sType is equal to 2, the combined Mv from Mv1 and Mv2 is stored. The combined Mv is generated using the following process: 1) If Mv1 and Mv2 are from different reference picture lists (one from L0 and the other from L1), then Mv1 and Mv2 are simply combined to form a bidirectional prediction motion vector. 2) Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored. 2.25. Combined Inter and Intra Prediction (CIIP) In VVC, when a CU is encoded and decoded in merge mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and if both the CU width and the CU height are less than 128 luma samples, an additional flag is signaled to indicate whether the combined inter / intra prediction (CIIP) mode is applied to the current CU. As its name suggests, CIIP prediction combines the inter prediction signal with the intra prediction signal. The inter prediction signal P in CIIP mode intra is derived using the same inter prediction process applied to the normal merge mode; and the intra prediction signal P intra It is derived after the conventional intra prediction process using planar mode. Then, the intra prediction signal and the inter prediction signal are combined using weighted averaging, where the weight values ​​are calculated depending on the coding mode of the top and left neighboring blocks (e.g. Fig.37 Described in) as follows: – If the top neighbor is available and is intra-coded, set isIntraTop to 1, otherwise set isIntraTop to 0; – If the left neighbor is available and is intra-coded, set isIntraLeft to 1, otherwise set isIntralLeft to 0; – If (isIntraLeft+isIntraTop) is equal to 2, wt is set to 3; – Otherwise, if (isIntraLeft+isIntraTop) is equal to 1, wt is set to 2; – Otherwise, set wt to 1. The CIIP forecast is constructed as follows: P CIIP =((4-wt)*P inter +wt*P intra +2)>>2 2.26. Decoder-side Intra Mode Derivation (DIMD) When DIMD is applied, two intra modes are derived from the reconstructed neighboring samples, and these two predictors are combined with the planar mode predictor, where the weights are derived from the gradients as described in JVET-00449. The derived intra modes are included in the master list of most probable modes (MPM), so the DIMD process is performed before building the MPM list. The main derived intra modes of a DIMD block are stored with the block and are used for MPM list construction for neighboring blocks. IBC campaign candidate proposed in 2.27.P21081069701H The detailed schemes below should be considered as examples to explain the general concept. These schemes should not be interpreted in a narrow manner. In addition, these schemes can be combined in any way. The term "block" may refer to a codec tree block (CTB), a codec tree unit (CTU), a codec block (CB), a CU, a PU, a TU, a PB, a TB, or a video processing unit including multiple samples / pixels. A block may be rectangular or non-rectangular. For IBC-coded blocks, a block vector (BV) is used to indicate the displacement from the current block to a reference block that has been reconstructed inside the current picture. W and H are the width and height of the current block (eg, luma block). The non-adjacent spatial candidates of the current codec block are the adjacent spatial candidates of the virtual block in the i-th search round (such as Fig. 9 As shown in FIG. 1 ). For the i-th search round, the width and height of the virtual block are calculated by the following formulas: newWidth = i×2×gridX+W, newHeight = i×2×gridY+H. Obviously, if the search round i is 0, the virtual block is the current block. In the following, BV predictor is also BV candidate. Skip mode is also merge mode. BV candidates can be divided into several groups according to some criteria. Each group is called a subgroup. For example, we can take adjacent spatial and temporal BV candidates as the first subgroup, and the remaining BV candidates as the second subgroup; in another example, we can take the first N (N≥2) BV candidates as the first subgroup, the last M (M≥2) BV candidates as the second subgroup, and the remaining BV candidates as the third subgroup. About the use of BV candidates 1. If a BV candidate is invalid, the BV candidate (eg, BV search point or BV predictor) is not allowed to be used in the encoding / decoding process of a block. a. In one example, only if a BV candidate is valid, it can be used in the encoding / decoding process of a block. i. For example, only if a BV candidate is valid, it can be used for BV search or BV prediction. b. In one example, whether to use a BV candidate in the encoding / decoding process of a block may depend on a validation check of the BV candidate. i. In one example, before inserting a new BV candidate into the BV candidate list, a validation check of the BV candidate needs to be performed. c. Only if a BV candidate is valid can it be inserted into the IBC candidate list. i. In one example, the IBC candidate list may be an IBC merge candidate list. ii. In one example, the IBC candidate list may be an IBC AMVP candidate list. iii. In one example, the IBC candidate list may be an IBC template matching candidate list. iv. In one example, the IBC candidate list may be an intra template matching candidate list. v. In one example, the IBC candidate list may be an IBC MMVD candidate list. vi. In one example, the IBC candidate list may be an IBC GPM candidate list. vii. In one example, the IBC candidate list may be an IBC TPM candidate list. viii. In one example, the IBC candidate list may be any list related to IBC coded blocks, ie the same process is applied. (i) Alternatively, whether to allow invalid BV candidates to be added to the IBC candidate list may depend on decoded information (eg, IBC mode). d. Only if a BV candidate is valid, it can be used for hash-based search for IBC. e. Only if a BV candidate is valid, it can be used for block matching based local search for IBC. f. Only if the BV candidate is valid, it can be used for intra template matching. g. Alternatively, additionally, the above-mentioned BV candidates may be those from specific neighboring blocks (eg, adjacent or non-adjacent) or the HMVP table or some virtual candidates generated from these BV candidates. h. Alternatively, additionally, the above BV candidates may exclude some default candidates (eg, default zero vector). i. Alternatively, additionally, when a BV candidate is marked as invalid, a virtual candidate derived from the invalid BV candidate may be used instead. i. In one example, virtual candidates may be derived by adding offsets to invalid BV candidates. ii. In one example, virtual candidates may be derived by applying a pruning function to invalid BV candidates. Verification checks on BV candidates 2. In one example, a BV candidate may be determined to be valid when the BV candidate satisfies one or a combination of at least one of the following conditions. a. The corresponding reference block has been reconstructed inside the current image. b. The corresponding reference block is located in the same CTU row as the current block. c. The corresponding reference block is located in the same tile / sub-picture as the current block. d. The corresponding reference block is located in the same stripe as the current block. e. The BV candidate satisfies the block vector constraints (eg, which are described in 2.4.2 and 2.4.3). f. The BV candidate satisfies the IBC virtual buffer conditions (eg, which are described in 2.4.4). 3. In one example, when a BV candidate violates one or a combination of at least one of the conditions in item 2, the BV candidate may be determined to be invalid. About BV Candidate List 4. BV candidates can be derived / obtained from non-neighboring blocks. a. In one example, the distance between the non-adjacent spatial candidate and the current codec block can be based on the width and height of the current codec block (e.g., Fig. 9 or Fig.10 , gridX=W, gridY=H). i. Alternatively, the distance between the non-adjacent spatial candidates and the current coding block may be a multiple of a constant value. (i) For example, the multiplication factor depends on the search round index (eg, the multiplication factor is i for the i-th search round) and gridX=M, gridY=N (M and N are constant values). b. In one example, non-adjacent spatial candidates may be inserted after the TMVP candidate. i. Alternatively, non-adjacent spatial candidates may be inserted after adjacent spatial candidates and before TMVP candidates. 5. BV candidates can come from spatial neighboring blocks, temporal neighboring blocks, HMVP, paired, and / or STMVP candidates. a. In one example, a spatial candidate may consist of adjacent and / or non-adjacent spatial candidates. i. In one example, the adjacent spatial domain candidates may be composed of spatial domain candidates on the left and / or above and / or upper right and / or lower left and / or upper left (example in Fig.38A ). b. In one example, Fig.38B As shown, for TMVP candidates, the position for the temporal candidate is selected between candidates C0 and C1. If the CU at position C0 is unavailable, intra-coded, outside the current row of the CTU, or its BV is invalid for the current block, then position C1 is used. Otherwise, position C0 is used for the derivation of TMVP candidates. i. Alternatively, for a TMVP candidate, such as Fig.38B Both candidates C0 and C1 depicted in can be used. (i) For example, the order is C0->C1. (ii) Alternatively, the order is C1->C0. c. In one example, for paired candidates, pairwise average candidates are generated by averaging predefined pairs of candidates in an existing BV candidate list, and the predefined pairs are defined as {(0,1), (0,2), (1,2), (0,3), (1,3), (2,3)}, where the numbers represent the BV candidate index of the BV candidate list. i. In one example, the number of pairwise candidates is P. P is an integer from 0 to 6. ii. In one example, pairwise candidates may be inserted after HMVP. d. In one example, for the STMVP candidate, it is generated by averaging the predefined E spatial domain BV candidates and the predefined G temporal domain BV candidates. i. In one example, E is less than or equal to the number of spatial candidates (F) inserted into the current BV candidate list before STMVP. ii. In one example, the predefined E spatial domain BV candidates may be the first E spatial domain BV candidates among the F spatial domain candidates inserted into the current BV candidate list before STMVP. (i) Alternatively, the predefined E spatial BV candidates may be the selected E spatial BV candidates from among the F spatial candidates inserted into the current BV candidate list before STMVP. iii. In one example, E is 2 and G is 1. iv. In one example, the STMVP may be inserted before the top left spatial BV candidate. v. In one example, STMVP can be inserted after the pairwise candidates. e. In one example, the BV candidate insertion order is adjacent airspace -> HMVP -> pairs. f. In one example, the BV candidate insertion order is adjacent spatial domain -> temporal domain -> HMVP -> in pairs. g. In one example, the BV candidate insertion order is adjacent spatial domain -> temporal domain -> non-adjacent spatial domain -> HMVP -> in pairs. h. In one example, the BV candidate insertion order is adjacent spatial domains -> non-adjacent spatial domains -> HMVP -> pairs. i. In one example, the BV candidate insertion order is adjacent spatial domain (STMVP is inserted before the upper left spatial domain BV candidate) -> temporal domain -> non-adjacent spatial domain -> HMVP -> paired. 6. The BV candidate list can also consist of pruned BV candidates. a. In one example, if a BV candidate is invalid in the sense of the third item, it can be converted into a valid BV following the given rules and then inserted into the BV candidate list. i. In one example, a BV candidate may be converted to the nearest valid BV candidate. ii. In one example, a BV candidate may be converted to the closest valid BV candidate from a predefined set of BV candidates. b. In one example, if a non-zero BV candidate is invalid, it may be pruned to the nearest valid BV and then inserted into the BV candidate list. c. In one example, if a non-zero BV candidate is invalid, it may be pruned to the closest valid BV from a predefined BV candidate set and then inserted into the BV candidate list. i. In one example, the predefined BV candidate set may consist of (-m*W, 0), (0, -n*H), (-m*H, 0), (0, -n*W). m and n are positive values. d. In one example, the cropped BV candidate may be inserted after the non-cropped BV candidate. 7. The BV candidate list can be used as the IBC merge / AMVP candidate list. a. Alternatively, the BV candidate list can be used to derive the IBC merge / AMVP candidate list. i. In one example, for the IBC merge mode, the first R entries of the BV candidate list will be used to construct the IBC merge candidate list; for the IBC AMVP mode, the first S entries of the BV candidate list will be used to construct the IBC AMVP candidate list. (i) In one example, R is 6 and S is 2. 8. In one example, sub-block based temporal block vector prediction (SbTMVP) may be supported as a BV candidate or BV prediction mode. a. Similar to SbTMVP, SbTMVP uses the BV motion field in the co-located picture to improve the block vector prediction and IBC merge mode for the CU in the current picture. The same co-located picture used by TMVP is used for SbTMVP. b. In one example, SbTMVP applies motion shifting before extracting temporal BV information from the co-located picture, where the motion shifting is obtained from the motion vector of one of the spatial neighboring blocks of the current CU (example in Fig.39 ). i. In one example, if A1 has a motion vector that uses the co-located picture as its reference picture, then this motion vector is selected as the motion shift to be applied. If no such motion is identified, then the motion shift is set to (0,0). ii. In one example, other spatial candidate positions (eg, A0, B0, B1, B2) may be used to derive motion shifts. (i) In one example, the checking order may be A1->B1->B0->A0->B2. (ii) In one example, the checking order may be B1->A1->B0->A0->B2. (iii) In one example, the checking order may be A0->A1->B0->B1->B2. c. In one example, after deriving the motion shift, for each sub-CU, the BV information of its corresponding block (the smallest motion grid covering the center sample) in the co-located picture is used to derive the BV information for the sub-CU ( Fig.39 Example in ) assuming that the motion shift is set to the motion of block A1). About re-ranking of BV candidate list 9. An initial BV candidate list may be derived first, followed by a reordering / refinement process. And the reordering / refinement list is utilized during the encoding / decoding process of the block. a. In one example, the BV candidate list may be an IBC merge candidate list. b. In one example, the BV candidate list may be an IBC AMVP candidate list. c. In one example, the BV candidate list may be an IBC template matching candidate list. d. In one example, the BV candidate list may be an intra template matching candidate list. e. In one example, the BV candidate list may be an IBC MMVD candidate list. f. In one example, the BV candidate list may be an IBC GPM candidate list. g. In one example, the BV candidate list may be an IBC TPM candidate list. h. Alternatively, the re-ranking / refinement process may not be applied to the IBC AMVP candidate list. i. Alternatively, the re-ranking / refinement process may not be applied to the IBC GPM candidate list. j. Alternatively, the re-ordering / refinement process may not be applied to the IBC TPM candidate list. k. In one example, the reordering / refining process may be applied to all types of IBC-coded blocks, ie, the same process is applied. i. Alternatively, whether to apply the reordering / refinement process may depend on decoding information (eg, IBC mode). (i) In one example, it is applied to IBC Merge / Skip mode, but not to IBC AMVP mode. 10. BV candidates can be adaptively rearranged in the final BV candidate list according to one or some criteria. a. In one example, part or all of the current BV candidate list building process is called first, followed by the reordering of the candidates in the list. i. Alternatively, the candidates in the first subgroup may be reordered and they should be added before those in the second subgroup, where the first subgroup is added before the second subgroup. b. In one example, all BV candidates in the BV candidate list may be reordered together (ie, no subgroups). i. Alternatively, some of the BV candidates in the BV candidate list may be reordered together (ie, without subgroups). (i) The BV candidates to be reordered may be selected based on codec information or signaling. c. In one example, the criteria may be based on template matching cost. i. In one example, the cost function between the current template and the reference template can be (i) SAD / MR-SAD (ii) SATD / MR-SATD (iii) SSD / MR-SSD (iv) SSE / MR-SSE (v) Weighted SAD / Weighted MR-SAD (vi) Weighted SATD / Weighted MR-SATD (vii) Weighted SSD / Weighted MR-SSD (viii) Weighted SSE / Weighted MR-SSE (ix) Gradient information ii. In one example, if LMCS is enabled, the current template and the reference template may consist of samples in the mapped domain. (i) Alternatively, the current template and the reference template may consist of samples in the original domain. iii. In one example, the BV candidates in each subgroup may be reordered in ascending order according to the cost value based on template matching. iv. In one example, if only the upper template is valid for the current block, the template matching reordering can only use the upper template. v. In one example, if only the left template is valid for the current block, the template matching reordering can only use the left template. vi. In one example, if both the above template and the left template are available for the current block, the template matching reordering may use the left template, the above template, or both the above and left templates. vii. In one example, the template matching process may include a component, such as brightness. (i) Alternatively, the template matching process may include multiple components, such as luminance and chrominance. a) In one example, the total template matching cost may be calculated as a weighted sum of the template matching costs on different color components. viii. In one example, the width of the left template and / or the height of the upper template may be fixed. (i) In one example, the width of the left template may be 1. (ii) In one example, the height of the upper template may be 1. ix. In one example, the BV used to locate the reference template can be cropped. (i) In one example, if the reference template is outside the current image (e.g. Fig.41A As shown in the figure), the BV used to locate the reference template can be cropped so that the reference template is located inside the current picture. The cropping operation will not change the BV of the corresponding BV candidate. Fig.41B is shown in . x. In one example, if the reference template is outside the current image (such as Fig.41A As shown), the corresponding BV candidates can still be reordered. (i) In one example, if the reference template is outside the current picture, it can be filled by the nearest sample from within the current picture. xi. In one example, the reference template should have been reconstructed inside the current picture. (i) In one example, if the reference template is not reconstructed or is outside the current picture, the corresponding BV candidates may not be reordered. xii. In one example, the samples included in the reference template should be available (eg, reconstructed and within the same stripe / tile / IBC virtual buffer). (i) Alternatively, additionally, if some or all samples contained in the reference template are not available, the above method of processing a reference template outside the current picture may be applied. d. In one example, whether and / or how to reorder BV candidates may depend on the category of the BV candidates. e. In one example, the BV candidates to be reordered may be BV candidates in the final BV candidate list. i. Alternatively, the BV candidate to be reordered may be part / all of the adjacent spatial BV candidates, even though it may not be included in the final BV candidate list. ii. Alternatively, the BV candidate to be reordered may be part / all of the non-adjacent spatial BV candidates, even though it may not be included in the final BV candidate list. iii. Alternatively, the BV candidate to be reordered may be part / all of the HMVP BV candidate, even though it may not be included in the final BV candidate list. iv. Alternatively, the BV candidate to be re-ranked may be part / all of the pairwise average BV candidate, even though it may not be included in the final BV candidate list. v. Alternatively, the BV candidate to be reordered may be a partial / full STMVP BV candidate, even though it may not be included in the final BV candidate list. 11. The template shape can be adaptive. a. In one example, the template may include only neighboring samples to the left of the current block. b. In one example, the template may only include neighboring samples above the current block. c. In one example, the template shape is selected according to the CU shape. d. In one example, the width of the left template is selected according to the CU height. i. For example, if H<=M, the size of the left template is w1xH; otherwise, the size of the left template is w2xH. e. In one example, M, w1, and w2 are set equal to 8, 1, and 2, respectively. f. In one example, the height of the upper template is selected according to the CU width. i. For example, if W<=N, the size of the upper template is Wxh1; otherwise, the size of the upper template is Wxh2. (i) In one example, N, h1, and h2 are set equal to 8, 1, and 2, respectively. g. In one example, the width of the left template is selected according to the CU width. i. For example, if W<=N, the size of the left template is w1xH; otherwise, the size of the left template is w2xH. (i) In one example, N, w1, and w2 are set equal to 8, 1, and 2, respectively. h. In one example, the height of the upper template is selected according to the CU height. i. For example, if H<=M, the size of the upper template is Wxh1; otherwise, the size of the upper template is Wxh2. (i) In one example, M, h1, and h2 are set equal to 8, 1, and 2, respectively. i. In one example, samples of the template and reference samples of the template samples may be downsampled or down-sampled before being used to calculate the cost. i. Whether and / or how downsampling is performed may depend on the CU size. ii. In one example, no downsampling is performed on the short edge of the CU. 12. In one example, the costs disclosed in item 10 may be derived for a first BV candidate, which may or may not be placed in the BV candidate list. a. In one example, whether to place the first BV candidate into the BV candidate list may depend on the cost derived for the first BV candidate. b. In one example, whether to place a first BV candidate into a BV candidate list may depend on a comparison between a first cost derived for the first BV candidate and a second cost derived for a second BV candidate, and the second BV candidate may or may not be placed into the BV candidate list. 13. In one example, for intra-frame TMP, the L-shaped template can be replaced with an upper template and a left template excluding the upper left portion (example in Fig.40 is shown in a. In one example, if only an upper template is available for the current block, the intra TMP can only use the upper template. b. In one example, if only the left template is available for the current block, the intra TMP can only use the left template. c. In one example, if both the above template and the left template are available for the current block, the intra TMP may use the left template, the above template, or both the above template and the left template. 2.28. Regarding the motion vector difference (MVD) sign prediction and extended merge with MVD (MMVD) reordering and MMVD encoding and decoding proposed in P21091135601H The following detailed schemes should be considered as examples to explain the general concept. These schemes should not be interpreted in a narrow manner. In addition, these schemes can be combined in any way. The method disclosed below can be applied to MMVD and extensions of MMVD (e.g., affine MMVD or GPM MMVD (GMVD), MMVD for IBC mode, MMVD for affine IBC mode). In the following description, the term "MMVD" may be used to refer to a codec in which part of the motion information (e.g., reference picture index, prediction direction from list 0 / 1, and basic motion vector) is inherited from the candidate, while some additional refined indications of the refined motion information (e.g., refined mv difference) are further signaled in the bitstream. Extensions on the directions used in MMVD design 1. Oblique / asymmetric direction or diagonal direction can be used for MMVD-coded blocks. a. The diagonal direction can be defined as M*pi / N, where M and N are both non-zero integers, M<N. i. Fig.42A In one example shown with a green square, at least one of the four diagonal positions may be added to the original four horizontal and vertical positions. ii. In Fig.42B In one example shown with green squares and yellow triangles, at least one of the additional 8 directions may be added to the previous 8 directions at an angle k*pi / 8. iii. In Fig.42C In one example shown with green squares and yellow triangles, at least one of the additional 8 directions may be added to the previous 8 directions at an angle k*pi / 8, with an asymmetric distance offset. iv. In one example, at least one of the additional 16 directions may be added to the previous 16 directions at an angle k*pi / 16. v. in such as Fig.43A and Fig.43B In one example shown with green squares and yellow triangles, at least one of the additional 4 or 8 directions may be added to the previous 4 or 8 directions at an angle k*pi / 8, offset with an exactly similar distance around the circle. b. Asymmetric angles and / or asymmetric distances can be utilized in MMVD design. i. An example such as Fig.44 , where the green squares and yellow triangles indicate additional directions. c. The above directions may be added as additional directions to those in the prior art. i. Alternatively, it can be used to replace at least one existing direction defined in the prior art. d. In one example, additional asymmetric / slant offsets or additional diagonal directions for MMVD and / or its extensions (eg, affine MMVD) may be indicated by an index to be jointly or separately encoded for direction and distance offsets. i. In one example, the index may be encoded with a truncated binary / binary code. ii. In one example, the index may all be encoded and decoded using truncated unary codes. iii. In one example, the indices can all be encoded and decoded using Rice codes or k-th exponential Golomb codes, where k can be 0, 1, or any number. (i) In one example, Rice codes with parameters 1, 2, 4, 8 or any other number may be used. iv. In one example, the prefix and suffix of the code can be encoded into a binary value (bin) encoded by any combination of bypass and context. v. In one example, the index can be encoded and decoded in bypass mode. vi. In one example, the index can be encoded and decoded in context mode. vii. In one example, at least one binary value (eg, only the first binary value) of the index may be context-decoded. viii. In one example, the first N binary values ​​may be context encoded and decoded. The encoded contexts may share the same context or have independent contexts. 2. Whether and / or how many directions should be utilized may be signaled or derived on the fly (eg, from decoded information). a. In one example, additional directions can be added only for some specific block sizes. i. In one example, additional directions can be added to blocks where width * height > C1. (i) Examples of C1 can be 64 or 256. ii. In one example, additional directions can be added to blocks where width * height < C1. (i) Examples of C1 can be 64 or 256. iii. In one example, additional directions can be added to blocks where width > C1 and / or height > C2. (i) Examples of C1 and C2 can be 16 and 32 respectively. iv. In one example, additional directions can be added to blocks where width < C1 and / or height < C2. (i) Examples of C1 and C2 can be 16 and 32 respectively. v. The above thresholds C1 / C2 can be predefined or signaled in the bitstream. b. In one example, whether to apply additional directions and / or which additional directions to use can be based on picture resolution and / or reference picture list and / or low latency check flag. c. In one example, whether to apply additional directions and / or which additional directions to use can be signaled from the encoder to the decoder, for example, in SPS / PPS / VPS / APS / strip header / picture header / CTU / CU / PU, etc. i. For example, pictures at low temporal layers can use more directions, and / or pictures at high temporal layers can use fewer directions. Extensions on distance offset used in MMVD design 3. It is proposed that at least one additional distance can be added to the original set of distances (or candidate list) for the MMVD mode and / or extensions of the MMVD mode, or at least one existing distance can be removed from the original set of distances (or candidate list) for the MMVD mode and / or extensions of the MMVD mode. a. It is proposed that at least one additional distance offset can be added to the original 8 distance offsets for MMVD and / or GMVD and the 5 original distance offsets for affine MMVD refinement candidates. b. Alternatively, at least one distance offset can be removed from the original 8 distance offsets for MMVD and the 5 original distance offsets for affine MMVD refinement candidates. c. In one example, at least one additional distance offset may be added between and / or beyond the original 8 MMVD distance offsets. The number of additional distance offsets may be 4 or 8 or any other number. d. In one example, at least one additional distance offset may be added between and / or beyond the original 5 affine MMVD distance offsets. The number of additional distance offsets may be 4, 5, or 8 or any other number. e. In one example, at least one additional distance offset may be added only between two distance offsets that are both smaller than a threshold. f. In one example, at least one additional distance offset may be added only between two distance offsets that are both greater than a threshold. g. In one example, some distance offsets may be removed, and the number of distance offsets may be reduced to N (eg, 3, or 4, or 5). h. In one example, the distance offset of every other one can be obtained from Fig.45 The 2nd distance offset depicted in is removed. i. In one example, every other distance offset may be removed starting from the first distance offset. j. In one example, the first half of the distance offsets (e.g., indexes 0, 1, 2, 3 of the MMVD) can be removed k. In one example, the second half of the range offsets (eg, indices 4, 5, 6, 7 of the MMVD) may be removed. l. In one example, the final offset can be indicated as a joint index of all offsets or divided into 2 indices for direction and distance offsets, which can be encoded and decoded using a truncated unary code, or a truncated binary code, or a Rice code with parameter R, a k-order exponential Golomb code, or any combination of binary values ​​encoded and decoded with a combination of bypass and context. 4. The initial range offset candidate list may be predefined or signaled or derived on the fly. a. In one example, based on the block size, an initial distance offset candidate list may be selected. i. As an example, for blocks with width*height > C, we choose one set of distance offsets, and for the rest of the blocks we choose a different set of distance offsets. 5. It is proposed that for MMVD and / or extensions of MMVD (e.g. affine MMVD and / or GMVD), whether to apply (one or more) additional directions and / or additional offsets, and / or which additional directions and / or additional offsets to apply may depend on the original base motion vector direction and / or its magnitude. a. In one example, the initial offset can be determined by the base MV amplitude. The i.MV magnitude can be derived from MVx and MVy. For example, the MV magnitude is calculated as |MVx| + |MVy|. ii. The MV magnitude can be derived from MVx and MVy. For example, the MV magnitude is calculated as (MVx)^2+(MVy)^2. iii. If both Ref0 and Ref1 are available, the magnitude can be a weighted average of the MV lengths of each reference list MV. iv. For affine MMVD, the MV magnitude of the top - left control point of the basic affine MV can be used to determine the initial offset by the methods specified in i, ii, iii. b. In one example, the initial distance offset for MV magnitude > C1 can be greater than the initial distance offset for MV magnitude < C1 i. In one example, the initial distance offset for MV magnitude > C1 can be N times the initial distance offset for MV magnitude < C1, where C1 can be, for example, 50 pixels and N can be, for example, 2. c. In one example, at least one direction offset can be derived from the base MV. For affine MMVD, the MV of the top - left control point of the basic affine MV can be used to derive an additional direction offset. i. In one example, the direction offset can be exact, such as Fig.46 shown, a direction offset parallel or perpendicular to the original base MV. ii. In one example, the additional direction offset can be approximated. For example, if the base MV direction is between pi / 8 and 3pi / 8, a diagonal direction offset can be used, otherwise a vertical / horizontal direction offset will be used. iii. The direction offset can replace the offset in the original design of MMVD / GMVD / affine MMVD. I. Alternatively, the direction offset can be added to be used together with the original design of MMVD / GMVD / affine MMVD. MMVD reordering 6. For the MMVD mode and / or extensions of MMVD (e.g., affine MMVD or GPM MMVD (GMVD), MMVD for IBC mode, MMVD for affine IBC mode), the basic motion candidates and / or the refined motion candidates (e.g., by adding MVD) can be reordered. a. In one example, the reordering process should be performed before interpreting the MMVD refinement method from at least one syntax element. b. In one example, building N1 refinement steps of N1*N2*N3 possibilities, N2 directions, and N3 base candidates can be reordered together. i.N1 can be 4, 5, 8, 16 or any other number. N2 can be 2, 4, 6, 8, 16, 32 or any other number. N3 can be 1, 2, 3, 4 or any other number. c. In one example, N possible refinement positions (which may be asymmetric in direction or step size, or have no explicit direction or step size) that construct N*N3 possibilities and N 3 base candidates may be reordered together. d. In one example, the re-ranking process can be performed on each base candidate separately. i. For example, for each base candidate, N1 refinement steps and N2 directions that construct N1*N2 possibilities can be reordered together. ii. For example, if for each basis candidate there are a total of N possible refinement positions (which may be asymmetric with respect to direction or step size, or have no explicit direction or step size) that can be reordered together. iii. In one example, the base candidates may be reordered in advance. Afterwards, refinement of the first base candidate is further applied. (i) For example, for the first base candidate, N1 refinement steps and N2 directions that construct N1*N2 possibilities can be reordered together. (ii) For example, for the first basis candidate, if there are a total of N possible refinement positions (which may be asymmetric in direction or step size, or have no clear direction or step size), they can be reordered together. e. In one example, the reordering process may be performed on the candidates with the same base candidate and the same direction respectively. For example, for the candidates with the same base candidate and the same direction, N1 refinement steps may be reordered together. f. In one example, the reordering process may be performed on the candidates with the specified base candidates and the specified directions respectively. For example, for a candidate with a specified base candidate and a specified direction, N1 refinement steps may be reordered together. g. In one example, the reordering process may be performed on candidates with the same base candidate and the same refinement step size respectively. For example, for candidates with the same base candidate and the same refinement step size, N2 directions may be reordered together. h. In one example, the reordering process may be performed on the candidates with the specified base candidate and the specified refinement step size respectively. For example, for the candidates with the specified base candidate and the specified refinement step size, N2 directions may be reordered together. i. In one example, any subgroup of possible options can be reordered within that subgroup. i. In one example, subgroups are divided from all candidates of the MMVD according to directions. ii. In one example, subgroups are divided from all candidates of MMVD according to distance. iii. In one example, subgroups are divided from all candidates of the MMVD according to base candidates. iv. In one example, subgroups are divided from all candidates of the MMVD according to any combination of direction, distance, and base candidates. j. In one example, the reordering process may be applied sequentially based on the characteristics. i. As an example, the first reordering process can be performed on the base candidates. Next, the reordering process for direction can be performed with a fixed distance offset. The last reordering process can be performed on the distance offset. ii. Alternatively, the first reordering process can be performed on the base candidates. Next, the reordering process can be performed on each direction and distance combination with the same base candidate. iii. Alternatively, the first re-ranking process may be performed on the base candidates. Next, the re-ranking process may be performed on each direction and distance combination with a designated (eg, first) base candidate. k. In one example, the reordered MMVD and / or its extension (eg, affine MMVD) may be indicated by an index to be transmitted by a signal, i. In one example, the indexes may all be encoded using truncated unary codes. ii. In one example, the index may all be encoded and decoded using truncated binary / binary codes. iii. In one example, the indices can all be encoded and decoded using Rice codes or k-th order Exponential Golomb codes, where k can be 0, 1, or any number. (i) In one example, Rice codes with parameters 1, 2, 4, 8 or any other number may be used. iv. In one example, the prefix and suffix of the code can be encoded into a binary value that is encoded by a combination of bypass and context. v. In one example, the index can be encoded and decoded in bypass mode. vi. In one example, the index can be encoded and decoded in context mode. vii. In one example, at least one binary value of the index (eg, only the first binary value) may be context-decoded. (i) In one example, the first N binary values ​​may be context-coded. The coded contexts may share the same context or have independent contexts. 1. In one example, the base candidate index may be encoded separately, for example, as truncated unary or truncated binary into a binary value that is context or bypass encoded. The remaining direction and distance may be encoded as described above. m. In one example, after reordering the MMVD candidates, we only keep the top N with the lowest cost. N can be any integer. Only the new limited options will be encoded or decoded. i. In one example, after re-ranking the MMVD candidates, we only keep the top half with the lowest cost. Only the new limited options are encoded or decoded. ii. In one example, after re-ranking the MMVD candidates, we only keep the top 1 / 4 with the lowest cost. Only the new limited options will be encoded and decoded. iii. In one example, after re-ranking the MMVD candidates, we only keep the top 1 / 8 with the lowest cost. Only new limited options will be encoded and decoded. iv. In one example, after re-ranking the MMVD candidates, we only keep the top 1 / 16 with the lowest cost. Only new limited options will be encoded and decoded. n. In one example, only candidates with a cost less than F*bestCost can be selected. F can be any number, such as 1.2, 2, 2.5, ..., and bestCost is the best (eg, minimum) template matching cost of the candidate. o. In one example, a fixed ratio from the best template matching cost may be used to select candidates, or top N may be selected, or any combination of restrictions based on block size and / or base MV amplitude and / or base MV direction may be used. p. In one example, the reordering process can be limited to a specific block size. For example, we can apply reordering to blocks with width*height>C, and we can not apply reordering to the rest of the blocks. q. In one example, after the re-ordering process, only the best MMVD candidate may be selected and no additional index signaling may be required. 7. Reordering can be based on template matching method. a. In one example, a candidate reordering criterion may be the template matching cost between the template surrounding the current block and a reference of the template. i. In one example, the cost may be the sum of absolute differences (SAD) between the template sample and its reference. ii. In one example, the cost may be the sum of absolute transformed differences (SATD) between the template sample and its reference or any other cost metric. iii. In one example, this cost may be the mean-removed sum of absolute differences (MR-SAD) between the template sample and its reference. iv. In one example, the cost may be a weighted average of the SAD / MR-SAD and SATD between the template sample and its reference. v. In one example, the cost function between the current template and the reference template can be (i) Sum of absolute differences (SAD) / mean removed SAD (MR-SAD) (ii) Sum of Absolute Transformed Differences (SATD) / SATD with Mean Removal (MR-SATD) (iii) Sum of Squared Difference (SSD) / Mean Removed SSD (MR-SSD) (iv) SSE / MR-SSE (v) Weighted SAD / Weighted MR-SAD (vi) Weighted SATD / Weighted MR-SATD (vii) Weighted SSD / Weighted MR-SSD (viii) Weighted SSE / Weighted MR-SSE (ix) Gradient information b. In addition to the SAD calculated in (f), the cost can also consider the continuity between the reference template and the reconstructed samples adjacent or non-adjacent to the current template (Boundary_SAD, boundary SAD). For example, consider the reconstructed samples to the left and / or above that are adjacent or non-adjacent to the current template. i. In one example, the cost can be calculated based on the SAD and the boundary SAD (i) In one example, the cost may be calculated as (SAD + w*boundary SAD). w may be predefined or signaled or derived from decoded information. c. In one example, the top K1 rows and / or the left K2 columns and / or the K1*K2 samples / pixels in the corners may be used as templates. i. K1 and K2 can be any numbers; for example, K1 and K2 can be 1, 2, 3, width / 2, height / 2, width, height. ii. Alternatively, use only the top K1 row as a template. iii. Alternatively, use only the left K2 column as a template. iv. Alternatively, use the top K1 row, left K2 column as the template. d. The template matching process can include a component, such as brightness. i. In general, the template matching process can include multiple components, such as brightness and chrominance. (i) In one example, the total template matching cost may be calculated as a weighted sum of the template matching costs of different color components. e. In one example, the template for bidirectional prediction (RT bi-pred ) is obtained by weighted averaging the reference samples of the template (RT0) in reference list 0 and the reference samples of the template (RT1) in reference list 1. An example is as follows: RT=((2 N -w)*RT0+w*RT1+2 N-1 )>>N, for example, N=3. f. In one example, the weight of the reference template in reference list 0, such as (8-w), and the weight of the reference template in reference list 1, such as (w), may be determined by the BCW index of the merge candidate. i. In one example, the BCW index is equal to 0 and w is set equal to -2. ii. In one example, the BCW index is equal to 1 and w is set equal to 3. iii. In one example, the BCW index is equal to 2 and w is set equal to 4. iv. In one example, the BCW index is equal to 3 and w is set equal to 5. v. In one example, the BCW index is equal to 4 and w is set equal to 10. g. In one example, if the local illumination compensation (LIC) flag of the merge candidate is true, the reference sample of the template is derived using the LIC method. i. Alternatively, a reference sample of the template is derived without LIC. h. In one example, when deriving reference samples for a template, the motion vector of the merge candidate is rounded to integer pixel precision, where the integer motion vector may be its closest integer motion vector. i. In one example, when deriving the reference samples of the template, an N-tap interpolation filter is used to obtain the reference samples of the template at the sub-pixel position. For example, N may be 2, 4, 6, 8, or 12. 8. Early termination of the reordering process can be applied. a. In one example, only candidates associated with a specific direction may be further checked if a specific condition is met. b. In one example, only candidates associated with a specific distance offset but in a different direction may be further examined if a specific condition is met. About MVD symbol prediction 9. It is proposed that for an advanced motion vector prediction (AMVP) mode and / or its extensions (e.g., affine AMVP), an MMVD mode and / or an extension of MMVD (e.g., affine MMVD or GPM MMVD (GMVD), MMVD for IBC mode, MMVD for affine IBC mode), the sign of the MVD can be predicted (or reordered). a. In one example, the sign of the MVD horizontal component (MVx) can be predicted. b. In one example, the symbols of MVx may be predicted (reordered) and a flag is encoded to determine if the prediction was correct. c. In one example, the sign of the MVD vertical component (MVy) may be predicted. d. In one example, the symbols of MVy may be predicted (reordered) and a flag may be encoded or decoded to determine if the prediction was correct. e. In one example, the signs of MVx and MVy can be jointly predicted. More precisely, there are 4 possible combinations of MVx and MVy signs: (+, +), (+, -), (-, +), (-, -). After the prediction, no additional information is encoded or decoded. f. In one example, the possible combinations may depend on whether MVx and / or MVy is equal to zero. g. In one example, the symbols of MVx and MVy can be predicted (reordered), and a flag is coded to determine whether to select the first option or the second option. The flag can be context coded or bypass coded. h. In one example, the symbols of MVx and MVy can be predicted (reordered), and the first N (e.g., N=2, 3, ..) options can be used with non-fixed length codes (e.g., unary or truncated unary codes, or binary codes) The index of the codec is transmitted by signaling. The index can be context coded or bypass coded. i. In one example, if the first option is located before the second option after reordering, the code length of the first option should not be longer than the code length of the second option. i. In one example, the symbols of MVx and / or MVy may be encoded and decoded using context encoding, where the context may be determined by prediction of MVx and / or MVy. j. In one example, the signs of MVx and / or MVy may be coded with context coding, where the context may depend on the amplitude of the MVD component. k. In one example, information indicating whether the prediction for MVx and / or MVy is correct may be conditionally signaled. i. In one example, if MVx and / or MVy is equal to 0, then this information may not be transmitted via a signal. 1. In one example, the sign of MVx and / or MVy may not be explicitly signaled but may be implicitly set equal to the predicted value. 10. The symbol prediction (or reordering) of MVD can be based on the template matching method or the bilateral matching method. a. In one example, a candidate reordering criterion may be the template matching cost between the template surrounding the current block and a reference of the template. i. In one example, the cost may be the sum of absolute differences (SAD) between the template sample and its reference. ii. In one example, the cost may be the sum of absolute transformed differences (SATD) between the template sample and its reference prior or any other cost measure. iii. In one example, the cost may be the mean removed sum of absolute differences (MR-SAD) between the template sample and its reference. iv. In one example, the cost can be the SAD / MR-SAD between the template sample and its reference and Weighted average of SATD. v. In one example, the cost function between the current template and the reference template can be (i) Sum of Absolute Difference (SAD) / Mean Removed SAD (MR-SAD) (ii) Sum of Absolute Transformed Difference (SATD) / Mean Removed SATD (MR-SATD) (iii) Sum of Squared Difference (SSD) / Mean Removed SSD (MR-SSD) (iv) SSE / MR-SSE (v) Weighted SAD / Weighted MR-SAD (vi) Weighted SATD / Weighted MR-SATD (vii) Weighted SSD / Weighted MR-SSD (viii) Weighted SSE / Weighted MR-SSE (ix) Gradient information b. In one example, MV candidates are constructed by creating possible combinations between symbols and absolute MVD values, and Add it to the MV predictor. Based on the template matching cost or the bilateral matching cost, derive The MVD symbols predict costs, and the MVD symbols are sorted in ascending order according to the cost values. i. In one example, the actual MVD symbol finally used may be the one with the minimum MVD symbol prediction cost MVD symbol. ii. In one example, the actual MVD symbol finally used may be the first N in the sorted MVD symbol column. (For example, N=2, 3, ..) MVD symbols are selected. (i) In one example, the selected MVD symbol (i.e., the actual MVD symbol used in the end) can be represented by a flag or The flag or index can be transmitted by signal. And the flag or index can be context coded or bypassed. On the combination of MVD symbol prediction and MMVD reordering 11. It is proposed that any MVD symbol prediction for AMVP mode and / or its extensions (e.g., affine AMVP), MMVD mode and / or its extensions can be combined with any MMVD reordering for MMVD mode and / or extensions of MMVD (e.g., affine MMVD or GPM MMVD (GMVD), MMVD for IBC mode, MMVD for affine IBC mode). a. In one example, any MVD symbol prediction for AMVP can be combined with any MMVD reordering for MMVD. b. In one example, any MVD sign prediction for affine AMVP can be combined with any MMVD reordering for affine MMVD. c. In one example, any MVD sign prediction for AMVP and affine AMVP can be combined with any MMVD reordering for MMVD and affine MMVD. d. In one example, any MVD sign prediction for AMVP and affine AMVP and affine MMVD can be combined with any MMVD reordering for MMVD. e. In one example, any MVD sign prediction for AMVP and affine AMVP and MMVD can be combined with any MMVD reordering for affine MMVD or other extensions thereof. f. In one example, both symbol prediction and MMVD reordering can be applied to MV simultaneously. For example, symbol prediction will be applied to MMVD symbols, and MMVD reordering can be applied to MMVD amplitude or its basis. g. In one example, symbol prediction can be applied to MVD codec methods that do not include MMVD (eg, AMVP), but MMVD reordering can be applied to MMVD mode. About MMVD for Bidirectional Prediction Base Candidates The MVD candidate list of list X (eg, X=0) is represented as {MvdLX i}, where i is in the range of [0, M-1] and M is the total number of allowed MVD candidates for list X. The MVD candidate list for list Y (eg, Y=1-x) is {MvdLY j}, where j is in the range [0, N-1] and N is the total number of allowed MVD candidates for list Y. 12. Instead of always signaling the MVD information for list 0, it is proposed to signal the MVD information for list 1. a. Alternatively, whether list 0 or list 1 is transmitted through signaling may be further indicated in the code stream or determined on the fly (eg, based on the reference picture information of the base candidate). 13. Suggest that N and M may not be equal. a. In one example, N and / or M may be predefined or determined on the fly or transmitted via a signal. 14. It is proposed to modify the MMVD for bidirectional prediction to: b. In one example, List 0 and List 1 have their own independent MVDs, where MvdLY j Instead of using existing technology MvdLX j Exported. c. In one example, only list 0 has MVD, list 1 has no MVD (0). d. In one example, only list 1 has MVD, list 0 has no MVD (0). e. In one example, only the reference picture closest to the current picture may have MMVD, and the other reference pictures have no MVD (0). f. In one example, only the reference picture that is farther away from the current picture may have MMVD, and the other reference picture has no MVD (0). g. In one example, only the reference directions (list 0 or list 1) whose MV has a larger cost can have an MVD. i. In one example, only the reference directions (list 0 or list 1) whose MV has a smaller cost may have an MVD. ii. In one example, the cost may be the template matching cost corresponding to the MV of one reference list (list 0 or list 1). iii. In one example, the cost may be the bilateral matching cost corresponding to the MV of one reference list (list 0 or list 1). h. In one example, only references before the current picture can have MVD. i. In one example, only references after the current picture can have MVD. j. In one example, depending on the MV size or angle of the reference block, only one may have MVD. k. In one example, a bi-directional prediction candidate may be converted to a uni-candidate and MVD may be applied thereon. 1. In one example, among both list 0 and list 1 having MVD, only list 0 having MVD, and only list 1 having MVD, which one is finally used can be determined by RD decision and transmitted to the decoder through signal. 15. It is proposed that the final MVD of list 0 and list 1 can be ( MvdLX i ,MvdLY j )Yes, where i is not equal to j. m. In one example, for all ( MvdLX i , MvdLY j ) pairs, the template cost can be calculated, and the pair that gives the minimum template cost can be selected as the final MVD of list 0 and list 1. i. Alternatively, additionally, early termination may be applied to reduce the number of pairs to check. 16. Propose to determine the final MVD of list 0 and list 1 respectively. n. In one example, not based on (MvdLX j , MvdLY j ) pairs to calculate the template cost, but the template cost can be calculated independently for each candidate in list 0 and list 1. i. Alternatively, additionally, MvdLX with minimum template cost j can be used as the final MVD of list X. 17. It is proposed to add zero MVD to the MVD candidate list in MMVD design. About template reference samples 18. It is proposed that the first interpolation filter used to generate template reference samples for MMVD reordering and / or MVD symbol prediction may be different from the second interpolation filter used to generate reference samples for inter-frame prediction. o. For example, the first interpolation filter may have fewer taps than the second interpolation filter. p. For example, the first interpolation filter may be a bilinear filter or a 4-tap filter. q. In one example, a 12-tap interpolation filter may be used. 19. It is proposed that the first interpolation filter used to generate template reference samples for MMVD reordering and / or MVD symbol prediction may be different from the second interpolation filter used to generate r template reference samples for another codec tool (eg, TM-based merge candidate list). 20. It is proposed that the modified MV (eg, the estimation of the MV amplitude) can be used for MVD symbol prediction or MMVD reordering. r. In one example, the nearest integer estimate can be used for prediction / re-ranking. s. In one example, the nearest half-pixel estimate can be used for prediction / re-ordering. t. In one example, the nearest 4 pixel estimates may be used for prediction / re-ranking. Extension of the number of basis candidates used in MMVD 21. It is proposed that at least one additional basis candidate may be added to the original basis candidates of the MMVD and / or its extension (eg, affine MMVD). a. Alternatively, at least one existing basis candidate may be removed from the original basis candidates for the MMVD and / or its extension (eg, affine MMVD). b. In one example, depending on the block size, additional base candidates may be added. c. In one example, depending on the picture resolution, additional base candidates may be added. d. In one example, additional base candidates may be added based on the similarity / difference between the original base candidates. e. Alternatively, at least one existing base candidate may be removed from the original base candidates based on block size and / or picture resolution and / or similarity / difference between the original base candidates. f. The base candidate index may be encoded using a truncated unary code, or a truncated binary code, or a Rice code or a k-order exponential Golomb code with a parameter R, and encoded by a binary value encoded by a combination of bypass and context. i. Alternatively, it can be combined with the offset index and encoded and decoded together. About early termination of cost calculation 22. It is proposed that the cost calculation for the first candidate or candidate position may be terminated early. a. In one example, if the cost of the left stencil sample is higher than the maximum allowable cost, the cost calculation for the upper stencil sample may be skipped. b. In one example, if the cost of the upper stencil sample is higher than the maximum allowable cost, the cost calculation for the left stencil sample may be skipped. c. In one example, depending on the width and height of the block, the longer template side cost may be calculated first, and if that cost is higher than the maximum allowable cost, the cost calculation for the shorter template side may be skipped. d. In one example, the above maximum allowable cost may be a fixed number. e. In another example, the maximum allowable cost mentioned above may be a variable and may be a function of block size, width, height, fixed threshold, last cost, best cost, etc. f. In one example, if N candidates (eg, those with the lowest cost) are selected from M candidates, then the maximum allowable cost described above may be the Nth lowest cost. g. In another example, the maximum allowable cost mentioned above may be the cost of the second candidate or candidate positions. i. In one example, when calculating the cost of a first candidate or candidate location, a second candidate or candidate location may have the kth lowest cost. About Affine MMVD 23. It is proposed that there may be some simplifications for affine MMVD reference template derivation. For a sub-block based merge candidate with sub-block size equal to Wsub*Hsub, the upper template includes several sub-templates of size Wsub×L, and the left template includes several sub-templates of size L×Hsub. Fig.47 As shown, the motion information of the sub-blocks in the first row and the first column of the current block is used to derive the reference samples of each sub-template. a. In one example, the prediction may be calculated for each 4x4 sub-block. b. In one example, the prediction may be calculated for each 8x8 sub-block. c. In one example, the prediction may be calculated for each min(4, width)×min(4, height) sub-block. d. In one example, the prediction may be calculated for each (width / 2)×(height / 2) sub-block. e. In one example, a prediction may be calculated for each (width)×(height) sub-block (e.g. Fig.48 shown). f. In one example, there may be no LIC application for affine MMVD. g. In one example, when calculating the cost for the affine MMVD, there may be no LIC application. 24. It is proposed that whether to check the affine MMVD option at the encoder side may depend on the affine merging cost. a. In one example, if the best affine merge cost is on the top N merge costs or non-merge costs, then the affine MMVD options can be checked in the encoder. Otherwise, they can be skipped. N can be any number, such as 1, 3, 5, 10, 20, etc. b. In one example, if the best affine cost is not among the top N costs, the affine MMVD options can be checked in the encoder. Otherwise, they can be skipped. N can be any number, such as 1, 3, 5, 10, 20, etc. c. In another example, if the best affine cost is less than alpha*cost_t, then the affine MMVD options can be checked in the encoder. Otherwise, they can be skipped. alpha can be any positive real number, such as 0.7, 1, 1.25, 1.73, etc., and cost_t can be the best non-affine cost or the Nth best cost, where N can be any integer, such as 5, 10. d. In another example, if the best affine cost is greater than alpha*cost_t, then the affine MMVD options can be checked in the encoder. Otherwise, they can be skipped. alpha can be any positive real number, such as 0.7, 1, 1.25, 1.73, etc., and cost_t can be the best non-affine cost or the Nth best cost, where N can be any integer, such as 5, 10. About the step size and additional direction of MMVD 25. It is proposed that the initial step size can depend on the video resolution. a. In one example, the initial step size of a video with width*height>C can be N times the initial step size of a video with width*height<=C, where C can be any integer, such as 10^5 or any other number, and N can be any number, such as 2, 4, etc. b. In one example, there may be several thresholds for deciding the initial step size of the MMVD. For example, for a video with width*height>C1, the initial step size will be L1. Otherwise, for a video with width*height>C2, the initial step size will be L2. Otherwise, for a video with width*height>C3, the initial step size will be L3, and so on, where C1>C2>… and L1>L2>…, which can be any integer. c. In one example, the initial step size may be signaled or derived from at least one syntax element, such as in SPS / PPS / picture header / slice header / tile / CTU / etc. 26. It is proposed that the initial step size can depend on the POC difference (delta POC). a. In one example, the initial step size for blocks with POC delta > C may be N times the initial step size for blocks with POC delta <= C. i. In one example, C can be 2 or 4 or any other integer. ii. In one example, N may be 2, 3, or any other integer. b. The POC delta can be calculated as the absolute difference between the POCs of the current picture and the reference picture. 27. It is proposed that the number of directions can depend on the step size. a. In one example, the smallest step size can have N1 directions, and the remaining step sizes can have N2 directions, where N1 and N2 can be any integers. An example is N1=8 and N2=16. b. In one example, for the first 2 smallest step sizes, there can be N1 directions, and the remaining step sizes can have N2 directions, where N1 and N2 can be any integers. An example is N1=8 and N2=16. c. In one example, for the first M (eg, M≥1) smallest steps, there may be N1 directions, and the remaining steps may have N2 directions, where N1 and N2 may be any integers. An example is N1= 8 and N2=16. About MMVD logo codec 28. It is proposed that the MMVD flag or the affine MMVD flag can be encoded and decoded using at least one context (the context index can be denoted as "ctx"). a. In one example, ctx may depend on information parsed before parsing the current MMVD flag or the affine MMVD flag (eg, codec mode / block size, etc.). b. In one example, ctx may depend on the MMVD flag of at least one neighboring block. i. In one example, if at least one of the blocks above or to the left of the current block uses MMVD, ctx will be 1, otherwise ctx will be 0. ii. In one example, if both the upper block and the left block of the current block use MMVD, ctx will be 2. If only one of the upper block or the left block of the current block uses MMVD, ctx will be 1, otherwise ctx will be 0. c. In one example, ctx may depend on the skip flag of the current block. i. In one example, if the current block is skipped, ctx will be 0 or 1, otherwise ctx will be 1 or 0. d. In one example, ctx may depend on the prediction direction of the current block. i. In one example, if the current block is uni-directionally predicted, ctx will be 0, otherwise if the current block is bi-directionally predicted, ctx will be 1. e. In one example, any combination of the above scenarios may be applied, and several possible ctx are available. f. Alternatively, the MMVD flag or affine MMVD flag may be bypassed for encoding and decoding. General 29. Whether and / or how to apply the above method may depend on the encoded information. g. In one example, the coded information may include block size and / or temporal layer, and / or slice / picture type, color component, etc. 30. Whether and / or how to apply the above method can be indicated in the code stream. h. The indication of enabling / disabling or which method to apply may be signaled at the sequence level / picture group level / picture level / slice level / tile group level, e.g. in the sequence header / picture header / SPS / VPS / DPS / DCI / PPS / PPS / slice header / tile group header. i. Enable / disable or indication of which method to apply may be signaled at PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU row / slice / tile / sub-picture / other kinds of areas containing more than one sample or pixel. 2.29. IBC mode extension proposed in P21091098901H The following detailed description should be considered as an example to explain the general concept. These embodiments should not be interpreted narrowly. In addition, these embodiments can be combined in any way. The term "block" may refer to a codec tree block (CTB), a codec tree unit (CTU), a codec block (CB), a CU, a PU, a TU, a PB, a TB, or a video processing unit including a plurality of samples / pixels. A block may be rectangular or non-rectangular. For IBC-coded blocks, a block vector (BV) is used to indicate the displacement from the current block to a reference block that has been reconstructed in the current picture. W and H are the width and height of the current block (eg, luma block). It should be noted that the following terms are not limited to the specific terms defined in existing standards. Any variants of codecs are also applicable. For example, the term "GPM" is used to represent any codec that derives two or more sets of motion information and uses the derived motion information and segmentation mode / weighted mask to obtain the final prediction. For example, TPM can also be regarded as GPM. Below, Mv1 and Mv2 are the motion vectors from the first and second parts of the triangulation or geometric partitioning. 1. In one example, IBC merge mode with block vector difference (MBVD) may be used. a. In MBVD, the BV can be derived based on the IBC merge candidate, which can be further refined by the signaled BVD information. b. In one example, the BVD information may include one or more IBC merge candidate indexes, one or more indications (e.g., multiple indices) for specifying (one or more) motion magnitudes, and one or more indications (e.g., multiple indices) for indicating (one or more) motion directions. i. In MBVD mode, at least one is selected from the candidates in the IBC merge list as the BV basis. At least one MBVD candidate index is signaled to specify which one or more of the IBC merge candidates to use. (i) In one example, an MBVD candidate index is signaled to specify which of the top N IBC merge candidates to use. a) In one example, N is set to 2. b) In one example, the candidate index may be binarized into a truncated code with a maximum value equal to N-1. (ii) In one example, IBC merge candidates may be reordered before use. ii In one example, the distance index specifies motion magnitude information and indicates a predefined offset from a starting point. (i) An offset may be added to either the horizontal or vertical component of the starting BV. (ii) An offset may be added to both the horizontal and vertical components of the starting BV. (iii) In one example, the distance set may be {1 pixel, 2 pixels, 4 pixels, 8 pixels, 16 pixels, 32 pixels}. (iv) In one example, the distance set may be {1 pixel, 2 pixels, 4 pixels, 8 pixels, 16 pixels, 32 pixels, 64 pixels, 128 pixels}. (v) In one example, the distance set may be {1 pixel, 2 pixels, 3 pixels, 4 pixels, 6 pixels, 8 pixels, 16 pixels}. (vi) In one example, the distance set may be {1 pixel, 2 pixels, 3 pixels, 4 pixels, 6 pixels, 8 pixels, 16 pixels, 32 pixels, 64 pixels}. (vii) In one example, the relationship between the distance index and the predefined offset is specified in Table 2-4. (viii) In one example, the relationship between the distance index and the predefined offset is specified in Table 2-5. (ix) In one example, the relationship between the distance index and the predefined offset is specified in Table 2-6. (x) In one example, the relationship between the distance index and the predefined offset is specified in Table 2-7. (xi) In one example, the relationship of the distance index to the predefined offset may be signaled from the encoder to the decoder at a sequence level / picture level / slice level / CTU level / CU level. (xii) The index may be binarized using a unary codec, a truncated unary codec, an exponential Golomb code, a truncated exponential Golomb code, a fixed length code, or any other binarization method. iii In one example, the direction index indicates the direction of the BVD relative to the starting point. The direction index may indicate M BVD directions. (i) In one example, M is set to 4. a) In one example, 4 horizontal / vertical directions may be used. b) In one example, 4 diagonal directions can be used. c) In one example, the relationship between the direction index and the predefined direction is specified in Table 2-8. For direction index 0, (offset, 0) is BVD; for direction index 1, (-offset, 0) is BVD; for direction index 2, (0, offset) is BVD; for direction index 3, (0, -offset) is BVD. d) In one example, the relationship between the direction index and the predefined direction is specified in Table 2-9. For direction index 0, (offset, offset) is BVD; for direction index 1, (offset, -offset) is BVD; for direction index 2, (-offset, offset) is BVD; for direction index 3, (-offset, -offset) is BVD. (ii) In one example, M is set to 8. a) In one example, 4 horizontal / vertical directions plus 4 diagonal directions may be used. b) In one example, the relationship between the direction index and the predefined directions is specified in Table 2-10. (iii) The index may be binarized using a unary codec, a truncated unary codec, an exponential Golomb code, a truncated exponential Golomb code, a fixed length code, or any other binarization method. iv. In one example, the number of distances and / or directions of the MBVD of a block may be determined by: (i) Resolution of the image. (ii) Configuration of the encoding and decoding process. (iii) BVD of the neighboring blocks of this block. a) In one example, the neighboring blocks above and to the left can be used (e.g. Fig.49 depicted). b) In one example, adjacent spatial neighboring blocks including left and / or above and / or above right and / or below left and / or above left spatial neighboring blocks may be used ( Fig.49 An example is shown). (iv) Alternatively, the number of distances and / or directions for MBVD can be calculated at the sequence level / graph. Slice level / slice level / CTU level / CU level, transmitted by signal from encoder to decoder. c. In one example, candidates / directions / amplitudes in MBVD that may produce BVs outside the valid range of BVs may be excluded from the set of candidates / directions / amplitudes to be selected or transmitted by signal. i. Alternatively, the BV generated in MBVD can be clipped to within the valid range. ii. Alternatively, the BV generated in the MBVD must be within the valid range of the conforming codestream. Table 2-4 – Relationship between distance index and predefined offsets Table 2-5 – Relationship between distance index and predefined offsets Table 2-6 – Relationship between distance index and predefined offsets Table 2-7 – Relationship between distance index and predefined offsets Table 2-8 – Sign of MV offset specified by direction index Directions to IDX 00 01 10 11 x-axis + - N / A N / A y-axis N / A N / A + - Table 2-9 – Sign of MV offset specified by direction index Directions to IDX 00 01 10 11 x-axis + + - - y-axis + - + - Table 2-10 – Sign of MV offset specified by direction index Directions to IDX 000 001 010 011 100 101 110 111 x-axis + - N / A N / A + + - - y-axis N / A N / A + - + - + - 2. In one example, a new CIIP prediction mode (referred to as CIIP_N) combines at least one IBC prediction signal and at least one prediction signal generated by a second prediction method. a. The second prediction method can be intra-frame prediction or inter-frame prediction. b. The second prediction signal and the IBC prediction signal may be combined by weighted averaging. i. The form of CIIP_N prediction is P CIIP_N =((2 N -wt)*P IBC +wt*P sec +offset)>>N ii. In one example, the offset is an integer, such as 2 N >>1. iii. In one example, N=2. iv. In one example, the weight values ​​may be predefined. (i) In one example, wt is set to 2. v. In one example, the weight value for each sample may be position-dependent. (i) For example, for some positions, wt = 2 N . (ii) For example, for some positions, wt=0. vi. In one example, the weight values ​​may be signaled from the encoder to the decoder. c. IBC prediction signal P in CIIP_N mode IBC It can be derived using the same IBC prediction process that is applied to the regular IBC merge mode. d. Second prediction signal P sec It can be derived according to the conventional intra prediction process, and the combined mode is named CIIP_N1. i. In one example, the intra prediction mode may be planar mode. ii. In one example, the intra prediction mode may be an intra prediction mode implicitly derived by the DIMD method. iii. In one example, the intra prediction mode may be an intra prediction mode implicitly derived by the TIMD method. iv. In one example, the weight value may be calculated according to the coding mode of the neighboring blocks. (i) In one example, the weight value may be calculated based on the encoding and decoding modes of the top and left neighboring blocks (e.g. Fig.50 As shown) as follows: If the top neighbor is available and is intra-coded, set isIntraTop to 1, otherwise set isIntraTop to 0; If the left neighbor is available and is intra-coded, set isIntraLeft to 1, otherwise set isIntraLeft to 0; If (isIntraLeft + isIntraTop) is equal to 2, set wt to 3. Otherwise, if (isIntraLeft+isIntraTop) is equal to 1, set wt to 2. In other cases, set wt to 1. e. Second prediction signal P sec It can be derived using the same inter prediction process applied to the regular merge mode, and the combined mode is named CIIP_N2. i. In one example, the weight value may be calculated according to the coding mode of the neighboring blocks. (i) In one example, the top and left neighboring blocks (such as Fig.50 The weight value is calculated based on the encoding and decoding mode shown in the figure, as shown below: If the top neighbor is available and is encoded and decoded by IBC, set isIBCTop to 1, otherwise set isIBCTop to 0; If the left neighbor is available and is encoded and decoded by IBC, isIBCLeft is set to 1, otherwise isIBCLeft is set to 0; If (isIBCLeft + isIBCTop) is equal to 2, set wt to 1; Otherwise, if (isIBCLeft + isIBCTop) is equal to 1, set wt to 2. In other cases, set wt to 3. f. In CIIP_N mode, select one of the candidates in the IBC merge list for IBC prediction. An IBC candidate index may be signaled to specify which of the IBC merge candidates to use. i. In one example, an IBC candidate index is signaled to specify which of the top N IBC merge candidates to use. (i) In one example, N is set to 4. (ii) In one example, N is set to the valid number of IBC merge candidates in the IBC merge list. (iii) In one example, N may be signaled from the encoder to the decoder. (iv) In one example, N is set to the number of complete RDs merged for IBC. (v) In one example, N is set to the number of complete RDs merged for IBC plus an integer. (vi) In one example, IBC merge candidates may be reordered before use. ii. In one example, an IBC candidate index is signaled to specify which one to use among the top N IBC merge candidates in ascending order of SATD cost value. (i) In one example, N is set to 4. (ii) In one example, N is set to the valid number of IBC merge candidates in the IBC merge list. (iii) In one example, N may be signaled from the encoder to the decoder. (iv) In one example, N is set to the number of complete RDs merged for IBC. (v) In one example, N is set to the number of complete RDs merged for IBC plus an integer. (vi) In one example, the IBC merge candidates may be reordered before calculating the SATD cost. g. In CIIP_N2 mode, select one of the candidates in the regular merge list for inter prediction. A merge candidate index may be signaled to specify which of the regular merge candidates to use. i. In one example, a merge candidate index is signaled to specify which of the top N regular merge candidates to use. (i) In one example, N is set to 4. (ii) In one example, N is set to the effective number of regular merge candidates in the regular merge list. (iii) In one example, N may be signaled from the encoder to the decoder. (iv) In one example, N is set to the number of complete RDs for inter-frame merging. (v) In one example, N is set to the number of complete RDs for inter-frame merging plus an integer. (vi) In one example, regular merge candidates may be reordered before use. ii. In one example, a merge candidate index is signaled to specify which of the top N regular merge candidates to use in ascending order of SATD cost value. (i) In one example, N is set to 4. (ii) In one example, N is set to the effective number of regular merge candidates in the regular merge list. (iii) In one example, N may be signaled from the encoder to the decoder. (iv) In one example, N is set to the number of complete RDs for inter-frame merging. (v) In one example, N is set to the number of complete RDs for inter-frame merging plus an integer. (vi) In one example, regular merge candidates may be reordered before calculating the SATD cost. h. In one example, whether and / or how to use CIIP_N mode may depend on codec information such as block size / QP / neighboring block mode, etc. i. In one example, when a block is encoded and decoded in IBC merge mode, if the block contains at least P luma samples (i.e., the block width multiplied by the block height is equal to or greater than P), an additional flag is transmitted by signaling to indicate whether CIIP_N mode is applied to the current block. (i) In one example, P is set to 64. ii. In one example, when a block is encoded and decoded in IBC merge mode, if both the block width and the block height are less than Q luma samples, an additional flag is signaled to indicate whether CIIP_N mode is applied to the current block. (i) In one example, Q is set to 128. (ii) In one example, Q is set to 64. iii. In one example, the above two conditions can be used together. i. In one example, whether and / or how to use CIIP_N2 mode may depend on codec information such as block size / QP / neighboring block mode, etc. i. In one example, when a block is encoded and decoded in normal merge mode, if the block contains at least P luma samples (i.e., the block width multiplied by the block height is equal to or greater than P), an additional flag is transmitted by a signal to indicate whether CIIP_N2 mode is applied to the current block. (i) In one example, P is set to 64. ii. In one example, when a block is encoded and decoded in normal merge mode, if both the block width and the block height are less than Q luma samples, an additional flag is signaled to indicate whether CIIP_N2 mode is applied to the current block. (i) In one example, Q is set to 128. (ii) In one example, Q is set to 64. iii. In one example, the above two conditions can be used together. 3. In one example, a triangulation mode may be supported for IBC prediction (referred to as TPM_IBC). a. When this mode is used, the block is evenly split into two triangle-shaped partitions using diagonal split or anti-diagonal split ( Fig.51 ). b. Each triangulation in the block is predicted by IBC using its own motion. c. The uni-directional prediction motion for each partition is derived from the uni-directional prediction IBC candidate list. i. In one example, the unidirectional prediction IBC candidate list is directly derived from part or all of the IBC merge candidate list. ii. In one example, the unidirectional prediction IBC candidate list may be reordered before use. d. After predicting each triangulation, a blending process with adaptive weights can be used to adjust the sample values ​​along diagonal or anti-diagonal edges. i. In one example, the weight can be adaptively determined by the distance between the sample and the segmentation line. (i) Section 2.23.2 shows an example. ii. Alternatively, the sample values ​​along a diagonal or anti-diagonal edge may not be adjusted using a blending process. Instead, the samples along a diagonal or anti-diagonal edge can only be predicted by one of the two predictions. e. For TPM_IBC's motion field storage, if sType is equal to 0 or 1, store Mv1 or Mv2 in the corresponding motion field, otherwise if sType is equal to 2, store Mv2. i. In one example, the sType calculation is the same as that for the inter-frame TPM. f. For TPM_IBC signaling, the following applies: i.TPM_IBC mode is signaled using CU level flags as an IBC merge mode. ii. If TPM_IBC mode is used for the current CU, a flag indicating the direction of the triangulation (diagonal or anti-diagonal) and two merge indices (one for each partition) are further signaled. iii. If TPM_IBC mode is used for the current CU, a triangulation partitioning index indicating the partitioning mode of triangulation partitioning and two merge indices (one index for each partition) are further signaled. 4. In one example, a geometric partitioning mode for IBC prediction (referred to as GPM_IBC) may be supported. a. When using this mode, the block is split into two parts by a geometrically positioned straight line (e.g. Fig.34 ). The location of the dividing lines is derived mathematically from the angle and offset parameters of the particular division. b. Each part of the geometric partition in the block is predicted by IBC using its own motion. c. The uni-directional prediction motion for each partition is derived from the uni-directional prediction IBC candidate list. i. In one example, the unidirectional prediction IBC candidate list is directly derived from part or all of the IBC merge candidate list. ii. In one example, the unidirectional prediction IBC candidate list may be reordered before use. d. After predicting each part of the geometric partition, a blending process with adaptive weights can be used to adjust the sample values ​​along the edges of the geometric partition. i. In one example, the weight can be adaptively determined according to the distance between the sample and the segmentation line. (i) Section 2.24.2 shows an example. ii. Alternatively, the sample values ​​along the edges of geometric partitions may not be adjusted using the blending process. Instead, the samples along the edges of geometric partitions can only be predicted by one of the two predictions. e. For the motion field storage of GPM_IBC, if sType is equal to 0 or 1, Mv1 or Mv2 is stored in the corresponding motion field, otherwise if sType is equal to 2, Mv2 is stored. i. In one example, the sType calculation is the same as that for inter-frame GPM. f. For GPM_IBC signaling, the following applies: i.GPM_IBC mode is signaled using CU level flags as an IBC merge mode. ii. If GPM_IBC mode is used for the current CU, a geometric partitioning index indicating the partitioning mode (angle and offset) of the geometric partitioning and two merge indices (one for each partition) are further signaled. 5. In one example, TM_AMVP for IBC is supported (referred to as TM_AMVP_IBC). a. In TM_AMVP_IBC mode, K IBC MVP candidates are determined based on template matching to select the candidates that achieve the top K minimum differences between the current block template and the reference block template from the IBC AMVP list. i. The selected set of starting point candidates consists of K IBC MVP candidates. b. TM can be performed only on a selected set of starting point candidates for MV refinement. i.TM refines the starting point candidates within the search range starting from full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode). (i) In one example, this is refined to a [–8, +8] pixel search range using an iterative diamond search. M rounds of searching are performed until the center search point has Fig.52A Minimum matching cost for the diamond search style shown. a) In one example, M is MAX_UINT. b) In one example, M is 375. ii The selected starting point candidates can be further refined by using a cross search with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode). (i) In one example, a search is performed such that Fig.52B Crosshair search style shown. c. In one example, TM_AMVP_IBC may generate K refined IBC AMVP candidates, and one of them may be selected and the selection may be signaled from the encoder to the decoder. In one example, K=1, and no selection information is transmitted via the signal. ii In one example, if at least one refined IBC AMVP candidate obtained by template matching is available, they are used as TM_AMVP_IBC candidates. Otherwise, the first K existing unrefined IBC AMVP candidates are used. iii. In one example, the selected refined IBC AMVP candidate is used as a starting point for a local search of the block matching based IBC mode. d. Alternatively, the TM_IBC derived BV is used as a starting point for a local search of the block matching based IBC mode. e. For example, when IBC AMVR is enabled, a refined IBC AMVP candidate at one MVD precision may be reused at another MVD precision. i. In one example, the refined IBC AMVP candidates at full-pixel MVD precision can be reused at 4-pixel MVD precision. 6. In one example, TM_merge for IBC is supported (referred to as TM_merge_IBC). a. In TM_merge_IBC mode, K IBC merge candidates are determined based on template matching to select the candidates that achieve the top K minimum differences between the current block template and the reference block template from the IBC merge list. i. The selected set of starting point candidates consists of K IBC merge candidates. b. TM can be performed only on a selected set of starting point candidates for MV refinement. i.TM refines the starting point candidates within the search range starting from full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode). (i) In one example, this is refined to a [–8, +8] pixel search range using an iterative diamond search. M rounds of searching are performed until the center search point has Fig.52A Minimum matching cost for the diamond search style shown. a) In one example, M is MAX_UINT. b) In one example, M is 375. ii. The selected starting point candidates can be further refined by using a cross search with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode). (i) In one example, Fig.52B The search round shown is for a cross search style. c. In one example, TM_merge_IBC can generate K refined IBC merge candidates, and One of them may be selected and the selection may be signaled from the encoder to the decoder. In one example, K=1, and no selection information is transmitted via the signal. ii In one example, if at least one refined IBC merge candidate obtained by template matching is available, they are used as TM_merge_IBC candidates. Otherwise, TM_merge_IBC mode is invalid. iii. In one example, the best TM-refined IBC merge candidate is selected by criteria. (i) In one example, the criterion is an RD decision. d. Alternatively, TM is performed for MV refinement for each IBC merge candidate. Then the best TM-refined IBC merge candidate is selected according to the criteria. i. In one example, the criterion is an RD decision. 2.30. IBC with Template Matching It is proposed to also use template matching with IBC for both IBC merge mode and IBC AMVP mode. In this proposal, the IBC-TM merge list has been modified compared to the merge list used by the regular IBC merge mode so that candidates are selected according to a pruning method, where the motion distance between candidates is as in the regular TM merge mode. The ending zero motion completion (which is meaningless with respect to intra codecs) has been replaced by the motion vector to the left of the CU (-W, 0), the motion vector at the top (0, -H) and the motion vector at the top left corner (-W, -H), then the list is completed with this left side without pruning if necessary. In IBC-TM merge mode, the selected candidates are refined with template matching method before RDO or decoding process. IBC-TM merge mode has been competed with regular IBC merge mode and TM merge flag is signaled. In IBC-TM AMVP mode, up to 3 candidates are selected from the IBC merge list. Each of th...

Claims

1. A method for video processing, comprising: For conversion between a current video block of a video and a bitstream of the video, determining that a first prediction mode and a merge mode are applied to the current block; In the prediction mode and the merge mode, deriving a block vector for the current video block based on at least one motion candidate and block vector difference information; as well as The conversion is performed based on the block vector.

2. The method of claim 1, wherein in the first prediction mode, prediction samples for the current video block are derived from a block of sample values ​​of the same video region determined by the block vector.

3. The method according to claim 1, wherein the at least one block vector candidate comprises at least one of the following: Basic candidates, Merge candidates. The method of claim 1 , wherein the at least one motion candidate is determined from a motion candidate list based on at least one indicator. The method of claim 4 , wherein the at least one base candidate comprises a base candidate, and the motion candidate list is a merge list. The method of claim 5 , wherein the merged list comprises a conventional merged list. 7 . The method according to claim 5 , wherein the merged list comprises a new merged list different from a regular merged list, and the new merged list is constructed based on the regular merged list, or the new merged list is constructed independently of the regular merged list. 8 . The method of claim 4 , wherein first N motion candidates of a merge list are selected to construct a base candidate list, and the at least one motion candidate is determined from the base candidate list, wherein N is an integer greater than 0. 9 . The method of claim 8 , wherein if N is equal to 1, an indicator associated with the base candidate list is omitted from the codestream.

10. The method of claim 8, wherein some or all of the candidates in the merged list are reordered according to one or more criteria before being selected. 11 . The method of claim 8 , wherein some or all of the candidates in the base candidate list are reordered according to one or more criteria, and the at least one motion candidate is determined from the reordered base candidate list.

12. The method of claim 4, wherein if a candidate is invalid, the invalid candidate will not be inserted into the motion candidate list; or The invalid candidate will not be selected as one of the at least one motion candidate.

13. The method of claim 4, wherein if a candidate is invalid, one or two components of the invalid candidate are clipped to a reference region, and the clipped candidate is inserted into the motion candidate list and / or is considered as a base candidate.

14. The method of claim 1, wherein the block vector difference information comprises at least a first index specifying displacement magnitude information indicating a predefined offset from a starting point of a base block vector (BV). 15 . The method of claim 14 , wherein the starting point of the base BV is determined based on a base candidate selected from the at least one motion candidate. The method of claim 14 , wherein the starting point of the base BV is determined based on a template candidate.

17. The method of claim 14, wherein in the first prediction mode and the merge mode, determining a block vector for the current video block based on at least one motion candidate and block vector difference information comprises: A final BV is derived based on the starting point and the displacement amplitude information of the basic BV.

18. The method of claim 14, wherein deriving a final BV based on the starting point and the displacement amplitude information comprises: An offset indicated by the displacement magnitude information is added to at least one of a horizontal component or a vertical component of the BV to derive the final BV. The method of claim 14 , wherein the first index indicates a target distance selected from a set of predefined distances.

20. The method according to claim 19, wherein the predefined distance set comprises at least one of the following: {1px, 2px, 4px, 8px, 16px, 32px}, {1px, 2px, 4px, 8px, 16px, 32px, 64px, 128px}, {1px, 2px, 3px, 4px, 6px, 8px, 16px}, {1px, 2px, 3px, 4px, 6px, 8px, 16px, 32px, 64px}, {1px, 2px, 4px, 8px, 16px, 32px, 48px, 64px, 80px, 96px, 112px, 128px}, {1px, 2px, 4px, 8px, 12px, 16px, 24px, 32px, 40px, 48px, 56px, 64px, 72px, 80px, 88px, 96px, 104px, 112px, 120px, 128px}, {1px, 2px, 3px, 4px, 6px, 8px, 10px, 12px, 16px, 20px, 24px, 32px, 40px, 48px, 56px, 64px, 72px, 80px, 88px, 96px, 104px, 112px, 120px, 128px}, or {1px, 2px, 4px, 8px, 12px, 16px, 20px, 24px, 28px, 32px, 36px, 40px, 44px, 48px, 52px, 56px, 60px, 64px, 68px, 72px, 76px, 80px, 84px, 88px, 92px, 96px, 100px, 104px, 108px, 112px, 116px, 120px, 124px, 128px}.

21. The method of claim 14, wherein the relationship of the first index to the predefined offset is transmitted via a signal at a specific level, the specific level comprising at least one of: a sequence level, a picture level, a slice level, a codec tree unit (CTU) level, or a codec unit (CU) level.

22. The method of claim 1, wherein the block vector difference information comprises at least a second index specifying direction information indicating a direction of a block vector difference (BVD) relative to a starting point of a base block vector (BV).

23. The method of claim 22, wherein the starting point of the base BV is determined based on a base candidate selected from the at least one motion candidate.

24. The method of claim 22, wherein the starting point of the base BV is determined based on a template candidate.

25. The method of claim 22, wherein determining the block vector comprises: A final BV is derived based on the starting point and the direction information of the base BV.

26. The method of claim 22, wherein the block vector difference information further includes a first index specifying displacement magnitude information, the displacement magnitude information indicating a predefined offset from the starting point of the base BV, and deriving a final BV based on the starting point and the direction information of the base BV comprises: The final BV is derived based on the starting point, the direction information and the displacement amplitude information of the basic BV.

27. The method of claim 27, wherein deriving the final BV based on the starting point, the direction information, and the displacement amplitude information of the basic BV comprises: Determine an offset BV based on the direction information and the displacement amplitude information; as well as Based on the starting point of the base BV and the offset BV, the final BV is derived.

28. The method of claim 22, wherein the second index indicates M BVD directions, and M is an integer greater than 0.

29. The method of claim 28, wherein the M BVD directions include: 4 horizontal / vertical directions, 4 diagonal directions, or 4 horizontal / vertical directions plus 4 diagonal directions.

30. The method of claim 22, wherein: If the second index is equal to the first value, then the BVD direction is (offset, 0); If the second index is equal to the second value, the BVD direction is (-offset, 0); If the second index is equal to a third value, then the BVD direction is (0, offset); and / or If the second index is equal to a fourth value, the BVD direction is (0, -offset).

31. The method of claim 22, wherein: If the second index is equal to the fifth value, then the BVD direction is (offset, offset); If the second index is equal to the sixth value, then the BVD direction is (offset, -offset); If the second index is equal to the seventh value, then the BVD direction is (-offset, offset); and / or If the second index is equal to the eighth value, the BVD direction is (-offset, -offset).

32. The method of claim 1 , wherein the block vector difference information indicates displacement amplitude information and / or direction information, the displacement amplitude information indicating a predefined offset from a starting point of a base block vector (BV), the direction information indicating a direction of a block vector difference (BVD) relative to the starting point of the base BV, and wherein the displacement amplitude information and / or the direction information are encoded and decoded in the same manner as the displacement amplitude information and / or the motion vector difference (MVD) direction for a merge mode with motion vector difference (MMVD).

33. The method of claim 1 , wherein the block vector difference information comprises at least a third index specifying both displacement magnitude information and direction information, the displacement magnitude information indicating a predefined offset from a starting point of a base block vector (BV), the direction information indicating a direction of a block vector difference (BVD) relative to the starting point of the base BV.

34. The method of claim 34, wherein the third index is encoded using one of: a unary code, a truncated unary code, a binary code, a truncated binary code, an exponential Golomb code, a truncated exponential Golomb code, a fixed length code, a bypass mode, or a context mode.

35. The method of claim 1, wherein the at least one motion candidate includes at least a first motion candidate, and a first set of merge mode with block vector difference (MBVD) candidates corresponding to the first motion candidate are reordered according to at least one criterion.

36. The method of claim 1, wherein the number of distances and / or directions for MBVD of a block is determined based on at least one of: The resolution of the image, Configuration of the encoding and decoding process, or A set of BVDs of a set of neighboring blocks of the block.

37. The method of claim 36, wherein the set of neighboring blocks includes an above neighboring block and a left neighboring block.

38. The method of claim 36, wherein the set of neighboring blocks comprises at least one of: The left neighboring block, The upper neighboring block, The upper right neighboring block, The lower left neighbor, or Top left neighbor block.

39. The method of claim 1, wherein a BVD direction is determined based on a BV direction of the at least one motion candidate.

40. The method of claim 40, wherein the BVD direction is the same as or opposite to the BV direction of the at least one motion candidate.

41. The method of claim 1, wherein the number of distances and / or directions of a block for MBVD is transmitted via a signal at a specific level, the specific level comprising at least one of: a sequence level, a picture level, a slice level, a codec tree unit (CTU) level, or a codec unit (CU) level.

42. The method of claim 1, wherein if an invalid block vector is to be derived based on a motion candidate and / or block vector difference information, the motion candidate and / or the block vector difference information is not selected and / or signaled.

43. The method of claim 1, wherein determining a block vector based on at least one motion candidate and block vector difference information comprises: If the block vector is in an invalid range, clipping the block vector to a valid range; as well as The block vector is determined based on the clipped block vector.

44. The method of claim 1, wherein a target flag is signaled to indicate whether MBVD merge mode is used or normal merge mode is used.

45. The method of claim 44, wherein the target flag is context-encoded.

46. ​​The method of claim 44, wherein if IBC-TM merge mode is used, the target flag is encoded before the IBC-TM merge flag.

47. The method of claim 44, wherein if IBC-TM merge mode is used, the target flag is encoded after the IBC-TM merge flag.

48. The method of claim 44, wherein the target flag is signaled in a CU level.

49. The method of claim 1, wherein the template matching refinement process is performed using only a cross search style.

50. The method according to any one of claims 1-49, wherein the converting comprises encoding the current video block into the codestream.

51. The method of any one of claims 1-49, wherein the converting comprises decoding the current video block from the codestream.

52. An apparatus for processing video data, comprising a processor and a non-volatile memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method according to any one of claims 1-51.

53. A non-transitory computer-readable storage medium storing instructions for causing a processor to execute the method according to any one of claims 1-51.

54. A non-transitory computer-readable recording medium storing a code stream of a video generated by a method performed by a video processing device, wherein the method comprises: For conversion between a current video block of a video and a bitstream of the video, determining that a first prediction mode is applied to the current block; In the prediction mode, determining a block vector for the current video block based on at least one motion candidate and block vector difference information; as well as The code stream is generated based on the motion information.

55. A method for storing a video code stream, comprising: For conversion between a current video block of a video and a bitstream of the video, determining that a first prediction mode is applied to the current block; In the prediction mode, determining a block vector for the current video block based on at least one motion candidate and block vector difference information; as well as The code stream is stored in a non-transitory computer-readable recording medium.

Images