Improvements to template matching based sub-block motion refinement for motion prediction

Abstract

Methods and systems are provided that can enable application of template matching based motion refinement to sub-blocks of a coding block. A VVC standard compliant encoder and a VVC standard compliant decoder can configure one or more processors of a computing system to obtain a sub-template for each sub-block of a coding block from respective motion vectors of neighboring sub-blocks.

Description

[0001] Cross-references to related applications This disclosure claims priority to U.S. Provisional Application No. 63 / 619,148, filed January 9, 2024, and U.S. Non-Provisional Application No. 19 / 008,482, filed January 2, 2025. All of the above applications are incorporated herein by reference in their entirety. Background Technology

[0002] In 2020, the Joint Video Experts Team (JVET) of the ITU-T Video Coding Expert Group (ITU-TVCEG) and the ISO / IEC Moving Picture Expert Group (ISO / IEC MPEG) released the final draft of the next-generation video codec specification, Versatile Video Coding (VVC). This specification further improves video coding performance compared to existing standards such as H.264 / AVC (Advanced Video Coding) and H.265 / HEVC (High-Efficiency Video Coding). JVET also proposed additional techniques beyond the scope of the VVC standard itself, collectively known as the Enhanced Compression Model (ECM).

[0003] In the VVC standard, the encoder and decoder perform inter-frame prediction using motion vectors (MVs). Inter-frame prediction can use bidirectional prediction, where two motion vectors, each pointing to its respective reference image, are used to generate the prediction value for the current block. Bidirectional prediction can be performed on the current CU based on decoder-side motion vector refinement (DMVR), such that the motion information of the current CU includes a weighted average of the two prediction signals, with weighted indices derived from neighboring blocks based on a merged candidate index.

[0004] Furthermore, at the time of writing, the latest draft of ECM (submitted as "Algorithmic Description of Enhanced Compression Model 15 (ECM 15)" at the 36th JVET meeting in November 2024) includes a proposal to further introduce a new merge candidate list for adaptive decoder-side motion vector correction. Template matching is performed to correct the motion vectors, and the merge candidate list is reordered. Additionally, the subblock merge candidate list is implemented based on subblock-based temporal motion vector prediction (SbTMVP).

[0005] To correct the sub-block motion vector, it is necessary to further improve the sub-block merging candidate list. Attached Figure Description

[0006] Detailed descriptions of the embodiments will be given with reference to the accompanying drawings. In the drawings, one or more numbers on the far left of the reference numerals indicate the figure in which the reference numeral first appears. The same reference numerals are used in different figures to indicate similar or identical items or features.

[0007] Figure 1A and 1B Exemplary block diagrams are shown for the encoding and decoding processes according to exemplary embodiments of the present disclosure.

[0008] Figure 2A and 2B Schematic diagrams of affine motion fields for blocks with four parameters (two control points) and six parameters (three control points) are shown respectively. Although various types of motion exist in the real world (e.g., scaling), block-based affine transformation motion compensation prediction is employed in the VVC standard.

[0009] Figure 3 A schematic diagram is shown of the affine motion vector of the brightness sub-block calculated for the center sample of each sub-block.

[0010] Figure 4 This illustrates the derivation of sub-block motion vectors based on the sub-block-based temporal motion vector prediction (“SbTMVP”) pattern.

[0011] Figure 5 The diagram illustrates the inheritance of control point motion vectors, with candidates from the inherited control point motion vectors used to form an affine merging candidate list.

[0012] Figure 6A and 6B The diagrams show the candidate for non-adjacent neighbor block inheritance used in the affine merge candidate list and the first type of candidate for constructing the affine merge candidate list, respectively.

[0013] Figure 7 This diagram illustrates the positions of affine construction candidates derived from adjacent blocks in the affine merging candidate list.

[0014] Figure 8 This diagram illustrates the positions of affine construction candidates derived from non-adjacent blocks used in the affine merging candidate list.

[0015] Figure 9 A schematic diagram of adjacent 4×4 sub-block information is shown. Motion information is obtained from the adjacent 4×4 sub-block information to perform affine model regression of the current coding block.

[0016] Figure 10 The diagram illustrates motion prediction performed on a current image using bidirectional prediction, wherein an offset block of a reference image is used to calculate a corrected motion vector, which is then used to generate a bidirectional prediction signal.

[0017] Figure 11 An example diagram showing the search pattern used in the first round of motion vector correction at the multi-round decoding end.

[0018] Figure 12 A schematic diagram illustrating the bilateral matching cost used in the second round of motion vector correction at the multi-round decoding end.

[0019] Figure 13 This demonstrates the search for the closest match between a template in the current image and a block in the reference image.

[0020] Figure 14A and 14B This shows the diamond search pattern used in template matching.

[0021] Figure 15 A flowchart illustrating the template-based correction process is shown.

[0022] Figure 16 This illustrates motion prediction performed on the current image 1602 using bidirectional prediction.

[0023] Figure 17A and 17B This shows a reference sample for each sub-template derived from the motion information of the sub-blocks in the first row and first column of the current block.

[0024] Figure 18 A flowchart illustrating the template-based reordering process applied in TM merge mode is shown.

[0025] Figure 19 This shows the sub-templates and sub-blocks of the encoded block.

[0026] Figure 20This shows a reference sub-template obtained by exporting the MV of each sub-template of the upper boundary sub-block, each sub-template of the left boundary sub-block, or each sub-template of both.

[0027] Figure 21 This shows that for each coded block, each sub-block has a different MV and a different prediction direction.

[0028] Figure 22A , 22B 22C illustrates a template for deriving a reference image list 0, a template for deriving a reference image list 1, and a final template for deriving the templates of reference image list 0 and reference image list 1, according to exemplary embodiments of the present disclosure.

[0029] Figure 23A , 23B 23C illustrates a template for deriving a reference image list 0, a template for deriving a reference image list 1, and a final template for deriving the templates of reference image list 0 and reference image list 1, according to another exemplary embodiment of the present disclosure. Figure 23D , 23E Figures 23F and 23F illustrate another exemplary embodiment according to this disclosure. Figure 23A , 23B An alternative to the export of 23C.

[0030] Figure 24 This shows a top template for a list of reference images, from which its individual sub-templates are obtained.

[0031] Figure 25 This shows how a weighted average is calculated for sub-templates and overlapping sub-templates when merging sub-templates to obtain a template.

[0032] Figure 26 The upper boundary sub-block and the left boundary sub-block of the coded block are shown to have their own motion vectors, which are used to obtain the sub-template.

[0033] Figure 27 This shows how to add the same MV offset dMV to the top boundary sub-block and the left boundary sub-block MV to obtain a sub-template.

[0034] Figure 28 This shows that the MV offset dMV is added to all child blocks MV.

[0035] Figure 29A and 29B This shows that the search patterns are divided into integer search patterns and fractional search patterns.

[0036] Figure 30An exemplary system is shown for implementing the process and method described herein for carrying out template matching-based motion correction. Detailed Implementation

[0037] The systems and methods discussed in this paper involve implementing a candidate list for merging sub-blocks for motion prediction, and more specifically, applying template-matching-based motion corrections to sub-blocks of a coded block.

[0038] According to the VVC video coding standard (“VVC Standard”) and the motion prediction described therein, a computing system includes at least one or more processors and a computer-readable storage medium communicatively coupled to said one or more processors. A computer-readable storage medium is a non-transient or non-temporary computer-readable storage medium that stores computer-readable instructions, as described below. Figure 30 As defined, at least a portion of computer-readable instructions stored on a computer-readable storage medium are executable by one or more processors of a computing system to configure said one or more processors to perform associated operations of the computer-readable instructions, including at least the operations of an encoder as described in the VVC standard and the operations of a decoder as described in the VVC standard. Certain encoder and decoder operations performed according to the VVC standard are then described in further detail, but these subsequent descriptions should not be construed as an exhaustive list of encoder and decoder operations according to the VVC standard. Subsequently, “VVC-compliant encoder” and “VVC-compliant decoder” will describe the corresponding computer-readable instructions stored on the computer-readable storage medium that configure one or more processors to perform these corresponding operations (as an example, a “reference embodiment” may be referred to as an encoder or decoder).

[0039] Furthermore, according to exemplary embodiments of this disclosure, the VVC-compliant encoder and VVC-compliant decoder further include computer-readable instructions stored on a computer-readable storage medium, which are executable by one or more processors of a computing system to configure the one or more processors to perform operations not specified in the VVC standard. The VVC-compliant encoder should not be construed as limited to the operation of the reference embodiment of the encoder, but includes additional computer-readable instructions described herein for configuring one or more processors of the computing system to perform further operations. Similarly, the VVC-compliant decoder should not be construed as limited to the operation of the reference embodiment of the decoder, but includes additional computer-readable instructions described herein for configuring one or more processors of the computing system to perform further operations.

[0040] Figure 1A and 1B Exemplary block diagrams of encoding process 100 and decoding process 150 according to exemplary embodiments of the present disclosure are shown respectively.

[0041] In the encoding process 100, a VVC-compliant encoder configures one or more processors of the computing system to receive one or more input images as input from an image source 102. The input images comprise a number of pixels sampled by an image acquisition device such as a photoelectric sensor array, and include an uncompressed multi-color channel (e.g., RGB color channels) data stream in which color data is stored at the original resolution of the image, wherein each channel uses a certain number of bits to store the color data of each pixel of the image. The VVC-compliant encoder configures one or more processors of the computing system to store this uncompressed color data in a compressed format, wherein the color data is stored at a resolution lower than the original resolution of the image and encoded as a luminance (“Y”) channel and two chrominance (“U” and “V”) channels with a resolution lower than the luminance channel.

[0042] A VVC-compliant encoder encodes an image by configuring one or more processors of a computing system to segment the original image into units and subunits according to a partitioning structure. The image being encoded is referred to as the "current image," distinguishing it from any other image received from image source 102. The VVC-compliant encoder further subdivides the image into macroblocks ("macroblocks," MB) of 16×16 pixels, which can be further subdivided into partitions. Alternatively, the encoder may configure one or more processors to subdivide the image into coding tree units ("coding tree units," CTUs, whose luminance and chrominance components can be further subdivided into coding tree blocks ("coding tree blocks," CTBs), which are further subdivided into coding units ("coding units," CUs). Optionally, the encoder may further subdivide the image into N×N pixel units, which can then be further subdivided into subunits. In this disclosure, each of these largest subdivision units of the image may generally be referred to as a "block."

[0043] When the image is not monochrome and is encoded using a coding tree, the CU uses one luminance sample block and two corresponding chrominance sample blocks for encoding.

[0044] A VVC-compliant encoder configures one or more processors in a computing system to subdivide a block into partitions, each partition being a multiple of 4×4 pixels in size. For example, a partition of a block could be 8×4 pixels, 4×8 pixels, 8×8 pixels, 16×8 pixels, or 8×16 pixels in size.

[0045] Instead of encoding the color information of the pixels in the full-resolution original image, the VVC-compliant encoder configures one or more processors of the computing system to encode the color information of the image at a resolution lower than that of the input image, thereby storing the color information in fewer bits than the input image.

[0046] Furthermore, encoders compliant with the VVC standard encode images by configuring one or more processors of the computing system to perform motion prediction on multiple blocks of the current image. Motion prediction coding refers to storing image data (wherein the block of the original image is referred to as the "input block" before encoding) of a block of the current image using motion information and a prediction unit ("prediction unit, PU") based on intra-frame prediction 104 or inter-frame prediction 106, rather than pixel data.

[0047] Motion information refers to data describing the motion of a block structure or its units or subunits in an image, such as motion vectors and reference information to blocks in the current or reference image. A PU can refer to a unit or subunits corresponding to one of multiple block structures in an image, such as an MB or CTU, where the block is segmented based on image data and encoded according to the VVC standard. The motion information corresponding to the PU can describe motion predictions encoded by an encoder conforming to the VVC standard, as described herein.

[0048] A VVC-compliant encoder configures one or more processors of a computing system to encode motion prediction information for each block of an image in an encoding order similar to that of a raster scan, where the first block decoded is the topmost and leftmost block of the image. The block being encoded is called the "current block" to distinguish it from any other block in the same image.

[0049] According to intra-frame prediction 104, one or more processors of the computing system are configured to encode blocks by referencing motion information and PUs of one or more other blocks in the same image. According to intra-frame prediction coding, one or more processors of the computing system perform intra-frame prediction 104 (also known as spatial prediction) computation by encoding the motion information of the current block based on spatial neighbor samples from spatially neighboring blocks of the current block.

[0050] According to inter-frame prediction 106, one or more processors of the computing system are configured to encode blocks by referencing motion information and PU from one or more other images. One or more processors of the computing system are configured to store one or more previously encoded and decoded images in a reference image buffer for use in inter-frame prediction coding; these stored images are called reference images.

[0051] One or more processors are configured to perform inter-frame prediction (also known as temporal prediction or motion-compensated prediction) operations by encoding motion information of the current block based on samples from one or more reference images. The inter-frame prediction can be further computed based on unidirectional or bidirectional prediction: in unidirectional prediction, a prediction signal for the current block is generated using only one motion vector pointing to a reference image. In bidirectional prediction, a prediction signal for the current block is generated using two motion vectors, each pointing to a corresponding reference image.

[0052] A VVC-compliant encoder configures one or more processors of the computing system to encode the CU (Cubic Frame) to include a reference index for identifying the prediction signal of the current block, for reference by the VVC-compliant decoder. The one or more processors of the computing system can encode the CU to include an inter-frame prediction indicator. The inter-frame prediction indicator indicates: performing list 0 prediction with reference to a first reference image list (referred to as list 0), performing list 1 prediction with reference to a second reference image list (referred to as list 1), or performing bidirectional prediction with reference to both reference image lists (referred to as list 0 and list 1, respectively).

[0053] When the inter-frame prediction indicator indicates prediction for list 0 or list 1, one or more processors of the computing system are configured to encode a CU, the CU including a reference index indicating a reference image of the reference image buffer referenced by list 0 or list 1, respectively. When the inter-frame prediction indicator indicates bidirectional prediction, one or more processors of the computing system are configured to encode a CU including a first reference index and a second reference index, the first reference index indicating a first reference image of the reference image buffer referenced by list 0, and the second reference index indicating a second reference image of the reference image buffer referenced by list 1.

[0054] A VVC-compliant encoder configures one or more processors in the computing system to individually encode each current block of the image, thereby outputting their respective prediction blocks. In the VVC standard, the CTU can be as large as 128×128 luma samples (plus the corresponding chroma samples, depending on the chroma format). The CTU can be further segmented into CUs according to a quadtree, binary tree, or ternary tree. One or more processors in the computing system are configured to ultimately record a set of coding parameters in the leaf node syntax structure of the segmentation structure, such as the coding mode (intra-frame mode or inter-frame mode), motion information of the inter-frame coded blocks (reference index, motion vectors, etc.), and quantization residual coefficients.

[0055] After the output prediction block, the VVC-compliant encoder configures one or more processors of the computing system to send a set of coding parameters, such as coding mode (i.e., intra-frame or inter-frame prediction), intra-frame prediction mode or inter-frame prediction mode, and motion information to the entropy encoder 124 (as described below).

[0056] The VVC standard provides semantics for recording the encoding parameter set of a CU. For example, regarding the aforementioned encoding parameter set, the CU's... Set to 0 for inter-frame coded blocks and 1 for intra-frame coded blocks; CU's Set to indicate whether to use merge mode in inter-frame prediction of the CU; CU's and Set to indicate whether affine motion compensation is used in the inter-frame prediction of the CU; and Set to indicate the motion vector index in list 0 or list 1 respectively; and These are set to indicate the reference image index in list 0 or list 1, respectively. It should be understood that the VVC standard includes semantics for various other information, flags, and options used to record information outside the scope of this disclosure.

[0057] An encoder conforming to the VVC standard further implements one or more mode decisions and encoder control settings 108, including rate control settings. One or more processors of the computing system are configured to perform mode decisions by selecting an optimized prediction mode for the current block based on a rate-distortion optimization method after intra-frame or inter-frame prediction.

[0058] The bitrate control settings configure one or more processors of the computing system to assign different quantization parameters ("QP") to different images. The magnitude of the QP determines the scale at which the image information is quantized by one or more processors during the encoding process (as described below), and thus determines the extent to which the encoding process 100 discards image information when encoding MBs in the sequence (because information falls between the intervals of the scale and is discarded).

[0059] The encoder, conforming to the VVC standard, further implements a subtractor 110. One or more processors of the computing system are configured to perform the subtraction operation by calculating the difference between the input block and the prediction block. Based on the optimized prediction pattern, the prediction block is subtracted from the input block. The difference between the input block and the prediction block is called the prediction residual, or, for simplicity, "residual".

[0060] Based on the predicted residual, the encoder conforming to the VVC standard further implements transformation 112. One or more processors of the computing system are configured to compute a set of coefficients (which may be referred to as "residual coefficients," "transform coefficients," etc.) through matrix arithmetic operations to perform a transformation operation on the residual, thereby encoding the current block as a transform block ("transform block, TB"). Transform coefficients may refer to coefficients characterizing one of several spatial transformations, such as diagonal flip, vertical flip, or rotation, which may be applied to sub-blocks.

[0061] It should be understood that a coefficient can be stored as two components: absolute value and sign, which will be described in more detail below.

[0062] Sub-blocks of the CU, such as PU and TB, can be arranged in any combination of sub-block sizes described above. A VVC-compliant encoder configures one or more processors of the computing system to subdivide the CU into a residual quadtree ("RQT"), a hierarchical structure of the TB. The RQT provides the order of motion prediction and residual coding for its sub-blocks at each level and recursively downwards along each level of the RQT.

[0063] The encoder, conforming to the VVC standard, further implements quantization 114. One or more processors of the computing system are configured to perform quantization operations on the residual coefficients via matrix arithmetic based on the quantization matrix and the QP specified above. Residual coefficients falling within the interval are retained, while those falling outside the interval step size are discarded.

[0064] The VVC-compliant encoder further implements inverse quantization 116 and inverse transform 118. One or more processors of the computing system are configured to perform inverse quantization and inverse transform operations on the quantized residual coefficients via matrix arithmetic operations, which are the inverse operations of the aforementioned quantization and transform operations. The inverse quantization and inverse transform operations produce a reconstructed residual.

[0065] The encoder, conforming to the VVC standard, further implements adder 120. One or more processors of the computing system are configured to perform addition by adding the predicted block and the reconstructed residual, thereby outputting the reconstructed block.

[0066] The encoder, which conforms to the VVC standard, further implements loop filtering 122. One or more processors of the computing system are configured to apply loop filtering, such as deblocking filtering, sample adaptive offset (SAO) filtering, and adaptive loop filter (ALF), to the reconstructed block, thereby outputting a filtered reconstructed block.

[0067] A VVC-compliant encoder further configures one or more processors of the computing system to output filtered reconstructed blocks to a decoded picture buffer (DPB) 200. The DPB 200 stores the reconstructed images, which the one or more processors of the computing system use as reference images when encoding images, instead of using the current image as a reference image, as described above regarding inter-frame prediction.

[0068] The VVC-compliant encoder further implements an entropy encoder 124. One or more processors of the computing system are configured to perform entropy encoding, wherein, according to a context-adaptive binary arithmetic codec (“CABAC”), encoding is performed by mapping the symbols that make up the quantized residual coefficients to binary strings (hereinafter referred to as “bins”), which can be transmitted in the output bitstream at a compressed bit rate. The symbols of the encoded quantized residual coefficients include the absolute values ​​of the residual coefficients (these absolute values ​​are hereinafter referred to as “residual coefficient levels”).

[0069] Therefore, the entropy encoder configures one or more processors of the computing system to encode the residual coefficient levels of the block; performs bypass encoding on the residual coefficient symbols and records the residual coefficient symbols together with the encoded block; records the encoding parameter set, such as the encoding mode, intra-frame prediction mode or inter-frame prediction mode, and motion information encoded in the syntax structure of the encoded block (similar to the image parameter set ("PPS") in the image header and the sequence parameter set ("SPS") in multiple image sequences); and outputs the encoded block.

[0070] A VVC-compliant encoder configures one or more processors of a computing system to output an encoded image, which consists of encoded blocks from an entropy encoder 124. The encoded image is output to a transmission buffer, where it is ultimately packaged into a bitstream for output from the VVC-compliant encoder. This bitstream is written by one or more processors of the computing system to a non-transient or non-temporary computer-readable storage medium of the computing system for transmission.

[0071] In the decoding process 150, the VVC-compliant decoder configures one or more processors of the computing system to receive one or more encoded images as input from the bitstream.

[0072] A VVC-compliant decoder implements entropy decoder 152. One or more processors of the computing system are configured to perform entropy decoding, wherein binary symbols are decoded according to CABAC via an inverse symbol-to-binary symbol mapping to recover the entropy-encoded quantization residual coefficients. The entropy decoder 152 outputs the quantization residual coefficients, outputs the sign of the bypass-encoded residual coefficients, and also outputs the syntax structures, such as PPS and SPS.

[0073] The VVC-compliant decoder further implements inverse quantization 154 and inverse transform 156. One or more processors of the computing system are configured to perform inverse quantization and inverse transform operations on the decoded quantized residual coefficients via matrix arithmetic operations, which are the inverses of the aforementioned quantization and transform operations. The inverse quantization and inverse transform operations produce a reconstructed residual.

[0074] Furthermore, based on the set of encoding parameters recorded in syntax structures such as PPS and SPS by the entropy encoder 124 (or, optionally, received via out-of-band transmission or encoded into the decoder) and the encoding modes included in the set of encoding parameters, the VVC-compliant decoder determines whether to apply intra-frame prediction 156 (i.e., spatial prediction) or motion-compensated prediction 158 (i.e., temporal prediction) to the reconstructed residual.

[0075] When intra-frame prediction is specified in the coding parameter set, a VVC-compliant decoder configures one or more processors of the computing system to perform intra-frame prediction 158 using the prediction information specified in the coding parameter set. The intra-frame prediction 158 thereby generates a prediction signal.

[0076] With the encoding parameter set specifying inter-frame prediction, a VVC-compliant decoder configures one or more processors of the computing system to perform motion-compensated prediction 160 using a reference image from the DPB 200. This motion-compensated prediction 160 thereby generates a prediction signal.

[0077] The VVC-compliant decoder further implements adder 162. Adder 162 configures one or more processors of the computing system to perform addition on the reconstructed residual and the predicted signal, thereby outputting a reconstructed block.

[0078] The VVC-compliant decoder further implements loop filtering 164. One or more processors of the computing system are configured to apply loop filtering, such as deblocking filtering, SAO filtering, and ALF, to the reconstructed blocks, thereby outputting filtered reconstructed blocks.

[0079] The VVC-compliant decoder further configures one or more processors of the computing system to output filtered reconstructed blocks to the DPB 200. As described above, the DPB 200 stores the reconstructed images, which are used as reference images by one or more processors of the computing system when encoding images, instead of using the current image; as described above regarding motion compensation prediction.

[0080] A VVC-compliant decoder further configures one or more processors of the computing system to output the reconstructed image from the DPB to a user-visual display of the computing system, such as a television display, a personal computing display, a smartphone display, or a flat panel display.

[0081] Therefore, as shown in the encoding process 100 and decoding process 150 above, the VVC-compliant encoder and decoder each perform motion prediction coding according to the VVC specification. Both the VVC-compliant encoder and decoder configure one or more processors of the computing system to generate a reconstructed image based on a previously reconstructed image of the DPB, according to motion compensation prediction as described in the VVC standard, where the previously reconstructed image serves as a reference image in the motion compensation prediction, as described herein.

[0082] A merging mode is a pattern for encoding motion information to be used in motion compensation for the current inter-frame prediction block. In this mode, motion information of the current block is inherited or borrowed from previously encoded blocks, including spatially adjacent blocks, non-adjacent blocks, temporal blocks, etc. Merging lists are constructed at both the encoder and decoder ends, with entries populated with motion information from various previously encoded blocks. Each merging list entry is called a merging candidate. The candidate selected and used for the current block in motion compensation is indicated by an index, which is signaled to the VVC-compliant decoder in the bitstream.

[0083] There are two different types of merging modes: coded block (“CB”) level merging mode (or “regular merging mode”) and sub-block merging mode. In regular merging mode, motion compensation is performed on the entire block, while in sub-block merging mode, the current coded block is divided into one or more sub-blocks, each with its own motion information; therefore, motion compensation is performed at the sub-block level. A sub-block merging list is constructed using the sub-block merging mode. There are two types of candidates in the sub-block merging list: the first type is a subblock temporal motion vector predictor (“SbTMVP”), and the second type is an affine merging candidate.

[0084] In the HEVC standard, only the translational motion model is applied to motion compensation prediction (MCP). However, real-world video content involves various types of motion, such as zooming in / out, rotation, viewpoint changes, and other irregular movements. The VVC standard employs block-based affine transformation motion compensation prediction. Figure 2A and 2B As shown, the motion vector is obtained through two control points ( Figure 2A The 4-parameter affine motion model shown) or the motion vectors of the three control points ( Figure 2B The motion information of the block is described by the affine motion field of the 6-parameter affine motion model shown in the figure.

[0085] In affine motion compensation, for a 4-parameter affine motion model, the sample position in the block... The motion vector at that point is derived according to the following formula 1: For a 6-parameter affine motion model, the sample position in the block... The motion vector at that point is derived according to the following formula 2: in, It is the motion vector of the top left control point. It is the motion vector of the upper right control point. It is the motion vector of the lower left control point.

[0086] To simplify motion compensation prediction, a block-based affine transformation prediction can be used. To derive the motion vector for each 4×4 brightness sub-block, the motion vector of the center sample of each sub-block is calculated according to Equations 1 and 2 above, as follows: Figure 3 As shown, it is rounded to 1 / 16 fractional precision. Figure 3 This diagram illustrates the affine motion vectors of the luma sub-blocks calculated for each sub-block's center sample. According to the ECM, the sub-block size is adaptively determined: if the difference in motion vectors between two adjacent luma sub-blocks is less than a threshold, the adjacent luma sub-blocks are merged into a larger sub-block. If the difference in motion vectors between two adjacent larger sub-blocks is still less than the threshold, the merging of the larger sub-blocks continues until the difference in motion vectors between the two adjacent sub-blocks is greater than the threshold, or until the sub-block equals the entire block.

[0087] After deriving the motion vectors of the sub-blocks, a motion-compensated interpolation filter is applied to generate a predicted value for each sub-block based on the derived motion vectors. The sub-block size for the chroma component depends on the size of the luma sub-block. The MV of the chroma sub-block is calculated based on the average of the MVs of the top-left and bottom-right luma sub-blocks in the co-located luma region.

[0088] In SbTMVP mode, a coding block is divided into 4×4 sub-blocks, and each sub-block obtains its own motion vector from the motion field in a co-location image. The co-location image is a reference image that has already been encoded or decoded. When obtaining the motion vector of each sub-block, a motion offset is applied, which is derived using the motion vector of the lower-left neighboring block of the current block. Figure 4 As shown, the motion offset MV is obtained from the adjacent neighboring block A1 of the current block. A1 This information is then used to derive the motion field in the co-located image. Next, for each sub-block, the motion information of that sub-block is derived using the motion information of its corresponding sub-block in the co-located image.

[0089] After identifying the motion information of the co-located sub-block, the motion information is converted into a motion vector and reference index for the current sub-block. The reference index of the sub-block is selected from any reference image in a list of reference images. The selected reference image is the one with a scaling factor closest to 1. After identifying the reference index, temporal motion scaling is applied. It should be noted that when the corresponding sub-block in the co-located image uses a non-inter-frame coding method, such as intra-frame coding or intra-block copy (IBC) coding, the motion information of the central sub-block of the coded block is used. Figure 4 As shown in the shaded area, the corresponding sub-block uses a non-inter-frame coding method: that is, the sub-block does not contain any motion information. Therefore, the motion information of the central sub-block (highlighted by a thick line) is used.

[0090] To further improve the coding efficiency of SbTMVP, two co-located images are employed, which are two reference images with the minimum point of intersection (POC) distance relative to the current image. Furthermore, instead of deriving motion offsets solely using the motion vectors of the left-side adjacent blocks, multiple locations are considered during the derivation, and these locations are adaptively determined based on the template matching (TM) cost. Specifically, two motion offset candidate lists are constructed for the two co-located images. The TM cost is calculated to reorder the candidates in the two motion offset candidate lists. Then, the SbTMVP candidate with the smaller TM cost in the two motion offset candidate lists is included in a sub-block-based merge list.

[0091] The affine merging mode (AF_MERGE) can be applied to block merging (CBs) with a width and height greater than or equal to 8. In this mode, a control point motion vector (CPMV) for the current CB is generated based on the motion information of spatially adjacent CBs. When constructing the sub-block merging list, SbTMVP candidates are added first, followed by affine merging candidates. The list can contain a maximum of fifteen candidates. The following nine types of candidates are used to construct the sub-block merging candidate list in sequence: SbTMVP candidates; candidates inherited from adjacent blocks; candidates inherited from non-adjacent blocks; candidates constructed from adjacent blocks; second-type affine construction candidates derived from non-adjacent blocks; first-type affine construction candidates derived from non-adjacent blocks; regression-based affine merging candidates; pairwise affine; and zero MV (i.e., motion vectors where all component values ​​are 0).

[0092] The inherited affine candidates are derived from the affine motion models of adjacent or non-adjacent blocks. When an adjacent or non-adjacent affine block (CB) is identified, its control point motion vector is used to derive the control point motion vector prediction (CPMVP) candidates from the affine merge candidate list of the current CB. Figure 5 As shown (a schematic diagram of control point motion vector inheritance is presented, and candidates from the inherited control point motion vectors are used to form an affine merging candidate list), if the adjacent lower left block A Encoding in affine pattern yields a block containing the block. A The motion vectors of the top left, top right, and bottom left corners of CB , and For blocks encoded using a 4-parameter affine model A ,according to and Calculate the two CPMVs of the current CB. For blocks encoded using a 6-parameter affine model. A ,according to , and Calculate the three CPMVs of the current CB.

[0093] For candidates inherited from non-adjacent blocks, the non-adjacent spatial blocks are examined from near to far based on their distance from the current block. At a specific distance, only the first available adjacent block from each side (e.g., the left and top sides) of the current block (which uses affine pattern encoding) is selected for the inheritance candidate derivation.

[0094] When the current block is located at the boundary of an image, slice, or tile, all adjacent samples on one side may be absent. Furthermore, even if the upper and left adjacent samples have been encoded or decoded before the current block, the right and lower adjacent samples may not have been encoded or decoded before the current encoded block, depending on the raster scan order. Other possible encoding orders may also alter the availability of adjacent samples along the upper, left, right, or lower edges as a whole. Therefore, this disclosure refers to the absence, non-encoded, or undecoded adjacent samples along the edges as "unavailable."

[0095] like Figure 6A As indicated by the dashed arrows, the inspection order for the adjacent blocks on the left and above is from bottom to top and from right to left, respectively. Figure 6A and 6B Candidates inherited from non-adjacent blocks are shown for the affine merge candidate list, and candidates constructed for the first type of affine merge candidate list are shown respectively.

[0096] The affine candidate constructed from neighboring blocks is the candidate constructed by fusing the adjacent translational motion information of each control point. The motion information of the control points is derived from specified spatial and temporal neighboring blocks, such as... Figure 7 As shown ( Figure 7 This shows the positions of the affine candidates built from adjacent blocks for the affine merge candidate list. Indicates the first One control point. For CPMV1, check... Block, and use the MV of the first available block. For CPMV2, check... For CPMV3, check If a temporal-based motion vector predictor (TMVP) is available, it is used as CPMV4.

[0097] After obtaining the motion values ​​(MVs) of the four control points, affine merging candidates are constructed based on the motion information. These are constructed sequentially using combinations of the following control point MVs: Combining three CPMVs constructs a 6-parameter affine merge candidate, and combining two CPMVs constructs a 4-parameter affine merge candidate. To avoid motion scaling, combinations of control point MVs are discarded if the reference indices of the control points are different.

[0098] For the first type of construction candidate originating from non-adjacent neighboring blocks, such as Figure 6BAs shown, first, the positions of a non-adjacent spatial block on the left and top are independently determined; then, the position of the upper left neighboring block can be determined accordingly, which, together with the aforementioned non-adjacent left and top neighboring blocks, can enclose a rectangular virtual block. Then, as... Figure 8 As shown, using the motion information of these three non-adjacent neighboring blocks, the top left corner CPMV 802, top right corner CPMV 804, and bottom left corner CPMV 806 of the virtual block 800 are constructed. These vectors are finally projected onto the current CB to generate corresponding construction candidates. Figure 8 This diagram illustrates the positions of affine construction candidates derived from non-adjacent blocks used in the affine merging candidate list.

[0099] For the second type of construction candidate, its affine model parameters are inherited from non-adjacent spatial neighboring blocks. Specifically, the second type of affine construction candidate is generated by a combination of the following two items: 1) the MV of a 4×4 neighboring block; and 2) such as Figure 6A The affine model parameters defined and inherited from non-adjacent spatial neighboring blocks.

[0100] For regression-based affine merging candidates, the sub-block motion fields from previously encoded affine CBs and the motion information of neighboring sub-blocks from the current CB are used as inputs to the regression process to derive the affine candidates. The previously encoded affine CBs can be identified by scanning non-adjacent locations and using an affine history-based motion vector predictor (HMVP) table. The neighboring sub-block information of the current CB is obtained from... Figure 9 The image shows the 4×4 sub-block 902 represented by the shaded area. For each sub-block, given a list of reference images, the motion vector and center coordinates corresponding to that sub-block can be used. For each affine CB, at most two regression-based affine candidates can be derived: one with neighboring sub-block information and the other without. All candidates generated by linear regression are pruned and aggregated into a candidate subgroup. When adaptive reordering of merge candidate (ARMC) is enabled, these merge candidates are reordered based on TM cost in conjunction with the template matching process. Then, when N affine CBs are found, at most N linear regression-generated candidates are added to the affine merge candidate list.

[0101] After inserting all the above merge candidates into the merge candidate list, if the list is still not full, insert zero MV at the end of the list.

[0102] Then, ARMC is applied to sort the candidates in the sub-block merge list. However, the first SbTMVP candidate is placed as the first entry without reordering, while the other SbTMVP candidates are sorted together with the other affine candidates. The number of candidates in the sub-block-based merge list is set to 30 before ARMC and 15 after ARMC.

[0103] Similar to merge mode, where motion information is directly inherited or borrowed from previously encoded blocks, these previously encoded blocks may not perfectly match the current block. Therefore, some decoder-side correction techniques are employed to correct the motion derived in merge mode. These correction techniques include decoder-side motion vector correction (“DMVR”) and TM.

[0104] The VVC standard employs DMVR based on bilateral-matching ("BM") in bidirectional prediction to improve the accuracy of the MV in the merged pattern. In DMVR, the initial MV in reference image list 0 ("L0") and reference image list 1 ("L1"), i.e. MV 0 and MV 1 Surrounding search corrects MV, wherein the corrected MV is respectively represented as and The BM method calculates the distortion between two corresponding candidate blocks in the reference images L0 and L1.

[0105] Figure 10 The diagram illustrates motion prediction performed on the current image 1002 using bidirectional prediction, where offset blocks of reference images are used to calculate corrected motion vectors, which are then used to generate bidirectional prediction signals. The current image 1002 includes a current block 1002A. According to the bidirectional prediction principle, two co-located reference images 1004 and 1006 are shown in the diagram, one from reference image list 0 in a first time domain direction and the other from reference image list 1 in a second time domain direction. The motion information of the current block 1002A points to co-located reference block 1004A of co-located reference image 1004 and also to co-located reference block 1006A of co-located reference image 1006. Co-located reference image 1004 further includes an offset block 1004B near co-located reference block 1004A, and co-located reference image 1006 further includes an offset block 1006B near co-located reference block 1006A.

[0106] like Figure 10As shown, the sum of absolute differences (SAD) between reference block 1004A and offset block 1004B, and the SAD between reference block 1006A and offset block 1006B are calculated. The candidate MV with the lowest SAD is set as the corrected MV and used to generate the bidirectional prediction signal.

[0107] Furthermore, in the VVC standard, the application of DMVR is limited to CBs encoded using the following modes and features: CB-level merging mode with bidirectional predictive MVs pointing to corresponding reference images in different temporal directions relative to the current image (i.e., one reference image in the past and the other in the future); the distances from the two reference images to the current image (i.e., the difference in picture order counts (POCs)) are the same; both reference images are short-term reference images; the current CB has more than 64 luminance samples; both the CB height and CB width are greater than or equal to 8 luminance samples; the bidirectional predictive weight index with coding unit weights (BCWs) indicates equal weights (it should be understood that, in the context of the weighted average bidirectional predictive formula for calculating the weighted average of two predictive signals, "equal weights" are weight parameters that assign equal weights to the two predictive signals in the formula); weighted bidirectional predictive (WP) is not enabled for the current block; and combined inter-intra-frame predictive (WP) is used. The "prediction, CIIP" mode is not used for the current block.

[0108] The corrected MV derived from the DMVR is used to generate the inter-frame prediction samples and also for temporal motion vector prediction for future image coding. The original MV is used for deblocking and also for spatial motion vector prediction for future CB coding.

[0109] Additional features of DMVR will be discussed later.

[0110] In DVMR, the modified MV search begins at the search center and covers the search range of modified MVs immediately adjacent to the initial MV. The span of this search range defines the search window, and the range of modified MVs searched follows the MV difference mirroring rule. In other words, the search is performed by DMVR, using candidate MV pairs... Any point represented follows Equations 3 and 4 respectively: in, The MV correction offset is defined as the MV correction offset between the initial MV and the corrected MV in one of the reference images. The corrected MV search range (hereinafter also referred to as the "search step") is a luminance sample at a distance of two integer pixels from the initial MV. The corrected MV search consists of two stages: integer sample offset search and fractional sample correction.

[0111] In order to understand the exemplary embodiments of this disclosure, all subsequent references to one or more “points” being searched should be understood as referring to individual brightness samples of a pixel block or sub-block, separated by an integer pixel distance.

[0112] A 25-point full search is applied for integer sample offset search. First, the SAD of the initial MV pair is calculated. If the SAD of the initial MV pair is less than a threshold, the integer sample stage of DMVR terminates. Otherwise, the remaining 24 search points are searched in raster scan order, and the SAD of each search point is calculated. The search point with the smallest SAD is selected as the integer pixel distance correction MV, which is output by the integer sample offset search. To reduce the adverse effects of uncertainty in DMVR correction, the original MV can be given priority during the DMVR process. The SAD value between reference blocks pointed to by the initial MV candidate is reduced by 1 / 4 based on the original SAD value.

[0113] Fractional sample correction can follow integer sample offset search. To reduce computational complexity, fractional sample correction is performed by solving the parametric error surface formula instead of performing an additional search using SAD comparison. The fractional sample correction is conditionally invoked based on the output of the integer sample offset search. The search terminates when the center point reaches the minimum SAD value in the first or second iteration, and the fractional sample correction is further applied. Otherwise, the integer pixel distance correction MV can be used as the corrected MV output.

[0114] In subpixel offset estimation based on parametric error surfaces, the cost at the center location and the costs at the four locations adjacent to the center are used to fit a 2-D parabolic error surface as described in Equation 5 below: in,( C corresponds to the minimum cost position, and C corresponds to the minimum cost value. The above formula is solved by using the cost values ​​of five search points. Calculate according to Formula 6 and Formula 7 respectively: and The value is limited by default to between -8 and 8 because all values ​​are positive and the minimum value is 8. This corresponds to a half-pixel offset with 1 / 16 pixel MV precision in VVC. The calculated score ( It is added to the integer pixel distance correction MV to obtain the subpixel precision correction increment MV. The subpixel precision correction increment MV can be used as the correction MV instead of the integer pixel distance correction MV output.

[0115] In VVC, the resolution of the MV is 1 / 16 of a luminance sample. An 8-tap interpolation filter is used to interpolate the samples at the fractional positions. In DMVR, since the corrected MV search points are points immediately adjacent to the initial fractional pixel MV with an integer sample offset, samples at these fractional positions need to be interpolated for the DMVR corrected MV search. To reduce computational complexity, a bilinear interpolation filter is used to generate the fractional samples for the DMVR corrected MV search. Furthermore, by using a bilinear filter with a 2-sample search range, the DVMR does not access more reference samples compared to the standard motion compensation process. After outputting the corrected MV through the DMVR corrected MV search, a standard 8-tap interpolation filter is applied to generate the final prediction. To avoid accessing more reference samples than the standard motion compensation process, samples that are not needed by the interpolation process based on the original MV but are needed by the interpolation process based on the corrected MV are filled from those available samples.

[0116] When the width and / or height of a CB is greater than 16 luminance samples, it will be further subdivided into sub-blocks with a width and / or height equal to 16 luminance samples. The maximum cell size for the DMVR corrected MV search is limited to 16×16.

[0117] According to ECM, to further improve coding efficiency, multi-round decoding motion vector correction is applied. In the first round, BM is applied to the coding block. In the second round, BM is applied to each 16×16 sub-block within the coding block. In the third round, MV in each 8×8 sub-block is corrected by applying bidirectional optical flow (BDOF). The corrected MV is stored for spatial and temporal motion vector prediction.

[0118] In the first round, the corrected MV is derived by applying BM to the coded block. Similar to DMVR, in bidirectional prediction, the corrected MV is searched around the two initial MVs (MV0 and MV1) in the reference image lists L0 and L1. Based on the minimum bilateral matching cost between the two reference blocks in L0 and L1, the corrected MV is derived around the initial MV. and ).

[0119] BM-based corrections perform local search to derive integer sample precision. intDeltaMV The local search employs a 3×3 square search pattern to traverse the search area horizontally. And iterating through the search range in the vertical direction. ,in, sHor and sVer The value is determined by the block dimension. sHor and sVer The maximum value is 8 or another value. For example, in Figure 11 In this context, point 0 is the location pointed to by the initial MV and is set as the first search center. Therefore, points 1 to 8 around the initial location are searched first, and the cost of each location is calculated.

[0120] In the first search iteration, point 7 is found to have the minimum cost, so it is set as the second search center, and points 9, 10, and 11 are searched. In the next search iteration, point 10 is found to have a lower cost than points 7, 9, and 11, so it is set as the third search center, and points 12, 13, and 14 are searched. In the next search iteration, point 12 is found to have the minimum cost among points 6 to 14, so it is set as the fourth search center. In the next search iteration, the costs of points 10, 11, 13, and 15 to 19 surrounding point 12 are all found to be greater than the cost of point 12, so point 12 is the optimal point, and the modified MV search terminates, outputting the modified MV corresponding to the optimal point. Therefore, Figure 11 An example diagram is shown showing the search pattern (e.g., a 3×3 square search pattern) used in the first round of motion vector correction at the multi-round decoding end.

[0121] The cost of a two-sided matching can be calculated using the following formula 8: in, It is the SAD between the L0 predicted value (i.e., the reference patch from the reference image L0) and the L1 predicted value (i.e., the reference patch from the reference image L1) at a certain search point, and mvDistanceCost Based on intDeltaMV (That is, the distance between the search point and the initial point). When the block size cbW (CB width, in pixels) × cbH When the (CB height in pixels) is greater than 64, a mean-removed SAD (MRSAD) cost function is applied to remove the discrepancy cosine (DC) effect that distorts the reference blocks. When the center point of the 3×3 search pattern... bilCostWhen the minimum cost is achieved, the intDeltaMV The local search terminates. Otherwise, the current minimum cost search point is set as the new center point of the 3×3 search pattern, and the search for minimum cost continues until the end of the search range is reached.

[0122] Further apply existing fractional sample corrections to derive fractional MV corrections. fracDeltaMV And the final deltaMV Export as Then, derive the corrected MV after the first round using the following formulas 9 and 10: In the second round, the corrected MV is derived by applying BM to 16×16 matrix subblocks. For each subblock, in the reference image lists L0 and L1, the two MVs obtained in the first round ( MV0_pass1 and MV1_pass1 The modified MV is derived by searching around the reference subblocks in L0 and L1, based on the minimum bilateral matching cost between the two reference subblocks. sbIdx2 ) and MV1_pass2( sbIdx2 )).

[0123] For each sub-block, a full search is performed based on the BM correction to derive integer sample precision intDeltaMV( sbIdx2 The full search has a search range in the horizontal direction. And it has a search range in the vertical direction. , wherein sHor and sVer The value is determined by the block dimension, and the sHor and sVer The maximum value is 8 or other values.

[0124] According to the following formula 11, the bilateral matching cost can be calculated by applying a cost factor to the sum of absolute transformed difference (SATD) cost between the two reference sub-blocks: The search area Divided into up to 5 diamond-shaped search areas, such as Figure 12 As shown. Figure 12This diagram illustrates the bilateral matching cost used in the second round of motion vector correction at the multi-round decoding end (each matching cost corresponds to a diamond search region with different shades). Each search region is assigned a costFactor, which is determined by the distance intDeltaMV between each search point and the starting MV. sbIdx2 The search is performed by determining the search points, and each diamond-shaped region is processed sequentially starting from the center of the search area. Within each region, the search points are processed in a raster scan order from the top left corner to the bottom right corner. The smallest search point within the current search area... bilCost Less than and When the integer pixel full search reaches the same threshold, the search terminates; otherwise, the next search region continues with the integer pixel full search until all search points have been checked. Additionally, if the difference between the previous minimum cost and the current minimum cost in an iteration is less than a threshold equal to the area of ​​the block, the search terminates.

[0125] Furthermore, as mentioned above, the bilateral matching cost can also be calculated based on MRSAD instead of SAD, or based on the mean-removed sum of absolute transformed difference (MRSATD) instead of SATD.

[0126] Further apply existing VVC DMVR score sample corrections to derive the final deltaMV ( sbIdx2 Then, derive the revised MV for the second round according to the following formulas 12 and 13 respectively: In the third round, the corrected MV is derived by applying BDOF to the 8×8 matrix sub-blocks. For each 8×8 sub-block, starting with the corrected MV of the parent sub-block in the second round, BDOF correction is applied to derive the scaled MV without clipping. V x and V y The exported bioMv ( V x , V y The sample size is rounded to 1 / 16 of the sample precision and is limited between -32 and 32.

[0127] The corrected MV for the third round is derived from Formula 14 and Formula 15 respectively. sbIdx3 ) and MV1_pass3( sbIdx3 )): According to ECM, adaptive decoder-side motion vector correction is an extension of multi-round DMVR, which includes two new merging modes to correct the motion vector (MV) only in one temporal direction of the bidirectional prediction (i.e., reference image L0 or reference image L1) for merging candidates that meet the DMVR conditions. The multi-round DMVR process is applied to the selected merging candidate to correct the motion vector; however, in the first round (i.e., PU level) of DMVR, MVD0 or MVD1 is set to zero. Therefore, a new list of merging candidates is constructed for adaptive decoder-side motion vector correction. The new merging mode used for this new list of merging candidates is called BM merging, as provided by ECM.

[0128] The merge candidates used for the BM merging mode are derived from spatially adjacent coded blocks, TMVPs, non-adjacent blocks, history-based motion vector predictors (“HMVPs”), and paired candidates, similar to the regular merging mode. The difference is that only those merge candidates that satisfy the DMVR criteria are added to the merge candidate list. Both new merging modes use the same merge candidate list. If the BM candidate list contains inherited BCW weights, the DMVR process remains unchanged, but when the weights are unequal and the bidirectional predictions are weighted using BCW weights, distortion calculation needs to be performed using MRSAD or MRSATD. The merge index is encoded according to the regular merging mode.

[0129] As described above, TM is the decoder-side MV export method, which corrects the motion information of the current encoded block by finding the closest match between a template in the current image (i.e., the adjacent block above and / or to the left of the current encoded block) and a block in the reference image (i.e., the same size as the template). Figure 13 As shown, within the [-8, +8] pixel search range, the initial motion search around the current coding block yields a better MV. To achieve efficient integration with AMVR and DMVR, the TM's search step size can be determined based on the AMVR mode, and the TM can be cascaded with the DMVR process in merge mode.

[0130] In addition to the merged mode, TM can also be applied to the non-merged inter-frame mode, which is often referred to as the Advanced Motion Vector Prediction (AMVP) mode. In AMVP mode, MVP candidates are determined based on template matching error to select the candidate that achieves the minimum cost. This cost is calculated based on the difference between the current block template and the reference block template. TM is performed only on this specific MVP candidate for MV correction. This MVP candidate is corrected by performing TM with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode) within a search range of [–8, +8] pixels using an iterative diamond search. The AMVP candidate can be further corrected by using a cross-search with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode), followed by half-pixel and quarter-pixel cross-searches sequentially according to the AMVR modes specified in Table 1 below. This search process ensures that the MVP candidate maintains the same MV accuracy as indicated by the AMVR mode after the TM process. During the search process, if the difference between the previous minimum cost and the current minimum cost in an iteration is less than a threshold equal to the area of ​​the block, the search process terminates.

[0131] In merge mode, a similar search method is applied to the merge candidates indicated by the merge index. As shown in Table 1, TM can perform up to 1 / 8 pixel MVD accuracy, or it can skip those higher accuracies beyond half-pixel MVD accuracy, depending on whether an alternative interpolation filter (used when AMVR is in half-pixel mode) is used based on the merged motion information. Furthermore, when TM mode is enabled, template matching can be performed as a standalone process, or as an additional MV correction process between block-based and sub-block-based bilateral matching (BM) methods, depending on whether BM is allowed based on the BM enabling condition check results.

[0132] Figure 14A and 14B The diamond search pattern used in template matching is shown in Figure 14. Figure 14 illustrates an 8-point diamond search pattern. In each search iteration, the current position (center) is set as the search center, and the eight adjacent positions (around the center) surrounding the current position are examined. The position that produces the minimum cost is selected as the next search center. Figure 14BA 16-point diamond search pattern is shown, where 16 adjacent positions are checked in each search iteration. The search process is executed round after round until a preset maximum number of search iterations is reached or the search position is outside the search range. An early termination algorithm can be used. For example, the search terminates if the cost of the search center is minimized in a single search iteration.

[0133] Template matching cost can be calculated by measuring the difference between the template of the current block and the template of the reference block. The SAD or SATD between the template of the current block and the template of the reference block can be calculated as the TM cost, i.e., the cost of the candidate motion vector pointing to the reference block. In some other cases, the mean-reduced SAD or mean-reduced SATD can be calculated as the template matching cost.

[0134] For bidirectional prediction candidates, two prediction models (MVs) are first independently corrected, one for reference image list 0 and the other for reference image list 1. Then, an iterative process is performed to jointly correct these two MVs. This process is achieved through... Figure 15 The template-based correction process is described in 1500.

[0135] In step 1502, the initial motion vector of list 0 is corrected using TM. To export corrections And the export corresponds to The cost of TM is C0.

[0136] In step 1504, the initial motion vector of List 1 is corrected using TM. To export corrections And the export corresponds to The cost of TM is C1.

[0137] In step 1506, if C0 is greater than C1, then fix. And through additional consideration by The obtained template, exported from list 0, is further modified. Otherwise, fixed. And through additional consideration by The obtained template, exported as list 1, is further revised. .

[0138] In step 1508, if the export in step 1506 Then fixed And through additional consideration by The obtained template, exported as list 1, is further revised. Otherwise, fixed. And through additional consideration by The obtained template, export list 0, further modified MV ( In either case, obtain the corresponding and The cost of TM as CostBi .

[0139] Steps 1506 and 1508 can be performed in additional iterations. After correcting the bidirectional forecast, the bidirectional forecast cost will be... CostBi Compare with the one-way prediction cost C0 or C1. If the MV of list 0 was corrected in the previous iteration step, then... CostBi Compare with C1; if the MV of list 1 was corrected in the previous iteration step, then... CostBi Compare with C0. If CostBi If the cost is much greater than the cost of one-way prediction, then the current block is converted into a one-way prediction block.

[0140] Furthermore, after constructing the merge candidate list, the merge candidates are reordered according to the adaptive reordering of merge candidates with template matching (hereinafter referred to as "ARMC-TM"). This reordering is performed adaptively by TM. This reordering applies to the regular merge mode, TM merge mode, and sub-block merge mode (excluding the first SbTMVP candidate).

[0141] First, an initial merge candidate list is constructed based on a given inspection order, such as spatially adjacent coded blocks, TMVP, non-adjacent neighboring blocks, HMVP, paired candidates, and virtual merge candidates. Then, the candidates in the initial list are divided into several subgroups. The merge candidates in each subgroup are reordered to generate a reordered merge candidate list, and this reordering is based on a cost value based on template matching. The index signaling of the selected merge candidates in the reordered merge candidate list is transmitted to the decoder. For simplicity, the merge candidates in the last subgroup (and which is not the first subgroup) are not reordered. During the construction of the merge motion vector candidate list, all zero candidates from the ARMC reordering process are excluded. For the regular merge mode and TM merge mode, the subgroup size is set to 5. For the subblock merge mode, the subgroup size is set to 3.

[0142] During the reordering process, the template matching cost of merging candidates is measured by the SAD (Solution Aspect Ratio) between the samples of the template of the current block and its corresponding reference samples. The template includes a set of reconstructed samples adjacent to the current block. The reference samples of the template are located using the motion information of the merging candidate. When a merging candidate uses bidirectional prediction, the reference samples of the merging candidate's template are also located using... Figure 16 The bidirectional predictions shown are generated.

[0143] Figure 16 This illustrates motion prediction performed on the current image 1602 using bidirectional prediction. The current image 1602 includes a current block 1602A, which includes a template 1602B. According to the bidirectional prediction principle, the template's reference sample references two co-located reference images 1604 and 1606, one from reference image list 0 in a first temporal direction and the other from reference image list 1 in a second temporal direction. The motion information of the current block 1602A points to the co-located reference block 1604A of the co-located reference image 1604, and also to the co-located reference block 1606A of the co-located reference image 1606. The template 1602B of the current block 1602A points to the reference sample 1604B of the co-located reference image 1604, and also to the reference sample 1606B of the co-located reference image 1606.

[0144] When template matching is used to derive the corrected motion, the template size is set to 1. When the block is flat (width greater than twice its height) or elongated (height greater than twice its width), only the top or left template is used during TM motion correction. TM is extended to support 1 / 16 pixel MVD accuracy. In TM merging mode, the first four merge candidates are reordered using the corrected motion.

[0145] Given the size of the candidate sub-blocks for affine merging. The template above includes several sizes. The sub-template, the left template includes several templates with dimensions of Sub-templates. For example... Figure 17A and 17B As shown, the motion information of the first row and first column sub-blocks in the current block is used to derive the reference sample for each sub-template.

[0146] During the reordering process, if the cost difference between a candidate and its predecessor is less than λ, for example... If so, the candidates are considered redundant, where D1 and D2 are the costs obtained during the first ARMC sorting, and These are the Lagrange parameters used in the RD criteria at the encoding end.

[0147] The reordering process is as follows: Determine the minimum cost difference between a given candidate and its preceding candidate from all candidates in the list. If the minimum cost difference is greater than or equal to... If the list is considered sufficiently diverse, the reordering stops. If this minimum cost difference is less than... If a candidate is deemed redundant, it is moved to a later position in the list. This later position is the first position where the diversity requirement is first met between the candidate and its preceding candidate.

[0148] The process is repeated over a finite number of iterations, while the minimum cost difference is not less than .

[0149] This reordering step is applied to regular merge mode, TM merge mode, BM merge mode, and sub-block merge mode. Similar reordering is applied to merge MMVD and symbolic MVD prediction methods, which also use ARMC for reordering.

[0150] The The value is set to: in a low-latency configuration, it is equal to λ of the rate-distortion criterion used to select the best merge candidate at the encoder; in a random access configuration, it is equal to the value corresponding to another QP. Values. A set of corresponding values ​​is provided in SPS for each signaling indication QP offset. The value, or, if the QP offset does not appear in the SPS, the set of λ values ​​is located in the Slice Header.

[0151] Figure 18 The template-based reordering process 1800 applied in the TM merge mode is shown.

[0152] In step 1802, the TM merge candidates are reordered before TM correction.

[0153] In step 1804, preliminary TM-based motion correction is performed with the reduced template size.

[0154] In step 1806, repeat step 1802.

[0155] In step 1808, the final TM-based correction is performed using a full-size template.

[0156] In the initial TM-based correction, if a multi-round DMVR is used, only the first round of the multi-round DMVR (i.e., the PU level) is applied, and in the final TM-based correction, both the PU level and the sub-block level of the multi-round DMVR are applied.

[0157] The ARMC-ARMC design also applies to the AMVP mode, where the AMVP candidates are reordered based on the TM cost. For the template matching for advanced motion vector prediction (TM-AMVP) mode, an initial AMVP candidate list is first constructed, followed by a revised AMVP candidate list constructed using corrections derived from TM. Additionally, MVP candidates with a TM cost greater than a threshold equal to five times the cost of the first MVP candidate are skipped.

[0158] When surround motion compensation is enabled, surround offset should be considered to limit the MV candidate.

[0159] In addition, ARMC based on MV candidate types is provided. The merged candidates for a single candidate type (e.g., TMVP or non-adjacent MVP, NA-MVP) are reordered based on the ARMC™ value. The reordered candidates are then added to the merged candidate list. For the TMVP candidate type, more TMVP candidates with more temporal locations and different inter-frame prediction directions are added to perform reordering and selection. Furthermore, the NA-MVP candidate type is further expanded to include more spatially non-adjacent locations. A target reference image for the TMVP candidate can be selected from any reference image in the list based on a scaling factor. The selected reference image is the one with a scaling factor closest to 1.

[0160] Based on the techniques of the aforementioned VVC and ECM specifications, in the current sub-block merging candidate, TM is applied to the affine merging candidate to correct the underlying MV of the affine model. This is equivalent to correcting the MV of each sub-block by adding the same MV offset. However, for the sub-block merging candidate, TM is only used to correct motion offset, that is, to locate the motion field in the co-located image, and does not correct the sub-block MV itself actually used in motion compensation.

[0161] Therefore, exemplary embodiments of this disclosure provide template-matching-based motion correction for sub-blocks of a coded block to select the corrected motion vector of the sub-block of the coded block based on the sub-template of the sub-block that constructs the coded block.

[0162] To apply template-matching-based motion correction to sub-blocks of a coded block, sub-templates should be constructed first, followed by the final template. Each sub-block of the coded block has its own motion vector; therefore, the template also includes sub-templates, each obtained by the MV of adjacent sub-blocks. If the template size used in the TM is equal to... Ts And the size of each sub-block is ( WsIt's the width. Hs If the height is [height], then the upper template includes several dimensions. The sub-template, the left template includes several templates with dimensions of Sub-templates. Figure 19 The diagram shows the sub-templates and sub-blocks of coded block 1900, where the unshaded area is the coded block comprising sixteen sub-blocks 1902, and the shaded area comprises four sub-blocks of size [missing information]. The upper sub-template 1904 and four sizes are The left-side sub-template 1906 template. Ts This refers to the size of the template: for example, it can be 1, 2, 3, or 4.

[0163] To obtain the template of the reference block, it is necessary to determine the MV of each sub-template of the upper boundary sub-block, each sub-template of the left boundary sub-block, or both. In one example, the MV of each sub-template is the same as the MV of the upper boundary sub-block and the left boundary sub-block within the current coding block. Figure 20 This diagram illustrates a reference sub-template obtained by determining the MV of each sub-template of the upper boundary sub-block, each sub-template of the left boundary sub-block, or both, where the non-shaded sub-blocks are sub-blocks of the current encoded block, and the upper and left boundary sub-blocks are labeled SB0, SB1, SB2, SB3, SB4, SB8, and SB12. The shaded sub-blocks are sub-templates labeled as... sub0 , sub1 , sub2 , sub3 , sub4 , sub8 and sub12 Each top boundary sub-block and left boundary sub-block SBi Each has a motion vector MVi Therefore, each sub-template subi They also each have motion vectors MVi For the upper boundary sub-block, via MVi Obtain the reference sub-template above; for the left boundary sub-block, through... MVi Obtain the left reference sub-template; while for the upper left boundary sub-block, through... MVi Obtain the upper left reference sub-template. Therefore, the reference sub-template is also adjacent to the reference sub-blocks in the reference image, where each reference sub-block is labeled "ref". For bidirectional prediction blocks, derive the templates for reference image list 0 and reference image list 1 respectively. The template for reference image list 0 includes the sub-templates obtained through the MV of reference image list 0, and the template for reference image list 1 includes the sub-templates obtained through the MV of reference image list 1. Derive the final template by calculating the weighted average of the templates from reference image list 0 and reference image list 1.

[0164] Optionally, in order to obtain the template of the reference block, it is necessary to determine the upper boundary sub-block, the left boundary sub-block, or the MV of both (i.e., as described above). MVi ).

[0165] For sub-blocks of a coded block, each sub-block not only has a different prediction value (MV), but can also have a different prediction direction. For example, as Figure 21 As shown, SB0 is bidirectionally predicted, with motion vectors MV0_l0 and MV0_l1 for reference image list 0 and reference image list 1, respectively. SB1 is unidirectionally predicted, with only motion vectors for reference image list 0. MV1_ l0 SB2 is bidirectionally predicted, with motion vectors for reference image list 0 and reference image list 1, respectively. MV2_l0 and MV2_l1 SB3 is a unidirectional prediction, possessing only the motion vectors used for the reference image list 1. MV3_l1 .

[0166] like Figure 22A As shown, when the template of reference image list 0 is obtained, since SB3 does not have the MV of reference image list 0, there is no sub-template. sub3 Therefore, the template above reference image list 0 has only three sub-templates. sub0 , sub1 and sub2 .like Figure 22B As shown, when the template of reference image list 1 is obtained, since SB1 does not have the MV of reference image list 1, there is no sub-template. sub1 Therefore, the template above reference image list 1 only has three sub-templates. sub0 , sub2 and sub3 . Figure 22A and 22B The dashed outline in the diagram indicates that the sub-template does not exist. The template for reference image list 0 is derived by combining the sub-templates of reference image list 0; the template for reference image list 1 is derived by combining the sub-templates of reference image list 1.

[0167] After exporting templates for reference image list 0 and reference image list 1, a final template is exported for each bidirectional prediction sub-block by calculating a weighted average of the sub-templates from reference image list 0 and reference image list 1. A final template is also exported for each unidirectional prediction sub-block by using sub-templates derived from motion vectors from reference image list 0 or reference image list 1. Figure 22C As shown, the final sub-template sub0 and sub2 Combined with two sub-templates of two lists; final sub-template sub1Taken directly from reference image list 1 sub1 The final sub-template sub3 Taken directly from reference image list 0 sub3 After obtaining the final template of the reference block, the difference between the final template of the reference block and the template of the current block can be calculated as the TM cost corresponding to the current MV.

[0168] When the motion vector changes, the above process can be repeated to obtain a new final template for the reference block with a new TM cost (if the MV changes, the reference block will be different, and therefore the template for the reference block will also be different). This new TM cost can be calculated using any differential computation as described above. Both the VVC-compliant encoder and the VVC-compliant decoder perform an MV search process to try different MVs with different TM costs. The MV that produces the minimum TM cost is selected as the final corrected MV.

[0169] Optionally, such as Figure 23A , 23B As shown in 23C, when obtaining a template for a specific list of reference images, if a sub-template is missing due to the absence of a motion vector (MV) in the corresponding upper or left boundary sub-block of the specific list of reference images, the default MV is used: when the upper or left boundary sub-block has no MV, the default MV is used to obtain the sub-template for the specific list of reference images. Then, by calculating the weighted average of the first sub-template derived from the motion vector of reference image list 0 or the motion vector of reference image list 1 and the second sub-template derived from the default motion vector, the final template is derived for each unidirectional prediction sub-block. Figure 23A and 23B The lighter shaded blocks in the image represent child templates obtained through the default MV.

[0170] In another example, such as Figure 23D , 23E As shown in Figure 23F, when obtaining the template of a specific reference image list, if a sub-template is missing because the corresponding upper boundary sub-block or left boundary sub-block does not have the MV of the specific reference image list, then the MV of the adjacent boundary sub-block of the corresponding upper boundary sub-block or left boundary sub-block is used to obtain the sub-template of the specific reference image list. Figure 23D In this case, the boundary sub-block SB3 does not have the MV of reference image list 0, therefore sub-template 3 does not exist (represented as sub3 and marked in yellow). In this situation, the sub-template 3 of reference image list 0 is obtained by utilizing the MV of reference image list 0 of SB2, which is the adjacent sub-block of SB3. Figure 23EIn the example, the boundary sub-block SB1 does not have the MV of reference image list 1, therefore there is no sub-template 1 (represented as sub1 and marked in yellow). In this case, the sub-template 1 of reference image list 1 is obtained by using the MV of reference image list 1 of SB0 or ​​SB2, which are adjacent sub-blocks of SB1.

[0171] In another embodiment, adjacent sub-templates are merged with each other to obtain a more reliable template. Figure 24 This example illustrates an upper template for a list of reference images, from which its individual sub-templates are obtained. SB0, SB1, SB2, and SB3 are upper boundary sub-blocks, whose MV is used to obtain the reference block labeled "ref," and the sub-templates are shown in shaded mode. However, unlike the previous example, each sub-template of the upper boundary sub-block is wider than the width of the upper boundary sub-block, and each sub-template of the left boundary sub-block is longer than the length of the left boundary sub-block. Therefore, each sub-template of a boundary sub-block overlaps with at most two overlapping adjacent samples of its two adjacent sub-templates. Figure 24 The text further illustrates that sub-template 0 is represented as... sub0_ l , sub0 and sub0_r Subtemplate 1 is represented as sub1_l , sub1 and sub1_r Subtemplate 2 is represented as sub2_l , sub2 and sub2_r Subtemplate 3 is represented as sub3_l , sub3 and sub3_r Here, sub1_l constitute sub0 Overlapping adjacent samples; sub0_r and sub2_l constitute sub1 Overlapping adjacent samples; sub1_r and sub3_l constitute sub2 Overlapping adjacent samples; and sub2_r constitute sub3 Overlapping adjacent samples.

[0172] Figure 25 This illustrates the calculation of a weighted average of the sub-templates and overlapping sub-templates when fusing the sub-templates to obtain a template. The weights of the samples of the non-leftmost, non-rightmost, non-topmost, and non-bottommost sub-templates and their overlapping neighboring samples can be respectively... a and b ;in, This can be a power of 2 to avoid division, since the samples of the sub-templates are weighted using overlapping neighboring samples from both sides. For the leftmost, rightmost, topmost, and bottommost sub-templates, the weights of the sub-template samples and their overlapping neighboring samples can be respectively... a and 2 b or and b ;in, It can be a power of 2 to avoid division operations, since the center sample is only weighted with overlapping neighboring samples from one side rather than both sides. After constructing a template for the reference image list, two reference image list templates can be combined to obtain the final template, and the TM cost is calculated using the final template of the reference block and the template of the current block.

[0173] During the MV search process, different sub-blocks have different MVs, and therefore can have different MV offsets. However, the template is only related to the MVs of boundary sub-blocks. Therefore, in one embodiment, template matching is performed to correct the MVs of boundary sub-blocks, but not the MVs of non-boundary sub-blocks.

[0174] Figure 26 The diagram shows the motion vectors of the upper boundary sub-blocks and the left boundary sub-blocks SB0, SB1, SB2, SB3, SB4, SB8, and SB12. MV0 , MV1 , MV2 , MV3 , MV4 , MV8 and MV12 This is done to obtain their respective sub-templates. During the MV search process, the independent offset of each sub-block can be determined by applying independent motion offsets. dMV0 Add to MV0 Get new sub-templates; dMV1 Add to MV1 Get new sub-templates; dMV2 Add to MV2 Get new sub-templates; dMV3 Add to MV3 Get new sub-templates; dMV4 Add to MV4 Get new sub-templates; dMV8 Add to MV8 To obtain a new sub-template; and will dMV12 Add to MV12 To obtain new sub-templates. These sub-templates can collectively form a new template, and a new TM cost can be calculated based on the new template. The new TM cost corresponds to the MV offset group ( dMV0 , dMV1 , dMV2 , dMV3 , dMV4 , dMV8 , dMV12 The set of MV offsets that produces the minimum TM cost is the offsets obtained during the TM process; these are added to the initial boundary subblock MV to obtain the corrected boundary subblock MV.

[0175] Because each boundary sub-block can have a different MV offset, the search complexity is particularly high. To control the complexity, in some embodiments, all boundary sub-blocks share the same MV offset: each sub-template has the same motion offset during the MV search, causing the template to shift as a whole during the MV search. Figure 27 Showing the same MV offset dMV Add to all boundary sub-blocks MV to obtain the sub-template. Since there is only one MV offset to search, the template matching search method used for regular coded blocks can be used.

[0176] If all sub-block MVs have the same offset, this MV offset can also be applied to the MVs of non-boundary sub-blocks, even though the MVs of non-boundary sub-blocks do not contribute to the template. Figure 28 As shown, MV offset dMV It is added to all sub-block MVs. Therefore, in this embodiment, the boundary sub-block MV is used to obtain the template, and an MV offset is searched for for all boundary sub-block MVs during the correction, and after the correction, the optimal MV offset is applied to all sub-block MVs.

[0177] When searching for the MV offset during the TM process, any search pattern described above can be used. For example, the search pattern described above can be used... Figure 11 shown Cross search or Square search. Optionally, it can also be applied. Figure 14A and 14B The search pattern shown.

[0178] The search process can be divided into an integer search process and a fractional search process. In the integer search process, the search step size is an integer pixel distance, and in the fractional search process, the search step size is a fractional pixel distance. For example... Figure 29A As shown, only twenty positions 2900A are searched around the initial position at the center. In the integer search, the position with the minimum TM cost is taken as the optimal position and set as the initial position in the subsequent fractional search. To reduce complexity, as... Figure 29BAs shown, during the fractional search process, eight half-pixel positions (2900B) are searched around the optimal integer position at the center. In the fractional search, the position with the lowest TM cost is selected as the optimal position and output as the best-choice position. The corresponding MV offset is added to all sub-block MVs to obtain the corrected sub-block MV.

[0179] To further reduce search complexity, in some other embodiments, the number of neighboring positions to be searched in each search iteration is adaptively reduced based on the previous search iteration. For example, in a 3×3 cross-search scheme, four neighboring positions are searched in each search iteration. Assume the current center is ( a , b If the eight adjacent positions to be checked are pa0=(a+s, b), pa1=(as, b), pb0=(a, b+s), and pb1=(a, bs), then the template matching cost for four adjacent positions is expressed as follows: cost_pa0 , cost_pa1 , cost_pb0 , cost_pb1 .

[0180] VVC-compliant encoders and decoders configure one or more processors of a computing system to compare cost_pa0 and cost_pa1 :if cost_pa0 Less than cost_pa1 In the next iteration, only the parameters are considered. a Positive offset; if cost_pa0 Greater than cost_pa1 In the next iteration, only the parameters are considered. a The negative offset.

[0181] VVC-compliant encoders and decoders configure one or more processors of a computing system to compare cost_pb0 and cost_pb1 :if cost_pb0 Less than cost_pb1 In the next iteration, only the parameters are considered. b Positive offset; if cost_pb0 Greater than cost_pb1 In the next iteration, only the parameters are considered. b The negative offset.

[0182] VVC-compliant encoders and decoders configure one or more processors of a computing system to compare cost_pc0 and cost_pc1 :if cost_pc0 Less than cost_pc1 In the next iteration, only the parameters are considered. c Positive offset; if cost_pc0 Greater thancost_pc1 In the next iteration, only the parameters are considered. c The negative offset.

[0183] VVC-compliant encoders and decoders configure one or more processors of a computing system to compare cost_pd0 and cost_pd1 :if cost_pd0 Less than cost_pd1 In the next iteration, only the parameters are considered. d Positive offset; if cost_pd0 Greater than cost_pd1 In the next iteration, only the parameters are considered. d The negative offset.

[0184] Suppose that for the current search iteration, cost_pa0 Less than cost_pa1 ,and cost_pb0 Greater than cost_pb1 Then, in the next round, the two adjacent positions to be checked are... ,in, This will be the center position for the next search iteration.

[0185] In some other embodiments, the minimum template matching cost of the current search iteration is compared with the minimum template matching cost of the previous search iteration, or multiplied by a factor. f The results are then compared. If the reduction in the minimum template matching cost is small, i.e., the reduction does not exceed a threshold, the search terminates. For example, if the cost of the previous search iteration was... A This means that the cost of the current search center is also A, then the minimum template matching cost of the adjacent positions is at position A. posb of B ,in, B < A According to this search rule, the search will proceed to the next iteration, with its search center being... posb However, in this embodiment, if or The search then terminates, and the selected... posb This is the optimal position for this search iteration. K and f It is a preset threshold. For example, f It is a factor less than 1, such as 0.95, 0.9 or 0.8.

[0186] QP controls quantization in video coding. A higher QP requires a larger quantization step size, thus introducing more distortion. Therefore, higher QPs require more search iterations for correction, increasing coding time. To reduce overall coding time, in this embodiment, a smaller maximum search iteration threshold is set at higher QPs compared to lower QPs.

[0187] Other methods for reducing complexity can also be used at high QPs. For example, reducing the number of adjacent positions to be searched; adaptively reducing search iterations; or terminating the search early based on the previous search process. Therefore, in this embodiment, different search strategies can be employed at different QPs.

[0188] The number of search rounds may also depend on the sequence resolution. For example, for video sequences with high resolution, the maximum search iteration threshold or the number of adjacent positions to search in each round is set to a larger value, while for video sequences with low resolution, the maximum search iteration threshold or the number of adjacent positions to search in each round is set to a smaller value. In another example, correction for low-resolution video sequences is disabled. That is, whether to enable TM-based sub-block merging correction depends on the resolution of the video sequence.

[0189] Inter-frame coded frames, such as B-frames or P-frames, have one or more reference frames. The temporal distance between the current frame and the reference frames affects the accuracy of inter-frame prediction. In video coding, this temporal distance between two frames is typically represented by the Proof-of-Concept (POC) distance. Generally, the longer the POC distance, the lower the accuracy of inter-frame prediction and the lower the accuracy of motion information, thus requiring more corrections. Therefore, in this embodiment, the search process depends on the POC distance between the current frame and the reference frames.

[0190] For layered B-frames, frames with higher temporal layers have shorter point of intersection (POC) distances to the reference frame, while frames with lower temporal layers have longer POC distances. Therefore, the search process can also depend on the temporal layer of the current frame. For example, affine parameter correction for higher temporal layers can be disabled because the shorter POC distance to the reference frame may not require correction. In another example, for higher temporal layer frames, a smaller search iteration threshold can be set, or the number of adjacent search positions can be reduced.

[0191] Furthermore, other methods for reducing the complexity of parameter correction can also be used for the higher temporal layer frames. Therefore, in this embodiment, the parameter correction process depends on the temporal distance or POC distance between the current frame and the reference frame.

[0192] After the sub-block MV correction, the TM cost costA It can be used with the TM cost (denoted as) of the initial subblock MV.cost0 ) for comparison. Only when costA < h × cost0 (in, h If the factor is less than 1, the modified sub-block MV is used for motion compensation; otherwise, the initial sub-block MV is used for motion compensation, which means that the correction is restored.

[0193] In some embodiments, the TM cost incorporates the MV offset as a factor, thereby increasing the cost of searching for positions far from the initial position. The MV offset refers to the difference between the corrected MV and the initial MV. Therefore, a larger MV correction will proportionally apply a larger template matching cost, thus preventing the corrected MV from deviating too far from the initial MV derived from the adjacent blocks.

[0194] Assume that the MV offset obtained during the sub-block MV search process is dMV =( mv x , mv y The cost of MV, denoted as cost(MV), can then be derived using the following formula 16: The TM cost, denoted as cost(TM), can be a weighted sum of the MV cost and the sample cost, as shown in Equation 17 below: The sample cost is derived from the sample difference between the template of the current block and the template of the reference block. It can be the SAD or SATD of the two templates.

[0195] As will be appreciated by those skilled in the art, all the foregoing aspects of this disclosure may be implemented simultaneously in any combination thereof, and all aspects of this disclosure may be combined as yet another embodiment of this disclosure.

[0196] Figure 30 An exemplary system 3000 is shown for implementing the process and method described above for implementing template matching-based motion correction.

[0197] The techniques and mechanisms described herein can be implemented by multiple instances of System 3000 as well as any other computing devices, systems, and / or environments. Figure 30The system 3000 shown is merely an example and does not impose any limitation on the scope or functionality of any computing device used to perform the above processes and / or programs. Other known computing devices, systems, environments, and / or configurations suitable for use with the embodiments include, but are not limited to: personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, game consoles, programmable consumer electronics devices, network PCs, microcomputers, mainframe computers, distributed computing environments including any of the above systems or devices, implementations using field-programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs), etc.

[0198] System 3000 may include one or more processors 3002 and system memory 3004 communicatively coupled to the one or more processors 3002. The one or more processors 3002 may execute one or more modules and / or processes to enable the one or more processors 3002 to perform various functions. In some embodiments, the one or more processors 3002 may include a central processing unit (“CPU”), a graphics processing unit (“GPU”), both a CPU and a GPU, or other processing units or components known in the art. Additionally, each of the one or more processors 3002 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0199] Depending on the precise configuration and type of the system 3000, the system memory 3004 may be volatile, such as RAM, or non-volatile, such as ROM, flash memory, micro hard disk drive, memory card, or a combination thereof. The system memory 3004 may include one or more computer-executable modules 3006 executable by the one or more processors 3002.

[0200] The module 3006 may include, but is not limited to, one or more of the encoder 3008 and the decoder 3010.

[0201] The encoder 3008 may be a VVC-compliant encoder for implementing any, some, or all of the exemplary embodiments of the present disclosure described above, and may be executed by the one or more processors 3002 to configure the one or more processors 3002 to perform the above operations.

[0202] The decoder 3010 may be a VVC standard encoder for implementing any, some, or all of the exemplary embodiments of the present disclosure described above, and may be executed by the one or more processors 3002 to configure the one or more processors 3002 to perform the above operations.

[0203] System 3000 may additionally include an input / output ("input / output, I / O") interface 3040 for receiving image source data and bitstream data, and for outputting the reconstructed image to a reference image buffer or a DPB and / or display buffer. System 3000 may also include a communication module 3050 enabling system 3000 to communicate with other devices (not shown) via a network (not shown). The network may include the Internet, wired media such as a wired network or direct wired connection, and wireless media such as acoustic, radio frequency ("RF"), infrared, and other wireless media.

[0204] Some or all of the operations described above can be performed by executing computer-readable instructions stored on the computer-readable storage medium 3030, as defined below. The term "computer-readable instructions" as used in the specification and claims includes routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on various system configurations, including single-processor or multi-processor systems, microcomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based systems, programmable consumer electronic devices, and combinations thereof.

[0205] Computer-readable storage media may include volatile memory (such as random-access memory, RAM) and / or non-volatile memory (such as read-only memory, ROM), flash memory, etc. Computer-readable storage media may also include additional removable and / or non-removable memory, including but not limited to flash memory, magnetic memory, optical memory, and / or magnetic tape memory, which can provide non-volatile storage of computer-readable instructions, data structures, program modules, etc.

[0206] Non-transient or non-temporary computer-readable storage media are examples of computer-readable media. Computer-readable media include at least two types: computer-readable storage media and communication media. Computer-readable storage media include volatile and non-volatile, removable and non-removable media, used in any process or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disk read-only memory (CD-ROM), digital versatile disk (DVD), or other optical storage, cassette tape, magnetic tape, disk storage, or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. In contrast, communication media may embody computer-readable instructions, data structures, program modules, or other data in a carrier or modulated data signal, such as a carrier or other transmission mechanism. The computer-readable storage medium used in this document should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (such as optical pulses through fiber optic cables), or electrical signals propagating through wires.

[0207] Computer-readable instructions stored on one or more non-transitory or non-temporary computer-readable storage media, when executed by one or more processors, can perform the above-mentioned references. Figures 1A - 29B The operations described. Typically, computer-readable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific abstract data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the operations can be combined in any order and / or in parallel to perform the process.

[0208] Although the subject matter has been described using terms specific to structural features and / or methodological operations, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described. Rather, the specific features and actions are disclosed as exemplary forms for implementing the claims.

Claims

1. A computing system, comprising: One or more processors, and A computer-readable storage medium communicatively coupled to the one or more processors, the computer-readable storage medium storing computer-readable instructions executable by the one or more processors, the computer-readable instructions performing related operations when executed by the one or more processors, the operations including: A template for a reference block of the coded block is constructed by exporting the corresponding sub-templates of the upper boundary sub-block or the left boundary sub-block of the coded block.

2. The computing system according to claim 1, wherein, Exporting the corresponding sub-templates for the top boundary sub-block or the left boundary sub-block includes: Determine the motion vector of the sub-template of the upper boundary sub-block or the sub-template of the left boundary sub-block, and Based on the motion vector, the sub-template is determined.

3. The computing system according to claim 1, wherein, Exporting the corresponding sub-templates for the top boundary sub-block or the left boundary sub-block includes: Determine the motion vector of the upper boundary sub-block or the left boundary sub-block, and Based on the motion vector, the sub-template is determined.

4. The computing system according to claim 3, wherein, Exporting the motion vectors of the top boundary sub-block or the left boundary sub-block includes: For a unidirectional prediction sub-block, derive one of the motion vectors from either reference image list 0 or reference image list 1; and For a bidirectional prediction sub-block, derive two corresponding motion vectors for reference image list 0 and reference image list 1.

5. The computing system according to claim 1, wherein, The operation further includes: A template for exporting reference image list 0, the template comprising at least one sub-template exported from motion vectors of reference image list 0; and A template for exporting reference image list 1 is provided, the template comprising at least one sub-template exported from motion vectors of reference image list 1.

6. The computing system according to claim 5, wherein, The operation further includes: The final template is derived by calculating the weighted average of the templates in reference image list 0 and reference image list 1.

7. The computing system according to claim 6, wherein, Calculating the weighted average of the templates in reference image list 0 and reference image list 1 includes: Calculate the weighted average of the at least one sub-template derived from the motion vector of reference image list 0 and the at least one sub-template derived from the motion vector of reference image list 1.

8. The computing system according to claim 5, wherein, The operation further includes: By employing sub-templates derived from motion vectors in reference image list 0 or reference image list 1, a final sub-template for unidirectional prediction of sub-blocks is derived; or The final sub-template for unidirectional prediction sub-blocks is derived by calculating the weighted average of the first sub-template derived from the motion vector of reference image list 0 or the motion vector of reference list 1 and the second sub-template derived from the default motion vector.

9. The computing system according to claim 1, wherein, The sub-template of the upper boundary sub-block is wider than the upper boundary sub-block, and the sub-template of the left boundary sub-block is longer than the left boundary sub-block.

10. The computing system according to claim 9, wherein, The sub-template of the upper boundary sub-block overlaps with at most two sub-templates of the adjacent upper boundary sub-block, and the sub-template of the left boundary sub-block overlaps with at most two sub-templates of the adjacent left boundary sub-block.

11. The computing system according to claim 10, wherein, The operation further includes: The upper template is exported by merging the sub-template of the upper boundary sub-block with the overlapping sub-template, or The left template is exported by merging the sub-template of the left boundary sub-block with the overlapping sub-template.

12. The computing system according to claim 11, wherein, The process of merging the sub-templates of the non-leftmost and non-rightmost top boundary sub-blocks with overlapping sub-templates, and merging the sub-templates of the non-topmost and non-bottommost top boundary sub-blocks with overlapping sub-templates, includes calculating a weighted average of the sub-templates and the overlapping sub-templates, wherein the weight of the overlapping sub-template is... b ;and The process of merging the sub-template of the leftmost or rightmost top boundary sub-block with the overlapping sub-template, and merging the sub-template of the topmost or bottommost left boundary sub-block with the overlapping sub-template, includes calculating the weighted average of the sub-templates and the overlapping sub-templates, wherein the weight of the overlapping sub-template is 2. b .

13. The computing system according to claim 5, wherein, The operation further includes: Based on the template matching TM cost, the motion vector of the upper boundary sub-block or the left boundary sub-block is corrected, wherein the TM cost includes the difference between the template of the coded block and the template of the reference block.

14. The computing system according to claim 13, wherein, The correction of motion vectors further includes adding independent offsets to the corresponding motion vectors of each sub-block.

15. The computing system according to claim 13, wherein, The correction of motion vectors further includes adding the same offset to the corresponding motion vector of each sub-block.

16. The computing system according to claim 13, wherein, Correcting the motion vector includes performing a motion vector search, wherein the motion vector search includes searching for twenty integer pixel distance positions around the initial position to determine the optimal integer pixel distance position.

17. The computing system according to claim 16, wherein, Performing the motion vector search further includes searching for eight half-pixel distance positions around the optimal integer pixel distance position to determine the optimal position.

18. The computing system according to claim 13, wherein, Correcting the motion vector includes performing a motion vector search, wherein the motion vector search is performed iteratively, and one iteration of the motion vector search includes searching eight integer pixel distance positions around the optimal integer pixel distance position to determine the optimal position.

19. The computing system according to claim 13, wherein, Correcting the motion vector includes performing a motion vector search, wherein the motion vector search is performed iteratively, and the motion vector search terminates if a subsequent iteration fails to reduce the minimum template matching cost by a threshold.

20. The computing system according to claim 13, wherein, The higher the quantization parameter QP, the fewer iterations are required for the motion vector search; conversely, the lower the QP, the more iterations are required.

Images