Decoder-side motion vector refinement for affine motion compensation

Abstract

The invention provides a VVC standard encoder and a VVC standard decoder, which realize the application of DMVR on an affine merging mode coding block so as to refine the motion vector precision and improve the coding efficiency. A refined motion vector (MV) search is performed on a control point motion vector (CPMV) of an inter-coded coded block (CB), and a refined MV of the CB is output. The refinement MV search includes deriving a sub-block MV of the CB based on the CPMV of the CB, performing a sub-block MV refinement on the MV of the sub-block, and outputting a refinement MV of the CB based on the refinement MV of the sub-block. The refined MV search further includes deriving affine model parameters based on the plurality of CPMVs of the CB, performing an affine parameter offset search on the affine model parameters, and outputting a refined MV of the CB based on an optimal parameter offset.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present disclosure claims the benefit of priority to U.S. Provisional Application No. 63 / 358,257, filed on July 5, 2022, entitled “DECODER SIDE MOTION VECTOR REFINEMENT FOR AFFIN MOTION COMPENSATION,” U.S. Provisional Application No. 63 / 406,122, filed on September 13, 2022, entitled “DECODER SIDE MOTION VECTOR REFINEMENT FOR AFFIN MOTION COMPENSATION,” U.S. Provisional Application No. 63 / 433,748, filed on December 19, 2022, entitled “DECODER SIDE MOTION VECTOR REFINEMENT FOR AFFIN MOTION COMPENSATION,” and U.S. Provisional Application No. 18 / 346,766, filed on July 3, 2023, entitled “DECODER SIDE MOTION VECTOR REFINEMENT FOR AFFIN MOTION COMPENSATION.” The entire contents of all of the foregoing provisional applications are incorporated herein by reference. Technical Field

[0002] The present disclosure relates generally to video processing, and more particularly, to methods and systems for decoder-side motion vector refinement for affine motion compensation. Background Art

[0003] In 2020, the Joint Video Experts Group ("JVET") of the ITU-T Video Coding Experts Group ("ITU-T VCEG") and the ISO / IEC Moving Picture Experts Group ("ISOIEC MPEG") released the final draft of the next-generation video codec specification Versatile Video Coding ("VVC"). The specification further improves video coding performance compared to previous standards such as H264 / AVC (Advanced Video Coding) and H265 / HEVC (High Efficiency Video Coding). JVET continues to propose additional technologies beyond the scope of the VVC standard itself, which are collected under the name Enhanced Compression Model ("ECM"), proposed in January 2021, and used as a new software foundation for developing tools outside the VVC standard.

[0004] Inter-image prediction is essential for video coding and is communicated through multiple motion vectors (MVs). Techniques that signal multiple MVs reduce the bitrate transmitted to the decoder, but produce inaccurate multiple MVs, resulting in imprecise predictions. To improve the results of these techniques, decoder-side refinement and / or compensation techniques can be used. Decoder-side motion vector refinement (DMVR), including multi-pass DMVR and adaptive DMVR, is a coding tool used to refine MVs at the decoder side without using additional signaling, because the MV inherited from a neighboring block may not exactly match the current block. And affine motion compensation can be used to capture the affine motion between two different frames.

[0005] However, in the current design, these two coding tools are mutually exclusive. That is, DMVR is only applied to non-affine coded blocks to refine multiple MVs of translational motion. And if the block is encoded with affine mode, DMVR is not used to refine multiple MVs. Summary of the invention

[0006] Embodiments of the present disclosure relate to decoder-side motion vector refinement for affine motion compensation.

[0007] In a first aspect, an embodiment of the present disclosure provides a computer system, comprising: one or more processors, and A computer-readable storage medium is communicatively coupled to the one or more processors, the computer-readable storage medium storing computer-readable instructions executable by the one or more processors, and when executed by the one or more processors, the related operations performed include: performing a refined motion vector (MV) search on a control point motion vector (CPMV) of an inter-coded coding block (CB); and Output the refined MV of CB.

[0008] In a second aspect, an embodiment of the present disclosure provides a method, comprising: performing a refined MV search on a CPMV of an inter-coded CB; and outputting the refined MV of the CB.

[0009] In a third aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium, which stores an instruction set that can be executed by one or more processors of a device to enable the device to initiate the method described in the second aspect above.

[0010] In a fourth aspect, an embodiment of the present disclosure provides a computer program product, which includes: computer program instructions, wherein the computer program instructions enable a computer to execute the method of the second aspect.

[0011] In a fifth aspect, an embodiment of the present disclosure provides a computer program, and the computer program enables a computer to execute the method described in the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The detailed description is given with reference to the accompanying drawings, in which the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. The use of the same reference numbers in different drawings indicates similar or identical items or features.

[0013] Figure 1A and Figure 1BExample block diagrams of a video encoding process and a video decoding process according to example embodiments of the present disclosure are respectively shown.

[0014] Figure 2 Motion prediction performed on a current picture according to bidirectional prediction is shown, wherein a plurality of offset blocks of a reference picture are used to calculate refined motion vectors which in turn are used to generate a bidirectional prediction signal.

[0015] Figure 3A , 3B and 3C show example diagrams of search patterns used in the first pass of multi-pass decoder-side motion vector refinement.

[0016] Figure 4 Diagram illustrating bilateral matching costs used in the second pass of multi-pass decoder-side motion vector refinement.

[0017] Figure 5A and 5B Diagrams illustrating affine motion fields of blocks with four parameters (two control points) and six parameters (three control points), respectively. Although there are various motions (eg, scaling) in the real world, according to the VVC standard, block-based affine transformation motion compensated prediction is applied.

[0018] Figure 6 Diagram illustrating multiple affine motion vectors for multiple luma sub-blocks calculated for each sub-block center sample.

[0019] Figure 7 A graph showing the inheritance of control point motion vectors from which multiple candidates are used to form an affine merge candidate list.

[0020] Fig. 8A and 8B Diagrams showing candidates inherited from non-adjacent neighboring blocks of an affine merge candidate list and first type of construction candidates of an affine merge candidate list, respectively.

[0021] Fig. 9 A diagram showing the positions of multiple affine candidates constructed from adjacent neighboring blocks for an affine merge candidate list.

[0022] Fig.10 A diagram showing the locations of multiple affine candidates constructed from non-adjacent neighboring blocks for an affine merge candidate list.

[0023] Fig.11 A diagram showing the acquisition of neighboring 4×4 sub-block information whose motion information is used for affine model regression of the current coding block.

[0024] Fig.12A diagram showing the difference between a sub-block motion vector at a position and a motion vector calculated at that position, which difference is used as part of optical flow prediction refinement for sub-block based affine motion compensated prediction.

[0025] Fig.13 A diagram showing the search window for each sub-block.

[0026] Fig.14 A graph showing multiple samples at integer and fractional search points.

[0027] Fig.15 An example of searching for a plurality of affine model parameters in a case where a plurality of MVs of the upper left, upper right, and lower right of a coding block are fixed is shown.

[0028] Fig.16 An example system is illustrated for implementing the processes and methods described herein for implementing decoder-side motion vector refinement for affine motion compensation. DETAILED DESCRIPTION

[0029] According to the VVC video coding standard ("VVC standard") and the motion prediction described therein, computer-readable instructions stored on a computer-readable storage medium can be executed by one or more processors of a computing system to configure the one or more processors to perform the encoder operations described in the VVC standard and the decoder operations described in the VVC standard. Some of the encoder operations and decoder operations performed according to the VVC standard will be described in further detail later, but these subsequent descriptions should not be construed as an exhaustive description of the encoder operations and decoder operations performed according to the VVC standard. Subsequently, the "VVC standard encoder" and "VVC standard decoder" will describe individual computer-readable instructions stored on a computer-readable storage medium that configure one or more processors to perform these respective operations (e.g., "multiple reference implementations" that may be referred to as encoders or decoders).

[0030] In addition, according to an exemplary embodiment of the present disclosure, the VVC standard encoder and the VVC standard decoder further include computer-readable instructions stored on a computer-readable storage medium, which can be executed by one or more processors of a computing system to configure one or more processors to perform multiple operations not specified by the VVC standard. The VVC standard encoder should not be understood as being limited to multiple operations of the reference implementation of the encoder, but rather includes multiple other computer-readable instructions that configure one or more processors of the computing system to perform further multiple operations as described herein. The VVC standard decoder should not be understood as being limited to multiple operations of the reference implementation of the decoder, but rather includes multiple other computer-readable instructions that configure one or more processors of the computing system to perform further multiple operations as described herein.

[0031] 1A and 1B respectively show example block diagrams of an encoding process 100 and a decoding process 150 according to an example embodiment of the present disclosure.

[0032] In the encoding process 100, the VVC standard encoder configures one or more processors of the computing system to receive one or more input images as input from an image source 102. The input image slices include a certain number of pixels sampled by an image capture device (such as a photosensor array), and include an uncompressed stream of multiple color channels (such as RGB color channels) storing color data at the original resolution of the image, wherein each channel uses a certain number of bits to store the color data of each pixel of the image. The VVC standard encoder configures one or more processors of the computing system to store the uncompressed color data in a compressed format, wherein the color data is stored at a lower resolution than the original resolution of the image, encoded as a luma ("Y") channel and two chroma ("U" and "V") channels with a lower resolution than the luma channel.

[0033] The VVC standard encoder encodes an image (the image being encoded is referred to as the "current image" to distinguish it from any other image received from the image source 102) by configuring one or more processors of the computing system to divide the original image into a plurality of units and a plurality of sub-units according to the partition structure. The VVC standard encoder configures one or more processors of the computing system to subdivide the image into a plurality of macroblocks ("MBs"), each of which has a size of 16×16 pixels, which can be further subdivided into a plurality of partitions. The VVC standard encoder configures one or more processors of the computing system to subdivide the image into a plurality of coding tree units ("CTUs"), whose luminance and chrominance components can be further subdivided into a plurality of coding tree blocks ("CTBs"), which are further subdivided into a plurality of coding blocks ("CBs"). Alternatively, the VVC standard encoder configures one or more processors of the computing system to subdivide the image slice into a plurality of units of N×N pixels, which can then be further subdivided into a plurality of sub-units. For the purposes of this disclosure, each of these largest subdivision units of an image can generally be referred to as a "block".

[0034] The CB is encoded using one block of luma samples and two corresponding blocks of chroma samples, where the image is not monochrome and is encoded using one coding tree.

[0035] The VVC standard encoder configures one or more processors of the computing system to subdivide the block into multiple partitions of a size that is a multiple of 4×4 pixels. For example, a partition of a block can have a size of 8×4 pixels, 4×8 pixels, 8×8 pixels, 16×8 pixels, or 8×16 pixels.

[0036] By encoding color information of multiple blocks of an image and multiple subdivisions thereof, instead of encoding the color information of pixels of a full-resolution original image, the VVC standard encoder configures one or more processors of a computing system to encode the color information of an image at a lower resolution than the input image, storing the color information in fewer bits than the input image.

[0037] In addition, the VVC standard encoder encodes the image by configuring one or more processors of the computing system to perform motion prediction on multiple blocks of the current image. Motion prediction coding refers to storing image data of a block of the current image (wherein the block of the original image is referred to as an "input block" before encoding) using motion information and multiple prediction units ("PUs") instead of pixel data according to intra-frame prediction 104 or inter-frame prediction 106.

[0038] Motion information refers to data describing the motion of a block structure of an image or a unit or subunit thereof, such as multiple motion vectors and references to multiple blocks of a current image or a reference image. Multiple PUs may refer to a unit or multiple subunits (e.g., MB or CTU) corresponding to a block structure in multiple block structures of an image, where multiple blocks are divided based on image data and encoded according to the VVC standard. The motion information corresponding to the PU may describe the motion prediction encoded by the VVC standard encoder described herein.

[0039] The VVC standard encoder configures one or more processors in a computing system to encode motion prediction information on each block of an image in a coding order between multiple blocks, such as a raster scan order, where the first decoded block is the top and leftmost block of the image. The block being encoded is called the "current block" to distinguish it from any other block of the same image.

[0040] According to intra prediction 104, one or more processors of the computing system are configured to encode the block by referencing motion information of one or more other blocks of the same image and multiple PUs. According to intra prediction coding, one or more processors of the computing system perform intra prediction 104 (also called spatial prediction) calculations by encoding motion information of the current block based on multiple spatial neighboring samples from multiple spatial neighboring blocks of the current block.

[0041] According to inter-frame prediction 106, one or more processors of the computing system are configured to encode the block by referencing motion information of one or more other images and multiple PUs. The one or more processors of the computing system are configured to store one or more previously encoded and decoded images in a reference image buffer for inter-frame prediction encoding; these stored images are referred to as multiple reference images.

[0042] One or more processors are configured to perform inter-frame prediction 106 (also called temporal prediction or motion compensated prediction) calculations by encoding motion information of the current block based on multiple samples from one or more reference images. Inter-frame predictions can also be further calculated according to unidirectional prediction or bidirectional prediction: in unidirectional prediction, only one motion vector pointing to one reference image is used to generate a prediction signal for the current block. In bidirectional prediction, two motion vectors are used to generate a prediction signal for the current block, each motion vector pointing to a respective reference image.

[0043] The VVC standard encoder configures one or more processors of the computing system to encode the CB so that it includes a reference index to identify one or more prediction signals of the current block for reference by the VVC standard decoder. One or more processors of the computing system can encode the CB to include an inter-frame prediction indicator. The inter-frame prediction indicator indicates list 0 prediction, list 1 prediction, or bidirectional prediction, where list 0 prediction refers to a first reference image list called list 0, list 1 prediction refers to a second reference image list called list 1, and bidirectional prediction simultaneously refers to two reference image lists called list 0 and list 1, respectively.

[0044] In the case where the inter-frame prediction indicator indicates list 0 prediction or list 1 prediction, one or more processors of the computing system are configured to encode a CB including a reference index that refers to a reference picture of a reference picture buffer referenced by list 0 or list 1, respectively. In the case where the inter-frame prediction indicator indicates bidirectional prediction, one or more processors of the computing system are configured to encode a CB including a first reference index that refers to a first reference picture of a reference picture buffer referenced by list 0, and a second reference index that refers to a second reference picture of a reference picture buffer referenced by list 1.

[0045] The VVC standard encoder configures one or more processors as a computing system to encode each current block in the image separately and output a prediction block for each block. According to the VVC standard, a CTU can be as large as 128×128 luminance samples (plus corresponding multiple chrominance samples, depending on the chrominance format). The CTU can be further divided into multiple CBs according to a quadtree, a binary tree, or a ternary tree. One or more processors of the computing system are configured to ultimately record a set of coding parameters at the syntax structure of the leaf nodes of the partition structure, such as a coding mode (intra-frame mode or inter-frame mode), motion information of multiple inter-frame coding blocks (reference indexes, motion vectors, etc.), and multiple quantized residual coefficients.

[0046] After outputting the prediction block, the VVC standard encoder configures one or more processors of the computing system to send a set of encoding parameters, such as encoding mode (i.e., intra or inter prediction), intra prediction mode or inter prediction mode, and motion information (as described later), to the entropy encoder 124.

[0047] The VVC standard provides semantics for recording multiple coding parameter sets of CB. For example, in the above coding parameter set, the pred_mode_flag of CB is set to 0 for inter-frame coding blocks and to 1 for intra-frame coding blocks; the general_merge_flag of CB is set to indicate whether the inter-frame prediction of CB uses the merge mode; the inter_affine_flag and cu_affine_type_flag of CB are set to indicate whether affine motion compensation is used in the inter-frame prediction of CB; mvp_l0_flag and mvp_l1_flag are set to indicate the motion vector index in list 0 or list 1 respectively; ref_idx_l0 and ref_idx_l1 are set to indicate the reference image index in list 0 or list 1 respectively. It should be understood that the VVC standard includes semantics for recording various other information, flags and options beyond the scope of this disclosure.

[0048] The VVC standard encoder further implements one or more mode decisions and encoder control settings 108, including rate control settings. One or more processors of the computing system are configured to perform mode decisions by selecting an optimized prediction mode for the current block based on a rate-distortion optimization method after intra-frame or inter-frame prediction.

[0049] The rate control setting configures one or more processors of the computing system to assign different multiple quantization parameters ("QPs") to multiple different images. The size of the QP determines the scale at which the one or more processors quantize the image information during the encoding process (as described later), thereby determining the extent to which the encoding process 100 discards image information from multiple MBs of the sequence during the encoding process (due to information loss between step spans).

[0050] The VVC standard encoder also implements a subtractor 110. One or more processors of the computing system are configured to perform a subtraction operation by calculating the difference between the input block and the prediction block. Based on the optimized prediction mode, the prediction block is subtracted from the input block. The difference between the input block and the prediction block is called a prediction residual, referred to as "residual".

[0051] Based on the prediction residual, the VVC standard encoder further implements transformation 112. One or more processors of the computing system are configured to perform a transformation operation on the residual through matrix arithmetic operations to derive a coefficient array (which may be referred to as "residual coefficients", "transform coefficients", etc.), thereby encoding the current block into a transform block ("TB"). The transform coefficient may refer to a coefficient of one of several spatial transforms, such as diagonal flipping, vertical flipping, or rotation, which may be applied to a sub-block.

[0052] It should be appreciated that coefficients may be stored as two components, absolute value and sign, as will be described in further detail below.

[0053] The sub-blocks of the CB, such as PUs and TBs, can be arranged in any combination of the above sub-block sizes. The VVC standard encoder configures one or more processors of the computing system to subdivide the CB into a residual quad tree ("RQT"), a hierarchical structure of TBs. The RQT provides an order for motion prediction and residual coding for multiple sub-blocks at each level, and recursively down to each level of the RQT.

[0054] The VVC standard encoder further implements quantization 114. One or more processors of the computing system are configured to perform quantization operations on the residual coefficients through matrix arithmetic operations based on the quantization matrix and QP assigned as above. Residual coefficients falling within a certain interval are retained, and residual coefficients falling outside the interval are discarded.

[0055] The VVC standard encoder also implements inverse quantization 116 and inverse transform 118. One or more processors of the computing system are configured to perform inverse quantization and inverse transform operations on the quantized residual coefficients by matrix arithmetic operations, which are inverse operations of the quantization and transform operations described above. The inverse quantization and inverse transform operations produce a reconstructed residual.

[0056] The VVC standard encoder further implements an adder 120. The one or more processors of the computing system are configured to perform an addition operation by adding the prediction block and the reconstructed residual, and output a reconstructed block.

[0057] The VVC standard encoder further implements a loop filter 122. One or more processors of the computing system are configured to apply a loop filter, such as a deblocking filter, a sample adaptive offset ("SAO") filter, and an adaptive loop filter ("ALF"), to the reconstructed block, and output a filtered reconstructed block.

[0058] The VVC standard encoder further configures one or more processors of the computing system to output the filtered reconstructed blocks to a decoded picture buffer ("DPB") 200. DPB 200 stores the reconstructed pictures, which are used by one or more processors of the computing system as reference pictures in multiple coded pictures other than the current picture, as described above with reference to inter-frame prediction.

[0059] The VVC standard encoder also implements an entropy encoder 124. One or more processors of the computing system are configured to perform entropy coding, wherein the signs constituting the quantized residual coefficients are encoded by mapping to binary strings (hereinafter referred to as "bins") according to a context-aware binary arithmetic codec ("CABAC"), which can be transmitted at a compressed bit rate in an output bitstream. The signs of the encoded quantized residual coefficients include the absolute values ​​of the residual coefficients (these absolute values ​​are subsequently referred to as "residual coefficient levels").

[0060] The entropy encoder configures one or more processors of the computing system to encode residual coefficient levels of a block; bypass encoding residual coefficient signs and record the residual coefficient signs together with the encoded block; record encoding parameter sets, such as coding modes, intra-frame prediction modes or inter-frame prediction modes, and motion information encoded in the syntax structure of the encoded block (such as a picture parameter set ("PPS") found in a picture header, and a sequence parameter set ("SPS") found in a sequence of multiple pictures); and output the encoded block.

[0061] The VVC standard encoder configures one or more processors of the computing system to output a coded image consisting of a plurality of coded blocks from the entropy encoder 124. The coded image is output to a transmission buffer and is eventually packaged into a bit stream and output from the VVC standard encoder.

[0062] In the decoding process 150 , the VVC standard decoder configures one or more processors of a computing system to receive one or more encoded images from a bitstream as input.

[0063] The VVC standard decoder implements an entropy decoder 152. One or more processors of the configuration computing system are configured to perform entropy decoding, wherein, according to CABAC, the bins are decoded by reversing the mapping of symbols to bins, thereby recovering a plurality of quantized residual coefficients encoded by entropy. The entropy decoder 152 outputs a plurality of quantized residual coefficients, outputs a plurality of residual coefficient symbols bypassed by encoding, and also outputs a plurality of syntax structures such as PPS and SPS.

[0064] The VVC standard decoder also implements inverse quantization 154 and inverse transform 156. One or more processors of the computing system are configured to perform inverse quantization and inverse transform operations on the decoded quantized residual coefficients by matrix arithmetic operations, which are the inverse of the quantization and transform operations described above. The inverse quantization and inverse transform operations produce a reconstructed residual.

[0065] In addition, based on multiple coding parameter sets recorded by the entropy encoder 124 in syntax structures such as PPS and SPS (or, optionally, received or encoded into the decoder via out-of-band transmission), and the coding modes included in the multiple coding parameter sets, the VVC standard decoder determines whether to apply intra-frame prediction 156 (i.e., spatial prediction) or motion compensated prediction 158 (i.e., temporal prediction) to the reconstructed residual.

[0066] In the event that the encoding parameter sets specify intra prediction, the VVC standard decoder configures one or more processors of the computing system to use the prediction information specified in the plurality of encoding parameter sets to perform intra prediction 158. The intra prediction 158 thereby generates a prediction signal.

[0067] Where the coding parameter set specifies inter prediction, the VVC standard decoder configures one or more processors of the computing system to perform motion compensated prediction 160 using reference pictures from the DPB 200. The motion compensated prediction 160 thereby generates a prediction signal.

[0068] The VVC standard decoder also implements an adder 162. The adder 162 configures one or more processors of the computing system to perform an addition operation on the reconstructed residual and the prediction signal, thereby outputting a reconstructed block.

[0069] The VVC standard decoder further implements a loop filter 164. One or more processors of the computing system are configured to apply a loop filter (such as a deblocking filter, a SAO filter, and an ALF) to the reconstructed block and output a filtered reconstructed block.

[0070] The VVC standard decoder further configures one or more processors of the computing system to output the filtered reconstructed block to DPB 200. As described above, DPB 200 stores a plurality of reconstructed images, which are used by one or more processors of the computing system as a plurality of reference images in a plurality of coded images other than the current image, as described above with reference to motion compensated prediction.

[0071] The VVC standard decoder further configures one or more processors of the computing system to output the reconstructed image from the DPB to a user-viewable display of the computing system, such as a television display, a personal computing monitor, a smartphone display, or a tablet computer display.

[0072] Therefore, as shown in the above encoding process 100 and decoding process 150, both the VVC standard encoder and the VVC standard decoder implement motion prediction coding according to the VVC specification. The VVC standard encoder and the VVC standard decoder both configure one or more processors of the computing system to generate a reconstructed image based on a previous reconstructed image of the DPB according to the motion compensation prediction described in the VVC standard, wherein the previous reconstructed image is used as a reference image in the motion compensation prediction described herein.

[0073] The VVC standard adopts decoder-side motion vector refinement ("DMVR") based on bilateral matching ("BM") in bidirectional prediction to improve the accuracy of multiple MVs in merge mode. In DMVR, multiple refined MVs are searched near the initial multiple MVs, MV0 and MV1 in reference image list 0 ("L0") and reference image list 1 ("L1"), where the multiple refined MVs are denoted as MV0' and MV1', respectively. The BM method calculates the distortion between two candidate blocks in each of the reference images L0 and L1.

[0074] Figure 2 Motion prediction performed on a current image 202 according to bidirectional prediction is shown, wherein a plurality of offset blocks of a plurality of reference images are used to calculate a refined motion vector, which in turn is used to generate a bidirectional prediction signal. The current image 202 includes a current block 202A. According to the bidirectional prediction technique, two co-located reference images 204 and 206 are shown, one from reference list 0 in a first temporal direction and the other from reference list 1 in a second temporal direction. Motion information of the current block 202A refers to the co-located reference block 204A of the co-located reference image 204, and refers to the co-located reference block 206A of the co-located reference image 206. The co-located reference image 204 further includes an offset block 204B adjacent to the co-located reference block 204A, and the co-located reference image 206 further includes an offset block 206B adjacent to the co-located reference block 206A.

[0075] like Figure 2 As shown, the sum of absolute differences ("SAD") between reference block 204A and offset block 204B, and the SAD between reference block 206A and offset block 206B are calculated. The MV candidate with the lowest SAD is set as the refined MV and used to generate the bidirectional prediction signal.

[0076] In addition, according to the VVC standard, the application of DMVR is limited and is only applicable to multiple CBs encoded with the following multiple modes and features: CB-level merge mode with multiple MVs with bidirectional prediction, where the multiple MVs of bidirectional prediction point to respective reference images in multiple different temporal directions relative to the current image (i.e., one reference image is in the past and the other reference image is in the future); the distances of the two reference images to the current image (i.e., the picture order count ("POC") difference) are the same; both reference images are short-term reference images; the current CB has more than 64 luma samples; the CB height and CB width are both greater than or equal to 8 luma samples; the bidirectional prediction weight index with coding unit weight ("BCW") indicates equal weight (it should be understood that in the context of the weighted average bidirectional prediction formula that calculates the weighted average of two prediction signals, "equal weight" is a weight parameter that causes the two prediction signals to be equally weighted in the formula); weighted bidirectional prediction ("WP") is not enabled for the current block; and the combined inter-frame intra-frame prediction ("CHP") mode is not used for the current block.

[0077] The refined MV derived by DMVR is used to generate multiple inter-frame prediction samples and is also used for temporal motion vector prediction for future image coding. The original MV is used for deblocking and is also used for spatial motion vector prediction for future CB coding.

[0078] Additional features of DMVR are mentioned below.

[0079] In DVMR, the refined MV search starts from the search center and includes the search range of multiple refined MVs adjacent to an initial MV. The span of the search range describes a search window, and the searched multiple refined MV ranges are offset according to the MV difference mirror rule. In other words, any point searched by DMVR is represented by a candidate MV pair (MV0, MV1), which follows the following formulas 1 and 2 respectively: MV0′=MV0+MV offset MV1′=MV1-MV offset MV offset Represents the MV refinement offset between the initial MV and the refined MV in one of the reference images. The refined MV search range (hereinafter also referred to as "search step") is two integer distance luma samples from the initial MV. The refined MV search consists of two stages: integer sample offset search and fractional sample refinement.

[0080] For purposes of understanding the example embodiments of the present disclosure, all subsequent references to one or more "points" being searched for should be understood to refer to a plurality of individual luma samples of a block or sub-block separated by integer distances.

[0081] A 25-point full search is applied to the integer sample offset search. The SAD of the initial MV pair is first calculated. If the SAD of the initial MV pair is less than the threshold, the integer sampling stage of DMVR is terminated. Otherwise, the remaining 24 search points are searched in raster scan order, and the SAD of each search point is calculated. The search point with the smallest SAD is selected as the integer distance refined MV, which is output by the integer sample offset search. In order to reduce the loss caused by the uncertainty of DMVR refinement, the original MV can be given priority in the DMVR process. The SAD between the reference blocks referenced by the initial MV candidates is reduced by 1 / 4 of the SAD value.

[0082] The integer sample offset search may be followed by fractional sample refinement. To reduce computational complexity, fractional sample refinement is performed by solving the parameter error surface formula instead of further searching by SAD comparison. Fractional sample refinement is conditionally called based on the output of the integer sample offset search. When the integer sample offset search terminates with the center with the minimum SAD in the first or second iteration search, fractional sample refinement is further applied. Otherwise, the integer distance refinement MV may be output as the refinement MV.

[0083] In the parametric error surface based sub-pixel offset estimation, a 2-D parabolic error surface is fitted using the center position cost and the costs at four neighboring positions from the center, as described in the following Equation 3: E(x, y) = A(x min ) 2 +B(yy min ) 2 +C Where (x min and min ) corresponds to the fractional position with the minimum cost, and C corresponds to the minimum cost value. By solving the above formula using the cost values ​​of the five search points, (x min and min ) are calculated according to the following formula 4 and formula 5 respectively: x min =(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0))) y min =(E(0,-1)-E(0,1)) / (2((E(0,-1)+E(0,1)-2E(0,0)))

[0084] x min and min The value of is clamped by default between -8 and 8, since all cost values ​​are positive and the minimum is E(0,0). This corresponds to a half-pixel shift with 1 / 16 pixel MV precision in VVC. minand min ) is added to the integer distance refinement MV to obtain a sub-pixel accurate refinement delta MV. The sub-pixel accurate refinement delta MV can be output as a refinement MV instead of an integer distance refinement MV.

[0085] In VVC, the resolution of multiple MVs is 1 / 16 luma samples. Multiple samples at fractional positions are interpolated using an 8-tap interpolation filter. In DMVR, since multiple refined MV search points are points directly surrounding the initial fractional pixel MV with an integer sample offset, samples at those fractional positions need to be interpolated for DMVR refined MV search. In order to reduce computational complexity, a bilinear interpolation filter is used to generate multiple fractional samples for DMVR refined MV search. In addition, by using a bilinear filter with a 2-sample search range, DVMR does not access more reference samples than the standard motion compensation process. After the refined MV is output by the DMVR refined MV search, a standard 8-tap interpolation filter is applied to generate the final prediction. In order not to access more reference samples than the standard motion compensation process, samples that are not required by the interpolation process based on the original MV but are required by the interpolation process based on the refined MV are filled in from those available samples.

[0086] When the width and / or height of a CB is greater than 16 luma samples, it will be further split into multiple sub-blocks with width and / or height equal to 16 luma samples. The maximum unit size of the DMVR refinement MV search is limited to 16×16.

[0087] In ECM, to further improve coding efficiency, multi-pass decoder-side motion vector refinement is applied. In the first pass, BM is applied to the coding block. In the second pass, BM is applied to each 16×16 sub-block within the coding block. In the third pass, MVs in each 8×8 sub-block are refined by applying bidirectional optical flow (“BDOF”). Multiple refined MVs are stored for spatial and temporal motion vector prediction.

[0088] In the first pass, a refined MV is derived by applying BM to the coding block. Similar to DMVR, in bidirectional prediction, the refined MV is searched near the two initial MVs (MV0 and MV1) in the reference picture lists L0 and L1. Based on the minimum bilateral matching cost between the two reference blocks L0 and L1, refined MVs (MV0_pass1 and MV1_pass1) are derived near multiple initial MVs.

[0089] A local search is performed based on the refinement of the BM to derive the integer sample precision intDeltaMV. The local search uses a 3×3 square search pattern, looping through the horizontal search range [-sHor, sHor] and the vertical search range [-sVer, sVer], where the values ​​of sHor and sVer are determined by the block size, and the maximum values ​​of sHor and sVer are 8 or other values. For example, in Figure 3A In , point 0 is the position of the initial MV reference and is set as the first search center. Therefore, points 1 to 8 around the initial point are first searched and the cost of each position is calculated.

[0090] In the first search iteration, it is found that point 7 has the smallest cost, so point 7 is set as the second search center, and points 9, 10, and 11 are searched. In the next search iteration, it is found that the cost of point 10 is less than the costs of points 7, 9, and 11, so the third search center is set to point 10, and points 12, 13, and 14 are searched. In the next search iteration, it is found that point 12 has the smallest cost among points 6 to 14, so point 12 is set as the fourth search center. In the next search iteration, it is found that the costs of points 10, 11, 13, and 15 to 19 around point 12 are all greater than the cost of point 12, then point 12 is the optimal point and the refined MV search is terminated, and the refined MV corresponding to the optimal point is output. Therefore, Figure 3A An example diagram showing a search pattern (eg, a 3x3 square search pattern) used in the first pass of multi-pass decoder-side motion vector refinement.

[0091] The bilateral matching cost can be calculated as: bilCost = mvDistanceCost + sadCost, where sadCost is the SAD between the L0 prediction value (i.e., the reference block from the reference image L0) and the Ll prediction value (i.e., the reference block from the reference image L1) at the search point, and mvDistanceCost is based on intDeltaMV (i.e., the distance between the search point and the initial point). When the block size cbW (CB width, in pixels) * cbH (CB height, in pixels) is greater than 64, the mean SAD ("MRSAD") cost function is applied to remove the discrete cosine ("DC") effect of distortion between reference blocks. When the bilCost at the center point of the 3×3 search pattern has the minimum cost, the intDeltaMV local search terminates. Otherwise, the current minimum cost search point is set to the new center point of the 3×3 search pattern, and the search for the minimum cost continues until the end of the search range is reached.

[0092] The existing fractional sample refinement is further applied to derive the fractional MV refinement fracDeltaMV, and the final deltaMV is derived as intDeltaMV+fracDeltaMV. Then, multiple refined MVs after the first pass are derived according to the following formulas 6 and 7 respectively: MV0_pass1=MV0+deltaMV MV1_pass1=MV1-deltaMV

[0093] In the second pass, refined MVs are derived by applying BM to 16×16 grid sub-blocks. For each sub-block, the refined MV is searched near the two MVs (MV0_pass1 and MV1_pass1) obtained in the first pass in the reference image lists L0 and L1. Refined MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the two reference sub-blocks in L0 and L1.

[0094] For each sub-block, a full search is performed based on the refinement of the BM to obtain integer sample precision intDeltaMV(sbIdx2). The horizontal search range of the full search is [-sHor, sHor], and the vertical search range is [-sVer, sVer], where the values ​​of sHor and sVer are determined by the block size, and the maximum values ​​of sHor and sVer are 8 or other values.

[0095] The bilateral matching cost can be calculated by applying a cost factor to the sum of the absolute transform difference (“SATD”) costs between the two reference sub-blocks as follows: bilCost = satdCost * costFactor. The search area (2*sHor+1)*(2*sVer+1) is divided into up to 5 diamond search areas, such as Figure 4 shown. Figure 4A diagram showing multiple bilateral matching costs (each matching cost corresponds to a differently shaded diamond search area) used in the second pass of multi-channel decoder-side motion vector refinement. Each search area is assigned a costFactor, which is determined by the distance intDeltaMV(sbIdx2) between each search point and the starting MV, and each diamond area is processed sequentially starting from the center of the search area. In each area, the search points are processed in a raster scan order starting from the upper left corner of the area to the lower right corner. The int-pel full search ends when the minimum bilCost in the current search area is less than or equal to a threshold of sbW (sub-block width)*sbH (sub-block height); otherwise, the int-pel full search continues to the next search area until all search points are checked. In addition, if the difference between the previous minimum cost and the current minimum cost in the iteration is less than a threshold equal to the area of ​​the block, the search is terminated.

[0096] Furthermore, the bilateral matching cost as described above may also be calculated based on MRSAD instead of SAD, and may also be calculated based on the mean-removed sum of absolute transform differences ("MRSATD") instead of SATD.

[0097] The existing VVC DMVR fractional sample refinement is further applied to derive the final deltaMV (sbIdx2). Then, the second pass of refined MV is derived according to the following formulas 8 and 9 respectively: MV0_pass2(sbIdx2)=MV0_pass 1+deltaMV(sbIdx2) MV1_pass2(sbIdx2)=MV1_passl-deltaMV(sbIdx2)

[0098] In the third pass, a refined MV is derived by applying BDOF to the 8×8 grid sub-blocks. For each 8×8 grid sub-block, BDOF refinement is applied to derive the scaled v x and v y , instead of starting from the refined MV of the second pass of the parent-child block. x , v y ) are rounded to 1 / 16 sample precision and clipped between -32 and 32.

[0099] The refined MVs (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) of the third pass are derived according to the following formulas 10 and 11, respectively: MV0_pass3(sbIdx3)=MV0_pass2(sbIdx2)+bioMv MV 1_pass3(sbIdx3)=MV0_pass2(sbIdx2)-bioMv

[0100] In ECM, adaptive decoder-side motion vector refinement is an extension of multi-pass DMVR, which includes two new merge modes to refine MVs in only one temporal direction (reference image L0 or reference image LI) of bidirectional prediction of merge candidates that meet DMVR conditions. The multi-pass DMVR process is applied to the selected merge candidate to refine multiple motion vectors, however, MVD0 or MVD1 is set to 0 in the first pass (i.e., PU level) DMVR. Therefore, a new merge candidate list is constructed for adaptive decoder-side motion vector refinement. The new merge mode for the new merge candidate list is called BM merge in ECM.

[0101] The multiple merge candidates for BM merge mode are multiple derived from spatially adjacent coding blocks, multiple temporal motion vector predictors ("HMVPs"), multiple non-adjacent blocks, multiple history-based motion vector predictors ("HMVPs"), and multiple pairwise candidates, similar to the conventional merge mode. The difference is that only those merge candidates that satisfy the DMVR conditions are added to the merge candidate list. The two new merge modes use the same merge candidate list. If the BM candidate list contains inherited BCW weights and the DMVR process remains unchanged, unless MRSAD or MRSATD is used for distortion calculation when the weights are unequal, and the BCW weights are used to weight bidirectional prediction. The encoding for merge indexes is the same as for regular merge mode.

[0102] In HEVC, only the translation motion model is applied to motion compensated prediction ("MCP"). However, in the real world, various motions occur, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensated prediction is applied. Figure 5A and 5B As shown, the affine motion field of the block is composed of two control point motion vectors (4 parameters) ( Figure 5A ) or three control point motion vectors (6 parameters) ( Figure 5B )’s motion information description.

[0103] In affine motion compensation, for a 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived according to the following formula 12: For the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived according to the following formula 13: Where (mv 0x , mv 0y ) is the motion vector of the upper left control point, (mv 1x ,mv 1y ) is the motion vector of the upper right control point, and (mv 2x ,mv 2y ) is the motion vector of the lower left control point.

[0104] In order to simplify motion compensation prediction, block-based affine transformation prediction is applied. In order to derive the motion vector of each 4×4 luminance sub-block, the motion vector of the center sample of each sub-block is calculated according to the above formula, such as Figure 6 As shown, and rounded to 1 / 16 fractional accuracy. Figure 6 Diagram illustrating affine motion vectors of multiple luma subblocks calculated for each subblock center sample. According to ECM, the subblock size is adaptively determined. If the motion vector difference of two adjacent luma subblocks is less than a threshold, the two adjacent luma subblocks will be merged into a larger subblock. If the motion vector difference of two adjacent larger subblocks is still less than the threshold, the merging of larger subblocks will continue until the motion vector difference of two adjacent subblocks is greater than the threshold or until the subblock is equal to the whole block.

[0105] After deriving the motion vectors of the sub-blocks, a motion compensation interpolation filter is applied to generate prediction values ​​for each sub-block using the derived motion vectors. The sub-block size of the chrominance component depends on the size of the luminance sub-block. The MV of the chrominance sub-block is calculated as the average of the MVs of the upper left and lower right luminance sub-blocks in the collocated luminance region.

[0106] As done for translational motion inter prediction, there are two affine motion inter prediction modes: affine merge mode and affine advanced motion vector prediction ("AMVP") mode.

[0107] Affine merge mode (AF_MERGE) can be applied to multiple CBs with width and height both greater than or equal to 8. In this mode, multiple control point motion vectors ("CPMV") of the current CB are generated based on the motion information of spatially neighboring CBs. There can be up to 15 affine candidates, and the index is signaled to indicate the affine candidate to be used for the current CB. The following eight types of candidates are used to form the affine merge candidate list: candidates inherited from adjacent neighboring blocks; candidates inherited from non-adjacent neighboring blocks; candidates constructed from adjacent neighboring blocks; affine candidates of the second type of construction from non-adjacent neighboring blocks; affine candidates of the first type of construction from non-adjacent neighboring blocks; regression-based affine merge candidates; pairwise affine; and 0MV.

[0108] The inherited affine candidates are derived from the affine motion models of multiple adjacent or non-adjacent blocks. When an adjacent or non-adjacent affine CB is identified, its multiple control point motion vectors are used to derive the control point motion vector prediction ("CPMVP") candidates in the affine merge candidate list of the current CB. Figure 7 As shown in (a diagram showing the inheritance of control point motion vectors, in which candidates are used to form an affine merge candidate list), if the adjacent lower left block A is encoded in affine mode, the motion vectors v2, v3 and v4 of the upper left corner, upper right corner and lower left corner of the CB containing block A are obtained. For block A encoded with a 4-parameter affine model, two CPMVs of the current CB are calculated based on v2 and v3. For block A encoded with a 6-parameter affine model, three CPMVs of the current CB are calculated based on v2, v3 and v4.

[0109] For those candidates inherited from multiple non-adjacent neighboring blocks, the non-adjacent spatial neighboring blocks are checked based on their distance to the current block, i.e., from near to far. At a certain distance, only the first available neighboring blocks (encoded using affine mode) from each side (e.g., left and top) of the current block are included to derive the inheritance candidates. Fig. 8A As shown by the dotted arrows in , the inspection order of the adjacent blocks on the left and above is from bottom to top and from right to left, respectively. Fig. 8A and Figure 8B Diagrams showing a plurality of inheritance candidates from non-adjacent neighboring blocks of an affine merge candidate list and a plurality of construction candidates of a first type of an affine merge candidate list, respectively.

[0110] Affine candidates constructed from multiple adjacent neighboring blocks are candidates constructed by combining the adjacent translational motion information of each control point. The motion information of these control points is derived from the specified adjacent spatial and temporal neighbors. Fig. 9 As shown ( Fig. 9 A diagram showing the positions of affine candidates constructed from multiple adjacent neighboring blocks of the affine merge candidate list). CPMV k (k=1, 2, 3, 4) indicates the kth control point. For CPMV1, check B2->B3->A2 blocks and use the MV of the first available block. For CPMV2, check B1->B0 blocks, and for CPMV3, check A1->A0 blocks. If available, use TMVP as CPMV4.

[0111] After obtaining the MVs of the four control points, multiple affine merge candidates are constructed based on the motion information. The following combinations of these control point MVs are used to construct in order: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3} The combination of 3 CPMVs constructs a 6-parameter affine merge candidate, and the combination of 2 CPMVs constructs a 4-parameter affine merge candidate. To avoid the motion scaling process, if the reference indexes of the control points are different, the relevant combinations of these control point MVs are discarded.

[0112] For the first type of candidates constructed from multiple non-adjacent neighboring blocks, such as Figure 8B As shown, firstly, the positions of a plurality of non-adjacent spatial adjacent blocks on the left and above are independently determined; then, the position of the upper left adjacent block can be determined accordingly, which can be enclosed together with the non-adjacent adjacent blocks on the left and above to form a rectangular virtual block. Then, as Fig.10 As shown, the motion information of three non-adjacent neighboring blocks is used to form the CPMVs of the upper left (A), upper right (B), and lower left (C) of the virtual block, which are finally projected to the current CB to generate the corresponding construction candidates. Fig.10 A diagram showing the locations of multiple affine candidates constructed from multiple non-adjacent neighboring blocks of an affine merge candidate list.

[0113] For the second type of constructed multiple candidates, multiple affine model parameters are inherited from multiple non-adjacent spatial neighboring blocks. Specifically, the second type of multiple affine constructed candidates are generated according to the following combination: 1) MVs of multiple adjacent 4×4 blocks; and 2) multiple affine model parameters inherited from multiple non-adjacent spatial neighboring blocks, such as Fig. 8A Defined in .

[0114] For multiple affine merge candidates based on regression, the sub-block motion field from the previously encoded affine CB and the motion information of multiple neighboring sub-blocks from the current CB are used as inputs to the regression process to derive multiple affine candidates. The previously encoded affine CB can be identified by scanning multiple non-adjacent positions and the affine HMVP table. Fig.11 As shown, the neighboring sub-block information of the current CB is obtained from multiple 4×4 sub-blocks represented by the gray area. For each sub-block, the corresponding motion vector and center coordinates of the sub-block are available given the reference list. For each affine CB, up to 2 regression-based affine candidates can be derived. One with neighboring sub-block information and one without. All candidates generated by linear regression are pruned and collected into a candidate subgroup, and when ARMC is enabled, the TM cost-based ARMC process is applied. Then, when N affine CBs are found, up to N multiple candidates generated by linear regression are added to the affine merge candidate list.

[0115] After all the above merge candidates are inserted into the merge candidate list, if the list is still not full, 0 MVs are inserted to the end of the list.

[0116] Compared with pixel-based motion compensation, sub-block based affine motion compensation can save memory access bandwidth and reduce computational complexity, but at the expense of prediction accuracy loss. In order to achieve finer granularity of motion compensation, prediction refinement with optical flow ("PROF") is used to refine sub-block based affine motion compensation predictions without increasing the memory access bandwidth for motion compensation. In VVC, after performing sub-block based affine motion compensation, the luma prediction samples are refined by adding the differences derived from the optical flow formula. PROF is described as the following four steps:

[0117] First, sub-block based affine motion compensation is performed to generate a sub-block prediction I(i, j).

[0118] Secondly, the spatial gradient g of the sub-block prediction x (i, j) and g y (i, j) is calculated using a 3-tap filter [-1, 0, 1] at each sample position according to the following formula 14 and formula 15, respectively. The gradient calculation is the same as the gradient calculation in BDOF. g x (i,j)=(I(i+1,j)>>shift1)-(I(i-1,j)>>shift1) g y (i,j)=(I(i,j+1)>>shift1)-(I(i,j-1)>>shift1) shift1 is used to control the accuracy of the gradient. Sub-block (i.e. 4×4) prediction is extended by one sample on each side for gradient calculation. To avoid extra memory bandwidth and extra interpolation calculations, those extended samples on the extended boundary are copied from the nearest integer pixel position in the reference image.

[0119] Third, the brightness prediction refinement is calculated according to the following optical flow formula 16: ΔI(i, j) = g x (i, j)*Δv x (i, j)+g y (i, j)*Δv y (i, j) Where Δv(i,j) is the difference between the sample MV calculated for the sample position (i,j) (denoted by v(i,j)) and the sub-block MV of the sub-block to which the sample (i,j) belongs, as Fig.12 As shown in Figure 1, Δv(i,j) is quantized in units of 1 / 32 brightness sample accuracy. Fig.12shows the sub-block motion vector v at a position SB A map of the difference Δv(i,j) between the motion vector v(i,j) calculated at that position and used as part of the refinement of the optical flow prediction based on affine motion compensated prediction of the sub-block.

[0120] Since the affine model parameters and the sample positions relative to the sub-block center do not vary from sub-block to sub-block, Δv(i,j) can be calculated for the first sub-block and reused for other sub-blocks in the same CB. Let dx(i,j) and dy(i,j) be the distance from the sample position (i,j) to the sub-block center (x SB ,y SB ) is derived according to the following formula 17. Then Δv(i,j) can be derived according to the following formula 18.

[0121] To maintain accuracy, the sub-block center (x SB ,y SB ) is calculated as ((WSB-1) / 2, (HSB-1) / 2), where WSB and HSB are the sub-block width and height, respectively.

[0122] For the 4-parameter affine model, the parameters C, D, E, and F are derived according to the following formula 19: For the 6-parameter affine model, the parameters C, D, E, and F are derived according to the following formula 20: Where (v 0x ,v 0y ), (v 1x ,v 1y ), (v 2x ,v 2y ) are the upper left, upper right and lower left control point motion vectors, w and h are the width and height of the CB.

[0123] Fourth, the luma prediction refinement ΔI(i,j) is added to the sub-block prediction I(i,j). The final prediction I′ is generated according to the following formula 21: I′(i,j)=I(i,j)+ΔI(i,j) PROF is not applicable to two cases of affine-encoded CBs: 1) all control point MVs are the same, which indicates that the CB has only translational motion; 2) multiple affine model parameters are larger than the specified limit, because sub-block based affine motion compensation is downgraded to CB based motion compensation to avoid large memory access bandwidth requirements.

[0124] According to the VVC standard and ECM proposal as described above, DMVR is only applied to multiple non-affine coded blocks to refine the motion vector of translational motion, and for multiple blocks encoded in affine mode, DMVR is not used to refine multiple motion vectors. However, the motion vectors of multiple affine merge mode encoded blocks (similar to the motion vectors of translational motion compensation encoded blocks) are also inherited from the previous encoded blocks, and therefore may not perfectly match the current block. Therefore, in the present disclosure, the VVC standard encoder and decoder configure one or more processors of the computing system to apply DMVR to multiple affine merge mode encoded blocks to refine the motion vector accuracy and thereby improve the encoding efficiency. .

[0125] It should be understood that “decoder side” does not mean that the method is only implemented by the decoder; on the contrary, the steps of the method can be similarly or identically implemented by the encoder and the decoder, as will be described later.

[0126] As described herein, in an affine model for encoding a block in affine merge mode, the motion vector at a sample position (x, y) may be derived according to the following formula 22: Where (mv x ,mv y ) is the motion vector derived at the sampling position (x, y); (mv 0x ,mv 0y ) represents the MV of the affine model, that is, the motion vector at the sample position (0,0); and a, b, c and d are the parameters of the affine model. Affine motion may include but is not limited to translation, rotation and scaling. The MV of the affine model represents the translational motion of the affine model. The affine model parameters a, b, c and d represent the non-translational motion of the affine model, including rotation and scaling and other non-translational motions. The affine model parameters can be derived based on the motion vectors at two sample positions of the 4-parameter affine model and three non-collinear sample positions of the 6-parameter affine model. As a generalization of the above formula 22, the MV of the affine model can be a motion vector at any sample position, not necessarily the motion vector at the position (0,0). The motion vector at the sample position (w,h) is taken as the MV of the affine model (expressed as (mv wx , mv wy ), the motion vector at the sample position (x, y) can be expressed according to the following formula 23:

[0127] For the 4-parameter affine model, b is equal to c and d is equal to a. Therefore, the 4-parameter affine model can be expressed according to the following formula 24:

[0128] Theoretically, all parameters of the affine model, including a, b, c, d, mv wx and mv wy can all be refined at once in DMVR. However, in order to limit the computational complexity of refinement, according to an exemplary embodiment of the present disclosure, the affine model parameters a, b, c and d are fixed while the affine model MV (mv wx , mv wy ) is refined, and the affine model MV is fixed while the affine model parameters a, b, c and d are refined. Subsequently, a refined MV search in which the affine model parameters are fixed is described, and an affine parameter shift search in which the affine model MV is fixed is also described.

[0129] Similar to the standard DMVR implementation, the VVC standard encoder and decoder configure one or more processors of the computing system to apply a DMVR refined MV search to the initial MV, as described above, except as noted below. Compared to the standard DMVR implementation, during the refined MV search, when calculating the bilateral matching cost (whether based on SAD or SATD, or based on MRSAD or MRSATD) between two prediction values ​​of the reference image L0 and the reference image L1 (i.e., a reference block from the reference image L0 and a reference block from the reference image L1), motion compensation is performed at the sub-block level. Therefore, a refined MV search is performed on the initial MV for a certain number of iterations in the integer sample offset search, which may be followed by fractional sample refinement, and the refined MV search outputs a refined MV. An affine model is used to derive the MV of each sub-block of the current affine coded block, and motion compensation is performed at the sub-block level to obtain the prediction value of the current affine coded block. The SAD or SATD (or MRSAD or MRSATD) of the two prediction values ​​(one from the L0 reference image and the other from the L1 reference image) of the current affine coded block is calculated to derive the bilateral matching cost for each current search point. An integer sample offset search terminates after a certain number of search iterations, producing an optimal point, which can be output as a refined MV, or fractional sample refinement can be further applied to it before outputting the refined MV.

[0130] In one example, the CPMVs are refined. Starting from a set of initial CPMVs with reference to an initial point, MV refinement offsets are added to each CPMV to obtain surrounding search points according to the following formulas 25, 26, 27, 28, 29, and 30: CPMV0_l0′=CPMV0_l0+MV_offset CPMV0_l1′=CPMV0_l1-MV_offset CPMV1_l0′=CPMV1_l0+MV_offset CPMV1_l1′=CPMV1_l1-MV_offset CPMV2_l0′=CPMV2_l0+MV_offset CPMV2_l1′=CPMV2_l1-MV_offset Where CPMVx_l0 is the x-th CPMV of reference image L0, and CPMVx_l1 is the x-th CPMV of reference image L1. MV_offset is the MV refinement offset of the search point, that is, the difference between the initial CPMV and the refined CPMV. An affine model is applied to calculate the MV of each sub-block, and then sub-block level motion compensation is applied to obtain the prediction value of the current block. During the refined MV search, the multiple affine model parameters as described above are fixed. It should be understood that although the refined MV search includes multiple search iterations, these iterations together constitute a search, and the refined MV search itself does not need to be performed multiple times (because the MV refinement offset is immutable).

[0131] During each iteration in the refined MV search, multiple search points are searched around the search center, where each search point is a separate luminance sample as described above; to search multiple search points, a standard search scheme may be applied. Figure 3A As shown, a 3×3 square search around the search center can be performed in the integer sample offset search; then, a fractional search and a fractional error surface estimation method can be applied to derive the optimal MV offset. As another example, a cross search around the search center is used to reduce the search points. Figure 3B As shown, the position point 0 referred to by the initial MV is set as the first search center. Therefore, points 1 to 4 around the search center are first searched, and the cost of each search point is calculated.

[0132] In the first search iteration, point 4 is found to have the minimum cost, point 4 is set as the second search center, and points 5, 6, and 7 are searched. In the next search iteration, it is found that the cost of point 7 is less than the costs of points 4, 5, and 6, so the third search center is set to point 7, and points 8 to 10 are searched. In the next search iteration, point 9 is found to have the minimum cost among points 7 to 10, so point 9 is set as the fourth search center. In the next search iteration, it is found that the costs of points 6, 16, and 18 around point 9 are all greater than the cost of point 9, so point 9 is the optimal point.

[0133] As yet another example, a 3×3 square search and a 3×3 cross search may be performed in combination, performing a square search in the first k iterations and then performing a cross search to determine the optimal point; or performing a cross search first and then performing a square search to determine the optimal point.

[0134] In another embodiment, the VVC standard encoder and decoder configure one or more processors of the computing system to apply an adaptive search step size to speed up the refinement of the MV search. In the previous or previous k earlier search iterations, the search step size is set to an initial step size (e.g., 2), and in one or more later search iterations after the earlier search iteration, the search step size is changed to a smaller step size (e.g., 1). Figure 3C As shown, point 0 is the position referenced by the initial MV and is set as the position of the first search center. First, points 1 to 8 around the initial point are searched, and the search step is set to 2, so the distance between points 1, 2, 3, 4, 5, 6, 8 and point 0 is 2 pixels (Manhattan distance is used here instead of Euclidean distance).

[0135] Next, in the second search iteration, the cost at point 7 is the minimum cost, the search center is point 7, and the search step size is still set to 2. Search points 9, 10, and 11 are all 2 pixels away from the search center point 7, and point 10 is found to have the minimum cost.

[0136] Next, in the third search iteration, the search center is point 10, and the search step size becomes 1. Search points 12 to 14 are all 1 pixel away from point 10. Assume that point 13 is found to have the minimum cost.

[0137] Next, in the fourth search iteration, the search center is 13, the search step is 1, and points 11, 15, 16, 17, and 18 are each 1 pixel away from the search center point 13. The adaptive search step can also be applied to the cross search mode or another search mode.

[0138] For each search point, the sub-block MV is recalculated using the CPMV of the current search point, and the prediction value of the block is derived by performing sub-block based affine motion compensation using the derived sub-block MV.

[0139] The cost of each search point can be calculated as cost = mvDistanceCost + SadCost, where SadCost is the SAD between the L0 prediction value and the L1 prediction value of the current block, and mvDistanceCost is based on the distance between the search point and the initial point (ie, the difference between the refined CPMV and the initial CPMV).

[0140] To control the complexity of refinement, PROF can be applied or not before SAD calculation during refined MV search. If PROF is applied, then for each search point, after obtaining the prediction value, PROF is applied to refine the prediction value and SAD is calculated between two PROF-refined prediction values. If PROF is not applied, then for each search point, SAD is calculated directly after obtaining the L0 prediction value and the L1 prediction value.

[0141] In addition, to further reduce the complexity of refining the MV search, for each search point, a 2-tap bilinear interpolation filter may be used to generate a prediction value instead of an 8-tap or 12-tap interpolation filter.

[0142] In another example, the VVC standard encoder and decoder configure one or more processors of the computing system to perform sub-block MV refinement. The sub-block MV is derived using multiple initial CPMVs, and then the MV refinement offset is added to each sub-block MV to refine the sub-block MV according to the following formulas 31 and 32: MV(sbx)_l0′=MV(sbx)_l0+MV_offset MV(sbx)_l1′=MV(sbx)_l1-MV_offset Where MV(sbx)_l0 and MV(sbx)_l1 are the MVs of sub-block X referring to reference image L0 and reference image L1, respectively. The MV_offset of the current search point is added to all sub-block MVs to obtain their refined sub-block MVs. Motion compensation is then performed on each sub-block MV at the sub-block level to obtain two prediction values ​​for the entire CB. SAD or SATD (or MRSAD or MRSATD) is calculated between the two prediction values, and the bilateral matching cost of the current search is obtained. The search point with the minimum cost is taken as the optimal point, and the corresponding MV_offset is obtained as the optimal MV refinement offset. The refined sub-block MV can be calculated according to Formula 31 and Formula 32. During the refined MV search, PROF may or may not be applied before the SAD calculation. An interpolation filter with a reduced number of taps can be used instead of an 8-tap or 12-tap interpolation filter to generate multiple prediction values ​​during the refined MV search. The search method and cost calculation method applied in CPMV refinement can also be applied in sub-block MV refinement.

[0143] To reduce the search complexity, pre-interpolated samples of the prediction values ​​at all search points within the search window can be stored in a buffer before refining the MV search. Then, for each search point, the prediction value of each sub-block can be taken directly from the buffer without interpolation. Fig.13As shown in FIG. 1 , the coding block is divided into 16 s4×4 subblocks for affine motion compensation. Multiple initial CPMVs are used to calculate the initial subblock MV for each subblock. For each subblock, the reference subblock (i.e., the prediction value of the subblock) can be located in the reference image through the initial subblock MV. Because refinement is applied to the subblock MV, for each search point, all reference subblocks are shifted by the same MV refinement offset. Therefore, for each subblock, the samples of the reference subblock at all search points within each search window can be pre-interpolated, as shown in FIG. Fig.13 Then, for a search point, the sample of the reference block at the search point can be directly extracted from the search window without interpolation, thus saving the computation time and resources for further multiple interpolations.

[0144] The VVC standard encoder and decoder can configure one or more processors of the computing system to apply pre-interpolation in integer sample offset search and fractional sample refinement (i.e., all samples in the search window are interpolated before the refined MV search, so that for each search point, the predicted sample can be obtained directly and no interpolation call is required). For integer sample offset search, the pre-interpolated samples are all one pixel away from each other. In fractional sample refinement, the pre-interpolated samples can be stored separately according to the phase. Fig.14 As shown, squares represent samples of integer search points that can all be pre-interpolated, X symbols represent samples at 1 / 2 pixel positions in the horizontal direction but at integer positions in the vertical direction; triangles represent samples at 1 / 2 pixel positions in the vertical direction but at integer positions in the horizontal direction; and circles represent samples at 1 / 2 pixel positions in both the horizontal and vertical directions. Given the 1 / 2 pixel search process, there can be three samples of different phases, and the distance between two adjacent fractional samples with the same phase is also a pixel. Therefore, fractional samples of the same type can be pre-interpolated together, and different phases of the samples can be stored separately.

[0145] After CB level MV refinement, sub-CB level MV refinement can also be applied. For example, in affine motion compensation, the CB is divided into multiple sub-CBs (e.g., 16×16 sub-CBs) larger than the sub-block. For each sub-CB, the affine model MV is further refined: multiple sub-blocks in one sub-CB share the same MV refinement offset. The final MV of each respective sub-block can be expressed according to the following formula 33 and formula 34: MV(sbx)_l0′=MV(sbx)_l0+MV_offset+MV_offset(sbCUx) MV(sbx)_l1′=MV(sbx) l1 -MV_offset-MV_offset(sbCUs) Where MV(sbx)_l0 and MV(sbx)_l1 are the MVs of L0 and L1 of sub-block X, respectively. MV_offset is the MV refinement offset obtained in the CB level MV refinement process, and MV_offset(sbCUx) is the MV refinement offset obtained in the sub-CB level MV refinement of the sub-CB X where sub-block X is located. MV(sbx)_l0′ and MV(sbx)_l1′ are the final refined MVs of sub-block X. Motion compensation is performed at the sub-block level using the refined sub-block MVs.

[0146] The search range can be set according to the PU size or multiple QPs, reducing the computational complexity at the expense of performance, or improving performance at the expense of increased computational complexity. As an example, because a larger CB can benefit from greater refinement, a larger search range can be set for a larger CB, and a smaller search range can be set for a smaller CB. As another example, a larger QP can benefit from a larger search range due to greater distortion. Therefore, a larger search range can be set for a larger QP, and a smaller search range can be set for a smaller QP. Alternatively, in order to reduce encoding time, setting a smaller search range for a larger QP can significantly reduce the points searched. Therefore, in some instances, a smaller search range can be set for a larger QP, and a larger search range can be set for a smaller QP.

[0147] To further reduce computational complexity, PU size restrictions can be added. The PU size restriction parameter specifies the process of skipping affine DMVR for multiple CBs of certain sizes. For example, affine DMVR is not applied to CBs smaller than 8×8 or 16×16, or affine DMVR is not applied to CBs larger than 64×64 or 128×128.

[0148] In some examples, to reduce computational complexity, the VVC standard encoder and decoder configure one or more processors of the computing system to apply early termination to the DMVR refined MV search of the affine block. As an example, if a search point is checked in the search of a previous candidate in the affine merge candidate list, the search point is skipped. As another example, if the SAD at the search point is less than a predefined threshold, the refined MV search is terminated and the search point is used as the optimal position after refinement.

[0149] In some examples, in order to reduce the complexity of the encoder and decoder, the VVC standard encoder and decoder configure one or more processors of the computing system to apply a fast algorithm. As an example, if the difference between the current affine merge candidate (i.e., the current initial MV) and the previously checked affine merge candidate (i.e., the previous initial MV) is less than a threshold, the entire process of DMVR for the current affine merge candidate is skipped. The current affine merge candidate is directly used for motion compensation of the current block without refinement. The threshold may depend on the current block size so that larger blocks have a larger threshold than smaller blocks. As another example, DMVR is disabled for affine coded blocks with a size less than / greater than a threshold. DMVR is not applied to blocks smaller than the threshold, or to blocks larger than the threshold, to refine motion. The threshold may be fixed or signaled in the bitstream.

[0150] As mentioned above, the multiple parameters of the affine model include a, b, c, d and mv wx ,mv wy can be refined in DMVR; therefore, in addition to the MV of the affine model, the parameters of the affine model can also be refined.

[0151] In some examples, the VVC standard encoder and decoder configure one or more processors of the computing system to perform 4-parameter refinement on an affine model. Based on the affine model represented in Formula 23, offset_a, offset_b, offset_c, and offset_d are added to parameters a, b, c, and d to refine these parameters. Both the encoder and the decoder search for offset_a, offset_b, offset_c, and offset_d to minimize the bilateral matching cost of the L0 prediction value (i.e., reference to the reference image L0) and the L1 prediction value (i.e., reference block of the reference image L1) of the affine coding block (subsequently referred to as "affine parameter offset search", where the parameter offset that minimizes the bilateral matching cost is subsequently referred to as the "optimal parameter offset"). The refinement of each parameter is performed according to the following Formulas 35, 36, 37, 38, 39, 40, 41, and 42: a_l0′=a_l0+offset_a a_l1′=a_l1-offset_a b_l0′=b_l0+offset_b b_l1′=b_l1−offset_b c_l0′=c_l0+offset_c c_l1′=c_l1-offset_c d_l0′=d_l0+offset_d d_l1′=d_l1-offset_d Where a_l0, b_l0, c_l0 and d_l0 are the parameters of the affine model of reference list 0, and a_l1, b_l1, c_l1 and d_l1 are the parameters of the affine model of reference list 1. Each of the four parameters is refined.

[0152] After the affine parameters are refined, the sub-block MVs of list 0 and list 1 are derived using a_l0', b_l0', c_l0', d_l0' and a_l1', b_l1', c_l1', d_l1' according to the affine model of formulas 22 to 24, and then the sub-block MVs can be derived and motion compensation can be performed at the sub-block level.

[0153] In addition, the bilateral matching cost (whether based on SAD or SATD or MRSAD or MRSATD) can be calculated at the sub-block level or the CB level. If the bilateral matching cost is calculated at the sub-block level, the cost of each sub-block is calculated after motion compensation of the sub-block; after obtaining the costs of multiple sub-blocks, the CB level cost is calculated by summing the costs of all sub-blocks.

[0154] If bilateral matching is costed at the CB level, motion compensation is first performed for each sub-block to obtain L1 and L0 prediction values ​​for the entire CB, and then the overall cost is calculated. In order to reduce computational complexity, in some embodiments, not all sub-blocks of the CB or not all samples of the CB are considered in the bilateral matching cost calculation. Only the differences of some sub-blocks or some samples are calculated, so motion compensation of sub-blocks or samples not considered in the cost calculation can also be skipped.

[0155] Alternatively, the VVC standard encoder and decoder configure one or more processors of the computing system to perform 2-parameter refinement on the affine model. For the 4-parameter affine model, since b is equal to -c and d is equal to a, the refinement can also follow the constraint: offset_b is equal to -offset_c, and offset_d is equal to offset_a. Therefore, the encoder and decoder only need to perform an affine parameter offset search for offset_a and offset_b, and derive offset_b and offset_d from offset_a and offset_b.

[0156] Since the computational complexity of the affine parameter offset search for 2-parameter offset is lower than that for 4-parameter offset, the constraints of offset_b equal to -offset_c and offset_d equal to offset_a can also be applied to the DMVR for the 6-parameter affine model. The refinement of the respective multiple parameters is performed according to the following formula 43, formula 44, formula 45, formula 46, formula 47, formula 48, formula 49 and formula 50: a_l0′=a_l0+offset_a a_l1′=a_l1-offset_a b_l0′=b_l0-offset_c b_l1′=b_l1+offset_c c_l0′=c_l0+offset_c c_l1′=c_l1-offset_c d_l0′=d_l0+offset_a d_l1′=d_l1-offset_a

[0157] In addition, the 4-parameter refinement can also be applied to the 4-parameter model or the 6-parameter model. For the 4-parameter and 6-parameter affine models, the same refinement as the above Formula 35, Formula 36, ​​Formula 37, Formula 38, Formula 39, Formula 40, Formula 41, and Formula 42 is applied.

[0158] In some examples, the VVC standard encoder and decoder configure one or more processors of the computing system to apply a refined MV search scheme in the affine parameter offset search. Figure 3A As shown in Figure 3B, a 3×3 square search or a 3×3 cross search can be applied to produce the best parameter shift. For 4-parameter refinement, a positive search or a cross search is performed in a 4-dimensional space. A 3×3×3×3 square search or a 3×3×3×3 cross search can be applied to obtain the optimal parameter shift. For the 3×3×3×3 square search, for each center position, there are 80 neighboring positions to search; for the 3×3×3×3 cross search, for each center position, there are 8 neighboring positions to search, which is much less than the 3×3×3×3 square search.

[0159] Assuming that the parameter offset of the current center position is (offset_a, offset_b, offset_c, offset_d), the eight adjacent positions to be searched in the 3×3×3×3 cross search are (offset_a+step_a, offset_b, offset_c, offset_d), (offset_a-step_a, offset_b, offset_c, offset_d), (offset_a, offset_b+step_b, offset_c, offset_d), (offset_a, offset_b, -step_b offset_c, offset_d), (offset_a, offset_b, offset_c+step_c, offset_d), (offset_a, offset_b, offset_c-step_c, offset_d), (offset_a, offsets et_b, offset_c, offset_d+step_d), (offset_a, offset_b, offset_c, offset_d-step_d), where step_a, step_b, step_c, step_d are the search steps of parameters a, b, c and d respectively.

[0160] After the optimal parameter offset is returned from the affine parameter offset search, the error surface based offset estimation can also be applied to further refine the returned parameters with higher accuracy. The search step size for the integer search can be a fixed value. According to some instances, since the MV accuracy in ECM is 1 / 16 and the basic sub-block used for affine motion compensation is 4×4, the search step size can be 1 / 64 so that the MV difference between two adjacent sub-blocks is 1 / 64*4=1 / 16, which is the minimum difference in MV. In some other examples, the search step size may be greater than 1 / 64. A larger search step size reduces the search iterations to reduce the search time, but may sacrifice the refinement accuracy. In another example, the search step size depends on the CB size. Denote the width of the CB as w, the height of the CB as h, and the search step size of a and c as step step ac , the search step length of d and b is expressed as step step bd , the search step size can satisfy the following formula 51: w×step ac =T1 h×step bd =T2 Wherein, T1 and T2 are two thresholds, which can be 1 / 16, 1 / 8, 1 / 4 or other values. The threshold defines the maximum MV difference between the MVs of any two samples in the block. According to this example, different parameters have different search steps.

[0161] For the cost of each search point, the difference in parameter offsets can also be considered. The cost can be a weighted sum of the SAD or SATD (or MRSAD or MRSATD) between two predicted values ​​of the coding block and the parameter offset, such as bilCost = w*ParameterOffsetCost + sadCost, where w is the weight, sadCost is the SAD / SATD or de-mean SAD / SATD cost of the predicted value, and ParameterOffsetCost is the cost of the parameter offset that depends on multiple refinement parameters. When w is equal to 0, only sadCost is considered.

[0162] Furthermore, according to an exemplary embodiment of the present disclosure, during the affine parameter shift search, the MV of the affine model may be fixed.

[0163] During any affine parameter shift search, one MV of the affine model is fixed. Unlike the refinement MV search, multiple affine parameter shift searches can be performed, and a different MV of the affine model can be fixed during different affine parameter shift searches. Any MV at any point on the plane can be considered an affine model MV and thus fixed during the parameter search. In one implementation, the CPMV is considered to be the affine model MV fixed during the search. For example, the top left CPMV is fixed and the affine model parameters are refined, such as Fig.15 As shown in (a). As the parameters change, the coding block rotates and enlarges / reduces, so the upper right CPMV and the lower right CPMV also change. Then, the sub-block MV is derived from the multiple refined CPMVs (i.e., derived from the multiple refined parameters and the new CPMV), and motion compensation is performed. Fig.15 (b) and Fig.15 (c) shows an example in which the upper right CPMV and the lower left CPMV are respectively regarded as affine model MVs, while the four affine model parameters are fixed while being refined. Fig.15 (a) Similarly, with refinement of affine model parameters, the coding block is rotated and enlarged / reduced, so multiple non-fixed CPMVs are changed. Multiple affine parameter offset searches can be performed because multiple non-fixed CPMVs can produce different multiple optimal parameter offsets (and multiple different MV refinement offsets).

[0164] In another implementation, various CPMVs (each CPMV is considered an affine model MV) are fixed in sequence for multiple affine parameter shift searches performed in sequence. In the first affine parameter shift search, the top left CPMV is fixed, and Fig.15 (a) shows the search parameter shift. As multiple parameters are refined, the CPMV in the upper right corner is also changed and can be calculated accordingly. Then in the second affine parameter shift search, Fig.15(b), the refined upper right CPMV is fixed, and multiple parameters are refined again. By using multiple refined parameters obtained in the second sub-step, the lower left CPMV is also changed and can be calculated accordingly. Then, in the third affine parameter shift search, the refined lower left CPMV is fixed, and multiple parameters are refined again, such as Fig.15 (c) is shown. The same affine parameter offset search can be repeated several times: the third affine parameter offset search can be followed by the first affine parameter offset search, in which the new upper left CPMV is fixed, and the search can continue until certain conditions are met. For example, the conditions may include but are not limited to: 1) a predetermined number of iterations; 2) the SAD or SATD (or MRSAD or MRSATD) between the L0 prediction value and the L1 prediction value is less than a threshold; 3) the current fixed CPMV is the same as or similar to the CPMV in the previous iteration; 4) multiple optimal parameter offsets generated by the latest affine parameter offset search are less than a threshold.

[0165] In yet another embodiment, instead of CPMV, finding MV 0 in a plane is considered to fix the affine model MV during refinement. First, the (x, y) point where MV is 0 is derived according to Equation 52 below.

[0166] Assume that the solution is (x1, y1), so the affine model can be expressed according to the following formula 53, and the MV is 0:

[0167] Then, the affine model parameters a, b, c and d are searched to find the refined values. The above-mentioned refinement methods can all be applied to this embodiment.

[0168] As mentioned before, the affine parameter refinement process is similar to the MV refinement process. The search is performed through multiple iterations, which together constitute an affine parameter shift search. At each iteration, if the bilateral matching cost of the center position is less than all neighboring positions, the current center position is found as the optimal position and the search terminates; otherwise, the neighboring position with the minimum bilateral matching cost is set as the new center position, and the search advances to the next iteration.

[0169] Therefore, in order to control the search complexity, the VVC standard encoder and decoder configure one or more processors of the computing system to perform the maximum number of search iterations in each affine parameter offset search, and the number of search iterations is configured on both the encoder and decoder sides. The search terminates when the center position has the minimum cost or the maximum search iteration threshold reaches the preset maximum iteration threshold. A larger maximum search iteration threshold can produce more coding performance gains, but consumes longer encoding and decoding time.

[0170] To achieve a good compromise between computational complexity and performance, the maximum search iteration threshold can be set according to fixed MV, QP, temporal layer, CB size, etc. For example, according to the search process shown in Fig.15 (a) to (c), in the first affine parameter offset search, the upper left CPMV is fixed, and the maximum search iteration threshold is set to a larger value, such as 8, relative to the subsequent thresholds; since multiple parameters are refined for the first time, the larger search iteration threshold can configure one or more processors of the computing system to utilize a longer computing time to improve encoding and decoding performance.

[0171] Next, in the second affine parameter offset search, the upper right CPMV is fixed, and the maximum search iteration threshold is set to a smaller value, such as 6, relative to the previous threshold; since the parameters have been refined in the first affine parameter offset search, the smaller search iteration threshold can save encoding and decoding time.

[0172] Next, in the third affine parameter offset search, the lower left CPMV is fixed, and the maximum search iteration threshold is set to a smaller value, such as 2, relative to the previous threshold to further save encoding and decoding time. Therefore, in this embodiment, the maximum search iteration threshold is set to a larger value at the beginning and changed to a smaller value in subsequent affine parameter offset searches.

[0173] In some other embodiments, the maximum search iteration threshold for subsequent affine parameter offset searches depends on the actual number of search iterations performed during the previous affine parameter offset search. For example, in the first affine parameter offset search, when the upper left CPMV is fixed, the maximum search iteration threshold is set to N. However, during the first affine parameter offset search, the search actually terminates at the kth (k < N) search iteration because the center position has the minimum bilateral matching cost. Next, in the second affine parameter offset search, the maximum search iteration threshold is set to k / 2 (or other value depending on k and less than P).

[0174] Conversely, if the maximum number of search iterations allowed by the threshold is performed in the first affine parameter offset search, then in the second affine parameter offset search, the maximum search iteration threshold is set to P, where P is a value less than N. The third affine parameter offset search can adopt a similar method: if the actual number of search iterations performed in the second affine parameter offset search is the maximum number, the maximum search iteration threshold for the third affine parameter offset search is set to L, where L is less than P; if the actual number of search iterations is t and t does not reach the maximum number, the maximum search iteration threshold for the third affine parameter offset search is set to t / 2. Therefore, the maximum search iteration threshold is adaptively determined in the previous search iteration.

[0175] In some other embodiments, in order to reduce complexity, the search neighboring positions of the search iteration are adaptively reduced according to the previous search iteration. For example, in a 3×3×3×3 cross search scheme, eight neighboring positions are searched in each search iteration. Assume that the current center is (a, b, c, d), and the eight neighboring positions to be checked are pa0=(a+s, b, c, d), pa1=(as, b, c, d), pb0=(a, b+s, c, d), pb1=(a, bs, c, d), pc0=(a, b, c+s, d), pc1=(a, b, cs, d), pd0=(a, b, c, d+s) and pd1=(a, b, c, ds). The bilateral matching costs of the eight neighboring positions are denoted as cost_pa0, cost_pa1, cost_pb0, cost_pb1, cost_pc0, cost_pc1, cost_pd0, and cost_pd1.

[0176] The VVC standard encoder and decoder configure one or more processors of the computing system to compare cost_pa0 and cost_pa1: if cost_pa0 is less than cost_pa1, parameter a only considers positive offsets in the next iteration; if cost_pa0 is greater than cost_pa1, parameter a only considers negative offsets in the next iteration.

[0177] The encoder and decoder of the VVC standard configure one or more processors of the computing system to compare costs cost_pb0 and cost_pb1: if cost_pb0 is less than cost_pb1, then in the next iteration, parameter b only considers positive offsets; if cost_pb0 is greater than cost_pb1, then in the next iteration, parameter b only considers negative offsets.

[0178] The encoder and decoder of the VVC standard configure one or more processors of the computing system to compare cost_pc0 and cost_pc1: if cost_pc0 is less than cost_pc1, then in the next iteration, parameter c only considers positive offsets; if cost_pc0 is greater than cost_pc1, then in the next iteration, parameter c only considers negative offsets.

[0179] The VVC standard encoder and decoder configure one or more processors of the computing system to compare cost_pd0 and cost_pd1: if cost_pd0 is less than cost_pd1, then in the next iteration, parameter d only considers positive offsets; if cost_pd0 is greater than cost_pd1, then in the next iteration, parameter d only considers negative offsets.

[0180] Assume that for the current search iteration, cost_pa0 is less than cost_pa1, cost_pb0 is greater than cost_pb1, cost_pc0 is less than cost_pc1, and cost_pd0 is greater than cost_pd1. Then the four neighboring positions to be checked in the next search iteration are (a'+s, b', c', d'), (a', b'-s, c', d'), (a', b', c'+s, d'), and (a', b', c', d'-s), where (a', b', c', d') is the central position of the next search iteration.

[0181] In some other embodiments, the minimum bilateral matching cost of the current search iteration is compared with the minimum bilateral matching cost of the previous search iteration, or with the product of the minimum bilateral matching cost of the previous search iteration and a factor f. If the reduction in the minimum cost is smaller, the search terminates. For example, if the cost of the previous search iteration is A, which means the cost at the current search center is A, and the minimum cost among multiple neighboring positions is B at position posb, where B < A. According to this search rule, the search enters the next iteration with search center posb. However, in this embodiment, if A - B < K or B > A * f, the search terminates, and posb is selected as the optimal position for this search iteration. K and f are preset thresholds. For example, f is a factor less than 1, such as 0.95, 0.9, or 0.8.

[0182] QP controls quantization in video coding. The higher the QP, the larger the quantization step size is used, and thus more distortion is introduced. Therefore, for a higher QP, more search iterations are required in the refinement, increasing the encoding time. To reduce the total encoding time, in this embodiment, a smaller maximum search iteration threshold is set for a higher QP than for a lower QP.

[0183] Other methods for reducing complexity can also be used in high QP. For example, the neighboring positions to be searched can be reduced; the search iterations can be adaptively reduced; or the search can be terminated prematurely depending on the previous search process. Therefore, in this embodiment, different search strategies can be adopted for different QPs.

[0184] An inter-coded frame, such as a B frame or a P frame, has one or more reference frames. The temporal distance between the current frame and the reference frame affects the accuracy of inter-frame prediction. The temporal distance between two frames in video coding is usually represented by the POC distance. Generally, the longer the POC distance, the lower the inter-frame prediction accuracy and the lower the motion information accuracy, and thus more refinement is required. Therefore, in this embodiment, the search process depends on the POC distance between the current frame and the reference frame.

[0185] For hierarchical B frames, frames with higher temporal layers have shorter POC distances to reference frames, and frames with lower temporal layers have longer POC distances to reference frames. Therefore, the search process may also depend on the temporal layer of the current frame. For example, affine parameter refinement may be disabled for high temporal layers because the POC distances of high temporal layers to reference frames are short and refinement may not be required. In another example, for high temporal layer frames, a small search iteration threshold is set, or the adjacent search positions are reduced.

[0186] In addition, for high temporal layer frames, other methods that reduce the complexity of parameter refinement can be used.Therefore, in this embodiment, the parameter refinement process depends on the time or POC distance between the current frame and the reference frame.

[0187] In all the above embodiments, multiple affine model parameters are directly refined. However, affine motion may include translation, rotation and scaling. Translation is represented by the MV of the affine model, and rotation and scaling are represented by affine model parameters. In another embodiment, the motion of rotation and scaling is explicitly refined. Based on the original affine model, additional rotation and scaling are added. If the original affine model is described as formula 22 above. Then, a rotation with an angle t and a scaling of a factor k is applied according to the following formula 54:

[0188] Where t and k are two parameters to be searched during the DMVR process. The current search method can be applied to obtain the optimal values ​​of t and k. Then, the sub-block MV is derived according to the above formula 49, and the sub-block based affine motion compensation is performed to obtain the prediction value of the current affine coded block.

[0189] All existing early termination methods in MV refinement can also be applied to parameter refinement. For example, during a refined MV search, if the SAD or SATD (or MRSAD or MRSATD) between two predicted values ​​is less than a threshold, the refined MV search is terminated.

[0190] Similar to MV refinement, PROF may or may not be applied before SAD calculation during the search process of multiple affine model parameters. If PROF is applied, for each search point, after obtaining the prediction value, PROF is applied to refine the prediction value, and SAD is calculated between the two PROF-refined prediction values. If PROF is not applied, for each search point, SAD is calculated directly after obtaining the L0 prediction value and the L1 prediction value.

[0191] And, to further reduce the search complexity, for each search point, the prediction value can generate an interpolation filter with a reduced number of taps, rather than an 8-tap or 12-tap interpolation filter.

[0192] Affine parameter refinement and MV refinement can be performed in combination. In one embodiment, MV refinement and affine parameter refinement are performed sequentially. Affine parameter refinement can be performed after MV refinement, or affine parameter refinement can be performed after MV refinement.

[0193] In an alternative embodiment, MV refinement and affine parameter refinement are performed in the same process. For an affine model according to the following formula 55: 6 parameters (a, b, c, d and mv 0x 、mv 0y ) are refined together. offset_a, offset_b, offset_c, offset_d, offset_mv 0x and offset_mv 0y The optimal value of is found in the search, and these parameters are refined according to the following formulas 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66 and 67 respectively: a_l0′=a_l0+offset_a al1′=a_l1-offset_a b_l0′=bl0+offset_b b_l1′=b_l1−offset_b c_l0′=c_l0+offset_c c_l1′=c_l1-offset_c d_l0′=d_l0+offset_d d_l1′=d_l1-offset_d mv 0x _l0′=mv 0x _l0+offset_mv 0x mv 0x _l1′=mv 0x _l1-offset_mv 0x mu 0y _l0′=mv 0y _l0+offset_mv 0y mv 0y _l0′=mv 0y _l0-offset_mv 0y where a_l0, b_l0, c_l0 and d_l0 are the initial parameters of the affine model for reference image list 0, a_l1, b_l1, c_l1 and d_l1 are the initial parameters of the affine model for reference image list 1, (mv 0x _l0, mv 0y _l0) is the initial MV of the affine model of reference image list 0, and (mv 0x _l1, mv 0y _l1) is the initial MV of the affine model of reference image list 1. a_l0', b_l0', c_l0' and d_l0' are the refinement parameters of the affine model of reference list 0, a_l1, b_l1, c_l1 and d_l1 are the refinement parameters of the affine model of reference list 1, (mv 0x _l0'、mv 0y _l0') is the refined MV of the affine model of reference image list 0, and (mv 0x _l1'、mv 0y _l1') is the refined MV of the affine model of reference picture list 1. After refining these 6 parameters, the CPMV is recalculated and then the sub-block MV can be derived. Sub-block level motion compensation can be performed. This embodiment can also be combined with a 4-parameter refinement restriction: offset_b is equal to -offset_c and offset_d is equal to offset_a.

[0194] Those skilled in the art will understand that all of the above aspects of the present disclosure may be implemented simultaneously in any combination thereof, and all aspects of the present disclosure may be implemented in combination as yet another embodiment of the present disclosure.

[0195] Fig.16 An example system 1600 is shown for implementing the processes and methods described herein for implementing motion vector refinement for decoder-side affine motion compensation.

[0196] The techniques and mechanisms described herein can be implemented by multiple instances of the system 1600, as well as by any other computing devices, systems, and / or environments. Fig.16 The system 1600 shown is only one example of a system and is not intended to impose any limitation on the scope of use or functionality of any computing device used to perform the above-described processes and / or processes. Other well-known computing devices, systems, environments, and / or configurations that may be suitable for use with embodiments include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, game consoles, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, implementations using field programmable gate arrays ("FPGAs") and application specific integrated circuits ("ASICs"), and the like.

[0197] System 1600 may include one or more processors 1602 and a system memory 1604 communicatively coupled to processor 1602. Processor 1602 may execute one or more modules and / or processes to enable processor 1602 to perform various functions. In some embodiments, processor 1602 may include a central processing unit ("CPU"), a graphics processing unit ("GPU"), both a CPU and a GPU, or other processing units or components known in the art. In addition, each processor 1602 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0198] Depending on the exact configuration and type of system 1600, system memory 1604 may be volatile, such as RAM, non-volatile, such as ROM, flash memory, micro hard drive, memory card, etc., or some combination thereof. System memory 1604 may include one or more computer executable modules 1606 that may be executed by processor 1602.

[0199] The module 1606 may include, but is not limited to, an encoder module 1608 and a decoder module 1610. The encoder module 1608 and the decoder module 1610 may be configured to perform any of the methods described above.

[0200] The system 1600 may additionally include an input / output (I / O) interface 1640 for receiving video source data and bitstream data, and for outputting decoded images to a reference image buffer and / or a display buffer. The system 1600 may also include a communication module 1650 that allows the system 1600 to communicate with other devices (not shown) over a network (not shown). The network may include the Internet, wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, radio frequency ("RF"), infrared, and other wireless media.

[0201] In some embodiments, a method is provided, comprising: performing a refined MV search on a CPMV of an inter-coded CB; and outputting a refined MV of the CB. The method may be performed by a decoder or an encoder.

[0202] Some or all of the operations of the above methods may be performed by executing computer-readable instructions stored on a computer-readable storage medium, as defined below. The term "computer-readable instructions" as used in the specification and claims includes routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on a variety of system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, embedded microprocessors, programmable consumer electronics, combinations thereof, etc.

[0203] Computer-readable storage media may include volatile memory (e.g., random access memory ("RAM")) and / or non-volatile memory (e.g., read-only memory ("ROM"), flash memory, etc.). Computer-readable storage media may also include additional removable and / or non-removable storage, including but not limited to flash memory, magnetic storage, optical storage, and / or tape storage that may provide non-volatile storage for computer-readable storage instructions, data structures, program modules, and the like.

[0204] Non-transitory computer-readable storage media are examples of computer-readable media. Computer-readable media include at least two types of computer-readable media, namely computer-readable storage media and communication media. Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented in any process or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media include, but are not limited to, phase change memory ("PRAM"), static random access memory ("SRAM"), dynamic random access memory ("DRAM"), other types of random access memory ("RAM"), read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), flash memory or other memory technology, compact disk read-only memory ("CD-ROM"), digital versatile disk ("DVD") or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. In contrast, communication media can embody computer-readable instructions, data structures, program modules or other data in a modulated data signal, such as a carrier wave or other transmission mechanism. The computer-readable storage medium employed herein should not be construed as a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (such as a light pulse through a fiber optic cable), or an electrical signal propagating through a wire.

[0205] Computer readable instructions stored on one or more non-transitory computer readable storage media, when executed by one or more processors, may be used as described above with reference to Figure 1A-15Typically, computer-readable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement a process.

[0206] In some embodiments, a computer program product is provided, wherein the program product comprises: computer program instructions, wherein the computer program instructions enable a computer to execute the steps of the method as described in any embodiment of the present disclosure.

[0207] In some embodiments, a computer program is provided, which enables a computer to execute the steps of the method described in any embodiment of the present disclosure.

[0208] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. A computer system comprising: one or more processors, and A computer-readable storage medium is communicatively coupled to the one or more processors, the computer-readable storage medium storing computer-readable instructions executable by the one or more processors, and when the computer-readable instructions are executed by the one or more processors, the related operations performed include: performing a refined motion vector (MV) search on a control point motion vector (CPMV) of an inter-coded coding block (CB); and Output the refined MV of the CB.

2. The computer system of claim 1, wherein the operations further comprise: Outputting the refined MVs of the multiple CPMVs of the CB; Each refined MV is derived from the same MV refinement offset applied to its respective initial CPMV.

3. The computer system of claim 1 , wherein performing a refined MV search comprises: deriving an MV of a sub-block of the CB based on the CPMV of the CB; performing sub-block MV refinement on the MV of the sub-block; Outputting a refined MV of the sub-block; as well as The refined MV of the CB is output based on the refined MV of the sub-block.

4. The computer system of claim 2, wherein performing a refined MV search further comprises: Multiple search iterations of integer sample offset search are performed at multiple search points around the search center to output integer distance refined MVs.

5. The computer system of claim 3, wherein the operations further comprise: The predicted sample value of the CB is obtained based on the MV of the sub-block by acquiring pre-interpolated samples with respect to each search point around the search center without performing interpolation.

6. The computer system of claim 1 , wherein performing the refined MV search comprises: deriving affine model parameters based on a plurality of said CPMVs of said CB; performing an affine parameter offset search on the affine model parameters; Output an optimal parameter offset; as well as The refined MV of the CB is output based on the optimal parameter shift.

7. The computer system of claim 6, wherein determining the affine model parameters based on the plurality of CPMVs comprises one of the following: Determine 4 affine model parameters based on 3 CPMVs; Determine two affine model parameters based on the two CPMVs; or The four affine model parameters are determined based on the three CPMVs.

8. The computer system of claim 6 or 7, wherein an affine parameter shift search is performed while fixing a CPMV of the CB; and The refined MV of the CB is further outputted based on the fixed CPMV.

9. The computer system according to any one of claims 6 to 8, wherein performing the refined MV search comprises sequentially performing a plurality of refined MV searches, each refined MV search comprising: The affine parameter shift search is performed on the affine model parameters while fixing the different CMPVs of the CB.

10. The computer system of claim 1, wherein the operations further comprise: Outputting the MVs refined by the multiple CMPVs of the CB; in, Each refined MV is derived from a different MV refinement offset applied to the respective initial CPMV.

11. The computer system of claim 1 , wherein each refinement MV search further comprises: A plurality of search iterations are performed at a plurality of search points around the search center, up to a maximum search iteration threshold, wherein for each subsequent refined MV search, the respective maximum search iteration threshold is smaller.

12. The computer system of claim 1, wherein performing a refined MV search further comprises: performing multiple search iterations of integer sample offset search at multiple search points around the search center to output integer distance refined MVs; as well as A fractional sample refinement is applied to the integer distance refinement MV to output a refinement delta MV of sub-pixel accuracy.

13. The computer system of claim 1, wherein each refined MV search further comprises performing a plurality of search iterations at a plurality of search points around a search center.

14. The computer system of claim 13, wherein performing an iteration of the plurality of search iterations comprises one of searching a plurality of search points by a square search about the search center, and searching a plurality of search points by a cross search about the search center.

15. The computer system of claim 13, wherein performing an iteration of the plurality of search iterations comprises: calculating a bilateral matching cost for each of the plurality of search points based on two derived prediction values ​​of the CB from a reference image in the first reference image list and a reference image in the second reference image list, respectively; determining a minimum of a plurality of bilateral matching costs for each of the plurality of search points; as well as The refined MV search is terminated based on determining that the minimum bilateral match cost is greater than a product of a minimum bilateral match cost of a previous iteration in the plurality of search iterations and a factor.

16. The computer system of claim 13, wherein performing an iteration of the plurality of search iterations comprises: calculating a bilateral matching cost for each of the plurality of search points based on two derived prediction values ​​of the CB from a reference image in the first reference image list and a reference image in the second reference image list, respectively; determining a minimum bilateral matching cost among a plurality of bilateral matching costs for each search point in the plurality of search points; as well as The refined MV search is terminated based on a difference between the minimum bilateral match cost and a minimum bilateral match cost of a previous iteration in the plurality of search iterations.

17. The computer system of claim 13, wherein performing an iteration of the plurality of search iterations comprises: calculating a bilateral matching cost for each of the plurality of search points based on two derived prediction values ​​of the CB from a reference image in the first reference image list and a reference image in the second reference image list, respectively; as well as Determine the minimum bilateral matching cost of each calculated bilateral matching cost of the search point.

18. The computer system of claim 17, wherein calculating the bilateral matching cost for each of the plurality of search points is based on a distance between the respective search point and the search center.

19. The computer system of claim 17, wherein the calculation of the bilateral matching cost is based on the sum of absolute differences, the sum of absolute transformed differences, the demeaned sum of absolute differences, or the demeaned sum of absolute transformed differences between the two derived prediction values.

20. The computer system of claim 17, wherein performing an iteration of the plurality of search iterations further comprises: No optical flow prediction refinement (PROF) is applied to the two derived prediction values ​​before computing the bilateral matching cost.

21. A method comprising: Perform refined MV search on the CPMV of inter-coded CB; as well as Output the refined MV of the CB.

22. The method according to claim 21, wherein the method further comprises: Outputting the refined MVs of the multiple CPMVs of the CB; Each refined MV is derived from the same MV refinement offset applied to its respective initial CPMV.

23. The method of claim 21, wherein performing a refined MV search comprises: deriving an MV of a sub-block of the CB based on the CPMV of the CB; performing sub-block MV refinement on the MV of the sub-block; Outputting a refined MV of the sub-block; as well as The refined MV of the CB is output based on the refined MV of the sub-block.

24. The method of claim 22, wherein performing a refined MV search further comprises: Multiple search iterations of integer sample offset search are performed at multiple search points around the search center to output integer distance refined MVs.

25. The method of claim 23, wherein the method further comprises: The predicted sample value of the CB is obtained based on the MV of the sub-block by acquiring pre-interpolated samples with respect to each search point around the search center without performing interpolation.

26. The method of claim 21, wherein performing a refined MV search comprises: deriving affine model parameters based on a plurality of said CPMVs of said CB; performing an affine parameter offset search on the affine model parameters; Output an optimal parameter offset; as well as The refined MV of the CB is output based on the optimal parameter shift.

27. The method of claim 26, wherein determining the affine model parameters based on a plurality of CPMVs comprises one of the following: Determine 4 affine model parameters based on 3 CPMVs; Determine two affine model parameters based on the two CPMVs; or The four affine model parameters are determined based on the three CPMVs.

28. The method of claim 26 or 27, wherein an affine parameter shift search is performed while fixing a CPMV of the CB; and The refined MV of the CB is further outputted based on the fixed CPMV.

29. The method according to any one of claims 26 to 28, wherein performing the refined MV search comprises sequentially performing a plurality of refined MV searches, each refined MV search comprising: The affine parameter shift search is performed on the affine model parameters while fixing the different CMPVs of the CB.

30. The method of claim 21, wherein the method further comprises: Outputting the MVs refined by the multiple CMPVs of the CB; Each refined MV is derived from a different MV refinement offset applied to the respective initial CPMV.

31. The method of claim 21, wherein each refined MV search further comprises: A plurality of search iterations are performed at a plurality of search points around the search center, up to a maximum search iteration threshold, wherein the maximum search iteration threshold is smaller for each subsequent refined MV search.

32. The method of claim 21, wherein performing a refined MV search further comprises: performing multiple search iterations of integer sample offset search at multiple search points around the search center to output integer distance refined MVs; as well as A fractional sample refinement is applied to the integer distance refinement MV to output a refinement delta MV of sub-pixel accuracy.

33. The method of claim 21, wherein each refined MV search further comprises performing multiple search iterations at multiple search points around a search center.

34. The method of claim 33, wherein performing an iteration of the plurality of search iterations comprises one of searching a plurality of search points by a square search around the search center, and searching a plurality of search points by a cross search around the search center.

35. The method of claim 33, wherein performing an iteration of the plurality of search iterations comprises: calculating a bilateral matching cost for each of the plurality of search points based on two derived prediction values ​​of the CB from a reference image in the first reference image list and a reference image in the second reference image list, respectively; determining a minimum of the bilateral matching costs of each of the plurality of search points; as well as The refined MV search is terminated based on determining that the minimum bilateral match cost is greater than a product of a minimum bilateral match cost of a previous iteration in the plurality of search iterations and a factor.

36. The method of claim 33, wherein performing an iteration of the plurality of search iterations comprises: calculating a bilateral matching cost for each of the plurality of search points based on two derived prediction values ​​of the CB from a reference image in the first reference image list and a reference image in the second reference image list, respectively; determining a minimum bilateral matching cost among a plurality of bilateral matching costs for each of the plurality of search points; as well as The refined MV search is terminated based on a difference between the minimum bilateral match cost and a minimum bilateral match cost of a previous iteration in the plurality of search iterations.

37. The method of claim 33, wherein performing an iteration of the plurality of search iterations comprises: calculating a bilateral matching cost for each of the plurality of search points based on the two derived prediction values ​​of the CB from a reference image in the first reference image list and a reference image in the second reference image list, respectively; as well as Determine the minimum bilateral matching cost of each calculated bilateral matching cost of the search point.

38. The method of claim 37, wherein calculating a bilateral matching cost for each of the plurality of search points is based on a distance between the respective search point and the search center.

39. The method of claim 37, wherein the bilateral matching cost is calculated based on the sum of absolute differences, the sum of absolute transformed differences, the demeaned sum of absolute differences, or the demeaned sum of absolute transformed differences between the two derived prediction values.

40. The method of claim 37, wherein performing an iteration of the plurality of search iterations further comprises: No optical flow prediction refinement (PROF) is applied to the two derived prediction values ​​before computing the bilateral matching cost.

41. A non-transitory computer-readable storage medium storing a set of instructions executable by one or more processors of a device to cause the device to initiate the method of any one of claims 21 to 40.

42. A computer program product comprising: Computer program instructions causing a computer to perform the method according to any one of claims 21 to 40.

43. A computer program causing a computer to perform a method according to any one of claims 21 to 40.

Images