Deriving a coding unit level weight bi-prediction index for merge candidates

Abstract

An encoder and decoder for the VVC standard are provided, enabling the derivation of the BCW index based on template matching cost values. The template matching cost can be calculated for each value in a possible set of BCW weight values, and the BCW weight value that generates the lowest template matching cost among all calculated template matching costs can be selected as the BCW index for bidirectional prediction merging candidates. Optionally, the template matching cost can be calculated for each possible subset of BCW weights based on inherited BCW weights. Furthermore, merged candidate BCW indices can be derived by adjusting the template matching cost in inherited BCW weights from values ​​calculated according to the VVC standard and ECM specification, or by adjusting the template matching cost of BCW weights with the same weights from values ​​calculated according to the VVC standard and ECM specification.

Description

[0001] Cross-references to related applications

[0002] This disclosure claims priority to the following applications: U.S. Provisional Application No. 63 / 358,215, filed July 4, 2022, entitled "Deriving a Bidirectional Prediction Index of Coding Unit-Level Weights for Merging Candidates"; U.S. Provisional Application No. 63 / 403,199, filed September 1, 2022, entitled "Deriving a Bidirectional Prediction Index of Coding Unit-Level Weights for Merging Candidates"; and U.S. Provisional Application No. 18 / 215,753, filed June 28, 2023, entitled "Deriving a Bidirectional Prediction Index of Coding Unit-Level Weights for Merging Candidates," the entire contents of which are incorporated herein by reference. Technical Field

[0003] Embodiments of this disclosure relate to video processing, and more specifically, to a method and system for deriving a bidirectional prediction index of coding unit-level weights for merging candidates. Background Technology

[0004] In 2020, the Joint Video Experts Group (JVET) of the ITU-T Video Coding Experts Group (“ITU-T VCEG”) and the ISO / IEC Moving Picture Experts Group (“ISO / IEC MPEG”) released the final draft of the next-generation video codec specification, “Universal Video Coding” (“VVC”). This specification further improves video coding performance compared to existing standards such as H.264 / AVC (Advanced Video Coding) and H.265 / HEVC (High-Efficiency Video Coding). Furthermore, at the time of writing, a new draft of the Enhanced Compression Model (“ECM”) extends the VVC standard; this draft was presented as an “Algorithmic Description of Enhanced Compression Model 8 (ECM 8)” at the 29th JVET meeting in January 2023.

[0005] Inter-frame prediction can be either unidirectional or bidirectional. In unidirectional prediction, only one motion vector pointing to a reference image is used to generate the prediction value for the current block; in bidirectional prediction, two motion vectors pointing to their respective reference images are used to generate the prediction value for the current block. According to the earlier HEVC standard, bidirectional prediction signals are generated by averaging two prediction signals obtained from two different reference images and / or using two different motion vectors. According to the VVC standard, the bidirectional prediction mode extends simple averaging by allowing a weighted average of the two prediction signals; for simplicity, this technique is referred to as bidirectional prediction with coding unit (“CU”) level weights (“BCW”).

[0006] Based on decoder-side motion vector refinement (“DMVR”), bidirectional prediction can be performed on the current CU, such that the motion information of the current CU includes a weighted average of the two prediction signals, with the weight index inferred from neighboring blocks based on the merge candidate index. For merge candidates that satisfy the DMVR conditions, the adaptive decoder-side motion vector refinement technique further extends multi-channel DMVR by refining the motion vector in only one of the two bidirectional prediction directions.

[0007] According to current VVC and ECM specifications, the BCW index of a merged coded block is inherited from neighboring blocks based on the merge candidate index notified by a signal. However, the inherited BCW index may not be applicable to the merged coded block. Summary of the Invention

[0008] Embodiments of this disclosure relate to methods and systems for deriving bidirectional prediction indexes with coding unit-level weights for merging candidates.

[0009] In a first aspect, embodiments of this disclosure provide a computer system including: one or more processors, and a computer-readable storage medium communicatively coupled to the one or more processors, the computer-readable storage medium storing computer-readable instructions executable by the one or more processors, the computer-readable instructions performing associated operations when executed by the one or more processors, the associated operations including:

[0010] For a merge candidate in the merge candidate list of a merge coding unit (CU), the CU-level weighted bidirectional prediction (BCW) index of the bidirectional prediction merge candidate is derived based on the template matching cost.

[0011] In a second aspect, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing an instruction set executable by one or more processors of an apparatus to cause the apparatus to perform a method comprising: bidirectional prediction of merge candidates for a merge candidate list of merged coding units (CUs) of a merged code, and deriving a CU-level weighted bidirectional prediction (BCW) index of the bidirectional prediction of merge candidates based on template matching cost.

[0012] In a third aspect, embodiments of this disclosure provide a computer system including: computer program instructions, and computer program instructions that enable the computer to perform a method comprising: bidirectional prediction of merge candidates for a merge candidate list of merged coding units (CUs) of a merged coding system, and deriving a CU-level weighted bidirectional prediction (BCW) index of the bidirectional prediction of merge candidates based on template matching cost.

[0013] In a fourth aspect, embodiments of this disclosure provide a computer program, and the computer program enables a computer to execute a method comprising: bidirectional prediction of merge candidates for a list of merge candidates of a merged coding unit (CU), and deriving a CU-level weighted bidirectional prediction (BCW) index of the bidirectional prediction of merge candidates based on template matching cost. Attached Figure Description

[0014] A detailed description is given with reference to the accompanying drawings. In the drawings, one or more numbers on the far left of the reference numerals identify the drawing in which that reference numeral first appears. The same reference numerals are used in different drawings to indicate similar or identical items or features.

[0015] Figure 1A and Figure 1B Example block diagrams illustrating the video encoding and video decoding processes according to embodiments of this disclosure are provided.

[0016] Figure 2 This illustrates motion prediction of the current image based on bidirectional prediction.

[0017] Figure 3 A flowchart is shown for a method to derive the BCW index for merge candidates.

[0018] Figure 4 An example system and method are shown for implementing the process of deriving the BCW index of merge candidates as described in this paper. Detailed Implementation

[0019] According to the VVC video coding standard (“VVC Standard”) and the motion prediction described therein, a computing system includes at least one or more processors and a computer-readable storage medium communicatively coupled to said one or more processors. The computer-readable storage medium is a non-transient or non-temporal computer-readable storage medium, as referenced below. Figure 4As shown, it stores computer-readable instructions. At least some of the computer-readable instructions stored on the computer-readable storage medium can be executed by one or more processors of a computing system to configure one or more processors to perform associated operations of the computer-readable instructions, including at least encoder operations as described by the VVC standard and decoder operations as described by the VVC standard. Some of these encoder and decoder operations based on the VVC standard will subsequently be described in further detail, although these subsequent descriptions should not be construed as an exhaustive description of encoder and decoder operations according to the VVC standard. Subsequently, “VVC Standard Encoder” and “VVC Standard Decoder” will describe the corresponding computer-readable instructions stored on the computer-readable storage medium that configure one or more processors to perform these corresponding operations (by way of example, which may be referred to as a “reference implementation” of the encoder or decoder).

[0020] Furthermore, according to exemplary embodiments of this disclosure, the VVC standard encoder and VVC standard decoder also include computer-readable instructions stored on a computer-readable storage medium, which can be executed by one or more processors of a computing system to configure one or more processors to perform operations not specified by the VVC standard. The VVC standard encoder should not be construed as limited to the operation of a reference implementation of the encoder, but includes further computer-readable instructions that configure one or more processors of the computing system to perform the further operations described herein. The VVC standard decoder should not be construed as limited to the operation of a reference implementation of the decoder, but includes further computer-readable instructions that configure one or more processors of the computing system to perform the further operations as described herein.

[0021] Figure 1A and Figure 1B Example block diagrams of a video encoding process 100 and a video decoding process 150 according to exemplary embodiments of the present disclosure are shown respectively.

[0022] In encoding process 100, the VVC standard encoder configures one or more processors of the computing system to receive one or more input images as input from image source 102. The input images comprise a number of pixels sampled by an image capture device (such as a light sensor array) and an uncompressed stream of multiple color channels (such as RGB color channels) that store color data at the original resolution of the image, wherein each channel uses a certain number of bits to store the color data of each pixel of the image. The VVC standard encoder configures one or more processors of the computing system to store the uncompressed color data in a compressed format, wherein the color data is stored at a resolution lower than the original resolution of the image and encoded as a luma (“Y”) channel and two chroma (“U” and “V”) channels with a resolution lower than the luma channel.

[0023] The VVC standard encoder encodes an image (the encoded image is referred to as the "current image," to distinguish it from any other image received from image source 102) by configuring one or more processors of the computing system to divide the original image into multiple units and subunits according to a partitioning structure. The VVC standard encoder configures one or more processors of the computing system to subdivide the image into multiple macroblocks ("MBs"), each macroblock having a size of 16×16 pixels, which can be further subdivided into multiple partitions. The VVC standard encoder configures one or more processors of the computing system to subdivide the image into multiple coding tree units ("CTUs"), the luma and chroma components of which can be further subdivided into multiple coding tree blocks ("CTBs"), which are further subdivided into multiple coding units ("CUs"). Alternatively, the VVC standard encoder configures one or more processors of the computing system to subdivide the image into units of N×N pixels, which can then be further subdivided into subunits. For the purposes of this disclosure, each of these largest subdivision units of the image may generally be referred to as a "block."

[0024] A CU is encoded using one luminance sample block and two corresponding chrominance sample blocks, where the image is not monochrome, and is encoded using a coding tree.

[0025] The VVC standard encoder configures a computing system with one or more processors to subdivide a block into partitions that are multiples of 4×4 pixels. For example, the partitions of a block can have dimensions of 8×4 pixels, 4×8 pixels, 8×8 pixels, 16×8 pixels, or 8×16 pixels.

[0026] Instead of encoding the color information of pixels in the full-resolution original image, the VVC standard encoder configures one or more processors of the computing system to encode the color information of an image at a lower resolution than the input image, and to store the color information in fewer bits than the input image, by encoding the color information of blocks and their subdivisions of the image.

[0027] Furthermore, the VVC standard encoder encodes an image by configuring one or more processors of the computing system to perform motion prediction on blocks of the current image. Motion prediction coding refers to storing image data of blocks of the current image (where the blocks of the original image before encoding are referred to as "input blocks") using motion information and multiple prediction units ("PUs") based on intra-frame prediction 104 or inter-frame prediction 106.

[0028] Motion information refers to data describing the motion of an image or its block structure of units or subunits, such as motion vectors and references to blocks in the current or reference image. A PU can refer to a unit or multiple subunits corresponding to a block structure within a block structure of an image, such as an MB or CTU, where blocks are partitioned based on image data and encoded according to the VVC standard. The motion information corresponding to a PU can describe motion predictions encoded by a VVC standard encoder as described herein.

[0029] The VVC standard encoder configures one or more processors in the computing system to encode motion prediction information on each block of an image in an encoding order (e.g., raster scan order, where the first block decoded is the topmost and leftmost block of the image). The encoded block is called the "current block" to distinguish it from any other block in the same image.

[0030] According to intra-prediction 104, one or more processors of the computing system are configured to encode blocks by referencing motion information and PUs of one or more other blocks in the same image. According to intra-prediction coding, one or more processors of the computing system perform intra-prediction 104 (also known as spatial prediction) computation by encoding motion information of the current block based on spatial neighbor samples from spatially neighboring blocks of the current block.

[0031] According to inter-frame prediction 106, one or more processors of the computing system are configured to encode blocks by referencing motion information and PU from one or more other images. One or more processors of the computing system are configured to store one or more previously encoded and decoded images in a reference image buffer for the purpose of inter-frame predictive coding; these stored images are referred to as reference images.

[0032] One or more processors are configured to perform inter-frame prediction¹⁰⁶ (also known as temporal prediction or motion-compensated prediction) computations by encoding motion information of the current block based on samples from one or more reference images. Inter-frame prediction can also be computed based on unidirectional or bidirectional prediction: in unidirectional prediction, only one motion vector pointing to a reference image is used to generate the prediction signal for the current block. In bidirectional prediction, two motion vectors, each pointing to a corresponding reference image, are used to generate the prediction signal for the current block.

[0033] Bidirectional prediction, as implemented in the early HEVC standard, generates a bidirectional prediction signal by averaging two prediction signals obtained from two different reference images and / or using two different motion vectors. In contrast, the bidirectional prediction mode implemented in the VVC standard is not limited to simple averaging; it can also perform a weighted average of the two prediction signals, a technique simply referred to as coding unit (“CU”) level weighted (“BCW”) bidirectional prediction. The weighted average of the two prediction signals P0 and P1 is calculated according to Equation 1 below to obtain P... bi-pred , where >> is the bitwise right shift operator.

[0034] P bi-pred =((8-w)*P0+w*P1+4)>>3

[0035] Formula 1 is mathematically equivalent to:

[0036]

[0037] Therefore, P0 and P1 can be weighted equally or unequally depending on the value of the weight w applied to Formula 1. For a weight w = 4, P0 and P1 can have equal weights. Thus, a weight w = 4 is an "equal weight" applied to Formula 1 (but not necessarily to the other weighted average bidirectional prediction formulas in this paper). Furthermore, when applied to Formula 1, a weight w that makes P0 and P1 both positively weighted is a "positive weight" relative to Formula 1 (where equal weights are also positive weights), and when applied to Formula 1, a weight w that makes one of P0 and P1 negatively weighted is a "negative weight" relative to Formula 1.

[0038] According to the bidirectional prediction mode implemented by the VVC standard, a set of 5 possible weight values, w∈(-2,3,4,5,10), are allowed in the weighted average bidirectional prediction. For each bidirectional prediction CU, the weight w is determined in one of two ways: 1) for non-merged coded CUs, the weight index is signaled after the motion vector difference; 2) for merged coded CUs, the weight index is inferred from neighboring blocks based on the merge candidate index.

[0039] Based on the bidirectional prediction mode implemented by the VVC standard, BCW is only applied to CUs with 256 or more luminance samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency images, all 5 weights are used. For non-low-latency images, a set of only 3 possible weight values ​​(w∈{3,4,5}) is used.

[0040] The VVC standard encoder configures one or more processors of the computing system to encode the CU (Cubic Frame) to include a reference index to identify one or more prediction signals for the current block, for reference by the VVC standard decoder. Motion vectors and the reference index are sent to the decoder to identify where the prediction signal for the current block originates. One or more processors of the computing system can encode the CU to include inter-frame prediction indicators. The inter-frame prediction indicators specify list 0 prediction, list 1 prediction, or bidirectional prediction, where list 0 prediction references a first reference image list called list 0, list 1 prediction references a second reference image list called list 1, and bidirectional prediction simultaneously references reference image lists called list 0 and list 1, respectively.

[0041] When the inter-frame prediction indicator indicates list 0 prediction or list 1 prediction, one or more processors of the computing system are configured to encode a CU, which includes a reference index that references a reference image in a reference image buffer referenced by list 0 or list 1, respectively. When the inter-frame prediction indicator indicates bidirectional prediction, one or more processors of the computing system are configured to encode a CU, which includes a first reference index that references a first reference image in a reference image buffer referenced by list 0, and a second reference index that references a second reference image in a reference image buffer referenced by list 1.

[0042] The VVC standard encoder configures one or more processors in the computing system to encode each current block of the image separately and output a prediction block for each encoding. According to the VVC standard, the CTU can achieve 128×128 luma samples (depending on the chroma format, plus the corresponding chroma samples). The CTU can be further partitioned into multiple CUs based on a quadtree, binary tree, or ternary tree. One or more processors in the computing system are configured to ultimately record the encoding parameter set at the syntax structure of the leaf nodes of the partitioned structure, such as the encoding mode (intra-frame mode or inter-frame mode), motion information for inter-frame coded blocks (reference index, motion vectors, etc.), and quantization residual coefficients.

[0043] After the output prediction block, the VVC standard encoder configures one or more processors of the computing system to send a set of coding parameters, such as coding mode (i.e., intra-frame or inter-frame prediction), intra-frame prediction mode or inter-frame prediction mode, and motion information, to the entropy encoder 124 (described below).

[0044] The VVC standard provides semantics for recording the coding parameter set of the CU. For example, regarding the aforementioned coding parameter set, the CU's `pre_mode_flag` is set to 0 for inter-coded blocks and 1 for intra-coded blocks; the CU's `general_merge_flag` is set to indicate whether a merging mode is used in the CU's inter-frame prediction; the CU's `inter_affine_flag` and `cu_affine_type_flag` are set to indicate whether affine motion compensation is used in the CU's inter-frame prediction; and `mvp_10_flag` and `mvp_11_flag` are set to indicate the reference image index in list 0 or list 1, respectively. It should be understood that the VVC standard also includes semantics beyond the scope of this disclosure for recording various other information, flags, and options.

[0045] The VVC standard encoder further implements one or more mode decision and encoder control settings 108, including rate control settings. One or more processors of the computing system are configured to perform mode decisions by selecting an optimized prediction mode for the current block, based on a rate-distortion optimization method, after intra-frame or inter-frame prediction.

[0046] The bitrate control settings configure one or more processors of the computing system to assign different quantization parameters (“QP”) to different images. The size of the QP determines the scale at which image information is quantized by one or more processors during the encoding process (as described below), thereby determining the extent to which the encoding process 100 discards image information from the sequence's MB during the encoding process (due to information loss between step spans).

[0047] The VVC standard encoder also implements a subtractor 110. One or more processors of the computing system are configured to perform the subtraction operation by calculating the difference between the input block and the prediction block. The prediction block is subtracted from the input block based on an optimized prediction pattern. The difference between the input block and the prediction block is called the prediction residual, or simply "residual".

[0048] Based on the predicted residuals, the VVC standard encoder further implements Transform 112. One or more processors of the computing system are configured to perform transform operations on the residuals through matrix arithmetic to compute an array of coefficients (which may be referred to as "residual coefficients," "transform coefficients," etc.) to encode the current block as a transform block ("TB"). The transform coefficients may be coefficients representing one of several spatial transformations, such as diagonal flips, vertical flips, or rotations, which may be applied to sub-blocks.

[0049] It should be understood that coefficients can be stored as two components, absolute value and sign, which will be described in further detail below.

[0050] Subblocks of the CU (e.g., PU and TB) can be any combination of subblock sizes as described above. The VVC standard encoder configures one or more processors of the computing system to subdivide the CU into a residual quadtree (“RQT”), which is a hierarchical structure of the TB. The RQT provides the order of motion prediction and residual coding for subblocks at each level and recursively goes down to each level of the RQT.

[0051] The VVC standard encoder further implements quantization 114. One or more processors of the computing system are configured to perform quantization operations on the residual coefficients via matrix arithmetic based on the quantization matrix and QP allocated above. Residual coefficients falling within a certain interval are retained, while those falling outside that interval are discarded.

[0052] The VVC standard encoder further implements inverse quantization 116 and inverse transform 118. One or more processors of the computing system are configured to perform inverse quantization and inverse transform operations on the quantized residual coefficients through inverse matrix arithmetic operations of the aforementioned quantization and transform operations. The inverse quantization and inverse transform operations produce the reconstructed residuals.

[0053] The VVC standard encoder further implements adder 120. One or more processors of the computing system are configured to perform the addition operation by adding the prediction block and reconstructing the residual, and output the reconstructed block.

[0054] The VVC standard encoder also implements a loop filter 122. One or more processors of the computing system are configured to apply loop filters, such as a deblocking filter, a sample adaptive offset (“SAO”) filter, and an adaptive loop filter (“ALF”), to the reconstructed blocks, outputting filtered reconstructed blocks.

[0055] The VVC standard encoder further configures one or more processors of the computing system to output the filtered reconstructed blocks to the decoded image buffer (“DPB”) 200. As described above, the DPB 200 stores the reconstructed image instead of the current image, which is used as a reference image in the encoded image by one or more processors of the computing system, referencing the intra-frame prediction as described above.

[0056] The VVC standard encoder also implements an entropy encoder 124. One or more processors of the computing system are configured to perform entropy encoding, wherein the symbols constituting the quantized residual coefficients are mapped and encoded into binary strings (hereinafter referred to as "bins") according to a context-aware binary arithmetic codec ("CABAC"), which can be transmitted in the output bit stream at a compressed bit rate. The symbols of the encoded quantized residual coefficients include the absolute values ​​of the residual coefficients (these absolute values ​​are subsequently referred to as "residual coefficient levels").

[0057] Therefore, the entropy encoder configures one or more processors in the computing system to encode the residual coefficient levels of the block; bypasses the encoding of the residual coefficient symbols and records the residual coefficient symbols together with the coded block; records the set of encoding parameters, such as the encoding mode, intra-frame prediction mode or inter-frame prediction mode, and motion information encoded in the syntax structure of the coded block (such as the set of image parameters (“PPS”) found in the image header and the set of sequence parameters (“SPS”) found in the sequence of multiple images); and outputs the coded block.

[0058] The VVC standard encoder configures one or more processors of the computing system to output an encoded image consisting of encoded blocks from the entropy encoder 124. The encoded image is output to a transmission buffer, where it is ultimately packaged into a bit stream for output from the VVC standard encoder. The bit stream is written by one or more processors of the computing system to a non-transient or non-temporary computer-readable storage medium of the computing system for transmission.

[0059] In the decoding process 150, the VVC standard decoder configures one or more processors of the computing system to receive one or more encoded images from the bit stream as input.

[0060] The VVC standard decoder implements the entropy decoder 152. One or more processors of the configured computing system perform entropy decoding, where, according to CABAC, bins are decoded by inverse mapping symbols to bins, thereby recovering the entropy-coded quantized residual coefficients. The entropy decoder 152 outputs the quantized residual coefficients, the bypass-coded residual coefficient symbols, and syntax structures such as PPS and SPS.

[0061] The VVC standard encoder also implements inverse quantization 154 and inverse transform 156. One or more processors of the computing system are configured to perform inverse quantization and inverse transform operations on the quantized residual coefficients via matrix arithmetic operations, which are the inverse operations of the quantization and transform operations described above. The inverse quantization and inverse transform operations generate the reconstructed residuals.

[0062] Furthermore, based on the set of coding parameters recorded by the entropy encoder 124 in syntax structures such as PPS and SPS (or optionally, received by out-of-band transmission or encoded into the decoder), and the coding modes contained in the set of coding parameters, the VVC standard decoder decides whether to apply intra-frame prediction 156 (i.e., spatial prediction) or motion compensation prediction 158 (i.e., temporal prediction) to the reconstruction residual.

[0063] When intra-frame prediction is specified in the coding parameter set, the VVC standard decoder configures one or more processors of the computing system to perform intra-frame prediction 158 using the prediction information specified in the coding parameter set. Intra-frame prediction 158 thus generates a prediction signal.

[0064] With the encoding parameter set specifying inter-frame prediction, the VVC standard decoder configures one or more processors of the computing system to perform motion-compensated prediction 160 using a reference image from the DPB 200. Motion-compensated prediction 160 thus produces a prediction signal.

[0065] The VVC standard decoder also implements adder 162. Adder 162 configures one or more processors of the computing system to perform addition operations on the reconstructed residual and the predicted signal, thereby outputting a reconstructed block.

[0066] The VVC standard decoder also implements a loop filter 164. One or more processors of the computing system are configured to apply loop filters (such as deblocking filters, SAO filters, and ALF) to the reconstructed blocks, thereby outputting filtered reconstructed blocks.

[0067] The VVC standard encoder further configures one or more processors of the computing system to output the filtered reconstructed blocks to the DPB 200. As mentioned above, the reconstructed image stored in the DPB 200 is used by one or more processors of the computing system as a reference image in the encoded image, rather than storing the current image, as described above in the reference motion compensation prediction section.

[0068] The VVC standard decoder further configures one or more processors of the computing system to output the reconstructed image from the DPB to the user-visual display of the computing system, such as a television display, personal computing monitor, smartphone display, or flat panel display.

[0069] Therefore, as shown in the encoding process 100 and decoding process 150 above, both the VVC standard encoder and the VVC standard decoder implement motion prediction coding conforming to the VVC specification. Each of the VVC standard encoder and VVC standard decoder is configured with one or more processors in the computing system to generate a reconstructed image based on a previously reconstructed image (DPB) according to the motion compensation prediction described by the VVC standard, wherein the previously reconstructed image is used as a reference image in the motion compensation prediction described herein.

[0070] The VVC standard encoder and VVC standard decoder are each configured with one or more processors in the computing system to predict motion information of the CU in the reconstructed image through various merging modes, such as regular merging mode, template matching merging mode, decoder-side motion vector refinement (“DMVR”) mode, motion vector difference (“MMVD”) mode, affine merging mode, affine plus MMVD merging mode, inter-frame and intra-frame combined prediction (“CHP”) mode, etc. This CU with motion information predicted by the merging mode is subsequently referred to as a “merged coded CU”. The motion information may include multiple motion vectors. The derivation of motion vectors is known to those skilled in the art and will not be elaborated here.

[0071] Motion information for the CU in the reconstructed image can include a list of motion candidates. The list of motion candidates can be a data structure containing references to multiple motion candidates. Motion candidates can be block structures or their sub-units, such as pixels or any other suitable subdivision of the current image block structure, or they can be references to motion candidates from another image. Motion candidates can be spatial motion candidates or temporal motion candidates. By applying motion vector compensation (“MVC”), the VVC standard decoder configures one or more processors of the computing system to select motion candidates from the list and derive the motion vectors of the motion candidates as the motion vectors of the CU in the reconstructed image.

[0072] The motion candidate list can be a merge candidate list and can include up to five types of merge candidates, or, depending on the ECM, up to six types. During the encoding of a CU, the merge candidate list can be sent to one or more processors in the computing system via merge candidate indices. For example, spatial merge candidates can be derived by searching for neighboring blocks of the current CU and adding the motion information of those neighboring blocks to the merge candidate list.

[0073] By providing an example, a list of time merging candidates can be exported by exporting the motion information of co-located CUs belonging to a co-located reference image.

[0074] The list of merging candidates for the current CU, encoded according to the merging pattern, can include the following merging candidates in order: spatial motion vector prediction (“MVP”) candidates from spatially adjacent CUs to the current CU; temporary MVP candidates from CUs at the same location in the current CU; history-based MVP (“HMVP”) candidates from the FIFO table; pairwise average MVP candidates; and zero motion vectors.

[0075] Furthermore, after constructing the merge candidate list, the merge candidates are reordered according to an adaptive reordering method based on template matching (hereinafter referred to as "ARMC-TM"). Based on template matching, the merge candidates are reordered in ascending order of cost value. For simplicity, merge candidates in the last but not the first subgroup are not reordered. The template matching cost value of a merge candidate is the sum of absolute differences (SAD) between the template sample of the current block and its corresponding reference sample. The template of the current block includes the set of reconstructed samples adjacent to the current block. The reference sample of the template of the current block is located using the motion information of the merge candidates.

[0076] ARMC-TM can be applied to motion prediction through merging modes, including: regular merging mode; CHP mode; adaptive decoder-side motion vector refinement mode; template matching (“TM”) merging mode; and affine merging mode, excluding sub-block-based temporal motion vector prediction (“SbTMVP”).

[0077] Any merge candidate in the merge candidate list can include motion information predicted via bidirectional prediction. In this case, the reference sample for the merge candidate template is also generated via bidirectional prediction; therefore, bidirectional template matching can be performed based on such merge candidates, which are subsequently referred to as "bidirectional matching ('BM') candidates".

[0078] Figure 2 The illustration shows motion prediction performed on the current image 202 based on bidirectional prediction. The current image 202 includes a current block 202A, which includes a template 202B. According to bidirectional prediction, the template's reference sample references two co-located reference images 204 and 206, one from reference list 0 in a first time direction and the other from reference list 1 in a second time direction. The motion information of the current block 202A references the co-located reference block 204A of the co-located reference image 204, and also references the co-located reference block 206A of the co-located reference image 206. The template 202B of the current block 202A references the reference sample 204B of the co-located reference image 204, and also references the reference sample 206B of the co-located reference image 206.

[0079] Based on the adaptive decoder-side motion vector refinement technique, two additional merging modes are provided for merging candidates that meet the DMVR conditions: one merging mode is that the motion vector of the current CU is refined by the multi-channel DMVR in the first time direction instead of the second time direction, and the other merging mode is that the motion vector of the current CU is refined by the multi-channel DMVR in the second time direction instead of the first time direction.

[0080] For the two additional merging modes, a common merging candidate list is constructed only from merging candidates that meet the DMVR conditions. Similar to the merging candidate list described above, merging candidates for the common merging candidate list are obtained from spatially adjacent coding blocks, TMVPs, non-adjacent blocks, HMVPs, and paired candidates.

[0081] If the merge candidate list includes, for example Figure 2 The BM candidates shown, for a weighted average of the predicted signals, such bidirectional prediction merge candidates inherit the weighted index inferred from neighboring blocks based on the merge candidate index notified by the used signals.

[0082] Given the list of merge candidates, a multi-path DMVR process is applied to the merge candidates to refine the motion vectors. The DMVR process is modified by measuring the motion vector difference ("MVD") between the motion vector and the motion vector prediction ("MVP"). MVD can be calculated as the sum of absolute differences after mean reduction ("MRSAD"), or, if the weights are unequal and bidirectional predictions are weighted with BCW weights, as the sum of absolute transformation differences after mean reduction ("MRSATD"); however, during the first path of the multi-path DMVR process (i.e., the PU level), either MVD0 (MVD in the first time direction) or MVD1 (MVD in the second time direction) is set to zero. The merge index is encoded in the same manner as in the regular merge mode.

[0083] According to an example embodiment of this disclosure, additional BCW weights can be derived based on the difference in image sequence number (“POC”).

[0084] As an example, if two reference images are in the same time direction relative to the current image (i.e., both are from the past or from the future), and the current image is a low-latency image, then a weight pair (-3, 11) can be added.

[0085] As another example, if two reference images are in the same time direction relative to the current image (i.e., both are from the past or from the future), and the current image is not a low-latency image, then a weight pair (-2, 10) can be added.

[0086] As yet another example, for any other case, the weight pair (2, 6) can be added.

[0087] According to any of the examples above, when the POC distance is the same, the larger value from the weight pair can be assigned to the nearest POC reference image or the list 0 reference image.

[0088] Based on any of the examples above, additional BCW weights can be assigned to the pairwise merge candidates and the zero merge candidate, instead of equal weights with respect to the weighted average bidirectional prediction equation.

[0089] As described above, according to the VVC standard and ECM specification, the BCW index of a merged coded block is inherited from neighboring blocks based on the merge candidate index notified by signaling. However, the inherited BCW index may not be suitable for the merged coded block. Therefore, the example embodiments of this disclosure derive the BCW index based on a cost value based on template matching.

[0090] In one or more aspects, exemplary embodiments of this disclosure derive a merged candidate BCW index based on template matching cost.

[0091] In one or more aspects, exemplary embodiments of this disclosure derive a merged candidate BCW index and a possible weight value reduction set based on template matching cost.

[0092] In one or more aspects, exemplary embodiments of this disclosure provide for deriving a merged candidate BCW index based on template matching cost, and a possible expanded set of weight values.

[0093] In one or more aspects, exemplary embodiments of this disclosure derive a merged candidate BCW index based on template matching cost, and a possible expanded set of weight values ​​that improves weight accuracy.

[0094] In one or more aspects, exemplary embodiments of this disclosure derive a merged candidate BCW index based on template matching cost, while adjusting the template matching cost of inherited BCW weights.

[0095] In one or more aspects, exemplary embodiments of this disclosure provide the application of bidirectional optical flow to bidirectional prediction merging candidates with equal weights relative to a weighted average bidirectional prediction formula.

[0096] In one or more aspects, exemplary embodiments of this disclosure derive a merge candidate BCW index based on template matching cost before applying ARMC-TM to the merge candidate.

[0097] In one or more aspects, exemplary embodiments of this disclosure derive the BCW index of the merge candidate based on template matching cost after applying the ARMC-TM merge candidate.

[0098] In one or more aspects, exemplary embodiments of this disclosure derive merge candidate BCW indexes based on template matching costs of subsets of possible merge patterns.

[0099] In one or more aspects, exemplary embodiments of this disclosure provide for deriving merge candidate BCW indices while merging candidate BCW indices from non-adjacent space merge candidates.

[0100] The above aspects of the exemplary embodiments of this disclosure will then be described in further detail.

[0101] Figure 3 A flowchart of method 300 is shown, in which a VVC standard encoder or VVC standard decoder configures one or more processors of a computing system to derive a merged candidate BCW index based on template matching cost, while utilizing a set of possible weight values ​​according to the VVC standard and ECM specification.

[0102] It should be understood that the use of "decoder side" here does not mean that the method is implemented solely by the decoder; rather, the steps of the method can be implemented by both the encoder and the decoder in a similar or identical manner. The reference to "VVC standard encoder or VVC standard decoder" should be understood as meaning that the steps are performed in a similar or identical manner by one or more processors of the computing system, regardless of whether the processor is configured as a VVC standard encoder or a VVC standard decoder.

[0103] In step 302, as described above, the VVC standard encoder or VVC standard decoder configures one or more processors in the computing system to build a merge candidate list for the merged encoding CU.

[0104] In step 304, for each merge candidate in the merge candidate list, if the merge candidate is a bidirectional prediction candidate, the VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to derive the BCW index of the bidirectional prediction merge candidate based on the template matching cost. The bidirectional prediction weights of the merge candidates are determined by the weight index of the bidirectional prediction merge candidates (hereinafter referred to as the "BCW index"). The VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to derive the BCW index in sub-steps 304A, 304B, 304C, 304D, and 304E, which will be described subsequently.

[0105] In step 304A, for each bidirectional prediction weight (i.e., BCW index), the VVC standard encoder or VVC standard decoder configures one or more processors to calculate the template matching cost using the SAD between the template samples of the merged coded CU and their respective corresponding reference samples. (It should be understood that, according to bidirectional prediction under the VVC standard, the template includes the reconstructed samples to the left and / or top of the merged coded CU, such as...) Figure 2 (As shown in template 202B.)

[0106] In step 304B, the VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to generate a reference sample for the template by using bidirectional prediction with corresponding bidirectional prediction weight values.

[0107] At step 304C, the VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to calculate the template matching cost for each of a set of possible BCW weight values, and selects the BCW weight value that produces the lowest template matching cost as the BCW index of the bidirectional prediction merging candidate for each calculated template matching cost. Various examples of performing step 304C are subsequently described according to exemplary embodiments of this disclosure.

[0108] For example, for bidirectional prediction merging candidates in non-low-latency images, three template matching cost values ​​are calculated for each of the three possible BCW weight values ​​in a set of three BCW weight values. The three possible BCW weight values ​​represent three different weight values ​​(w∈€{3,4,5}, where w is defined as in Equation 1 above).

[0109] Therefore, among the three template matching costs calculated based on these three possible weight values, the VVC standard encoder or VVC standard decoder configures one or more processors in the computing system to determine the weight value that produces the lowest template matching cost as the BCW index for bidirectional prediction merging candidates.

[0110] According to example step 304C, a possible set of weight values ​​corresponds to a possible set of weight values ​​in the weighted average bidirectional prediction provided by the VVC standard and / or ECM specification. That is, for low-latency images, the VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to calculate the template matching cost from 5 possible weight values ​​(w∈{-2,3,4,5,10}), and for non-low-latency images, the VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to calculate the template matching cost from 3 possible weight values ​​(w∈{3,4,5}). Furthermore, according to Equation 1 above, the two prediction signals P0 and P1 are weighted and averaged to obtain P... bi-pred .

[0111] According to example step 304C, a set of possible weight values ​​is a reduced set of possible weight values ​​compared to the possible weight values ​​in the weighted average bidirectional prediction provided by the VVC standard and / or ECM specification. For both low-latency and non-low-latency images, the VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to calculate the template matching cost from a set of three possible weight values ​​(w∈(3,4,5}); that is, the negative weights in Equation 1 are excluded. Furthermore, according to Equation 1 above, the weighted average of the two prediction signals P0 and P1 is calculated as P bi-pred .

[0112] According to example step 304C, a set of possible weight values ​​is an expanded set of possible weight values ​​compared to the possible weight values ​​in the weighted average bidirectional prediction provided by the VVC standard and / or ECM specification. For example, a VVC standard encoder or VVC standard decoder configures one or more processors in the computing system to compute the template matching cost from a set of seven possible weight values ​​(w∈{1,2,3,4,5,6,7}) for both low-latency and non-low-latency images. This can be achieved by removing the bit overhead used in the CU encoding for signal notification of the BCW index. According to Equation 1 above, the weighted average of the two prediction signals P0 and P1 yields P... bi-pred .

[0113] According to a further example of step 304C, a set of possible weight values ​​is an expanded set of possible weight values ​​compared to the possible weight values ​​in the weighted average bidirectional prediction provided by the VVC standard and / or ECM specification. For example, the VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to calculate the template matching cost of the low-latency image and the non-low-latency image from a set of 5 possible weight values ​​(w∈(6,7,8,9,10}). Furthermore, the weighted average of the two prediction signals P0 and P1 is calculated as P according to Equation 2 below. bi-pred , where >> is the bitwise right shift operator.

[0114] P bi-pred =((16-w)*P0+w*P1+8)>>4

[0115] Formula 2 is mathematically equivalent to:

[0116]

[0117] Therefore, depending on the value of the weight w applied to Formula 2, P0 and P1 can be weighted equally or unequally. For a weight w = 8, P0 and P1 have equal weights. Therefore, a weight w = 8 is an "equal weight" applicable to Formula 2 (but not necessarily to the other weighted average two-way prediction formulas here). Furthermore, when applied to Formula 2, a weight w that positively weights both P0 and P1 is a "positive weight" relative to Formula 2 (where equal weights are also positive weights), and a weight w that negatively weights one of P0 and P1 is a "negative weight" when applied to Formula 2.

[0118] According to a further example embodiment of this disclosure, in step 304D, the VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to calculate the template matching cost for each of a possible subset of BCW weight values ​​based on inherited BCW weight values, and selects the BCW weight value that produces the lowest template matching cost as the BCW index of the bidirectional prediction merging candidate for each calculated template matching cost. Various examples of performing step 304D are subsequently described according to example embodiments of this disclosure.

[0119] Following the example in step 304D, the VVC standard encoder or VVC standard encoder decoder configures one or more processors in the computing system to calculate the template matching cost for each of a possible subset of BCW weight values, based on inherited BCW weight values ​​that are positive weights relative to the weighted average bidirectional prediction formula (i.e., w in Formulas 1 and 2). For positive inherited weights, the subset of possible weight values ​​used to calculate the template matching cost only includes positive weights relative to the weighted average bidirectional prediction formula (the determination of positive weights differs for Formulas 1 and 2 above). Note that for positive weight w, it can be concluded that the weight of P0 (i.e., w is 8 for Formula 1 or 16 for Formula 2) and the weight of P1 (i.e., w) are both greater than 0.

[0120] As an example, a possible set of weight values ​​for Equation 1 is determined as (-2, 3, 4, 5, 10). For one of the positive inheritance weights (i.e., for Equation 1, the inherited weight is {3, 4, 5}), the template matching cost for {3, 4, 5} is calculated according to step 304D, but the template matching cost for {-2, 10} is not calculated. For non-positive inheritance weights, as in step 304C, the template matching cost for {-2, 3, 4, 5, 10} is calculated.

[0121] According to the example of step 304D, the VVC standard encoder or VVC standard encoder decoder configures one or more processors in the computing system to calculate the template matching cost for each of a subset of possible BCW weight values ​​based on inherited BCW weights, where the inherited BCW weights are negative weights relative to the weighted average bidirectional prediction. For negative inherited weights, the subset of possible weight values ​​for which the template matching cost is calculated includes only the negative weights with respect to the formula for weighted average bidirectional prediction (the determination of the negative weights differs for Formulas 1 and 2 above). For a negative weight w, it is thus determined that at least one of the weights of P0 and P1 is less than 0.

[0122] As an example, a possible set of weight values, {-2,3,4,5,10}, is determined based on one or more examples from step 304C above. For a negative inherited weight (e.g., for one of {-2,10} in Formula 1), the template matching cost for {-2,10} is calculated according to step 304D.

[0123] According to a further embodiment extending the foregoing example, even for negative inheritance weights, the subset of possible BCW weight values ​​used to calculate template matching costs includes equal weights with respect to the weighted average bidirectional prediction formula.

[0124] As an example, a possible set of weight values ​​{-2,3,4,5,10} is determined based on one or more examples from step 304C above. When the inherited weight is negative (i.e., with respect to one of {-2,10} in Formula 1), the template matching cost for {-2,4,10} is calculated according to step 304D (because 4 is an equal weight for Formula 1, even though 4 is not a negative weight for Formula 1).

[0125] Following the example in step 304D, the VVC standard encoder or VVC standard encoder decoder configures one or more processors in the computing system to calculate the template matching cost for each of a subset of possible BCW weight values, where the possible BCW weight values ​​are similar to inherited BCW weights. A weight w is considered similar to an inherited weight when the absolute difference between the weight w and the inherited weight is less than a predefined threshold. The predefined threshold is a non-zero positive integer.

[0126] As an example, a possible set of weight values ​​determined based on one or more examples in step 304C is {-2,3,4,5,10} and the predefined threshold is set to 2. Therefore, for the inherited weight 4, the template matching cost of (3,4,5) is calculated according to step 304D. For the inherited weight 3, the template matching cost of {3,4} is calculated according to step 304D. When the inherited weight is -2, the template matching cost of {-2} is calculated according to step 304D.

[0127] In another example, a possible set of weight values ​​is determined as {-4,-3,-2,-1,1,2,3,4,5,6,7,9,10,11,12}, and a predefined threshold is set to 2. Therefore, for the inherited weight 4, the template matching cost for {3,4,5} is calculated according to step 304D. For the inherited weight 3, the template matching cost for {2,3,4} is calculated according to step 304D. For the inherited weight -2, the template matching cost for {-3,-2,-1} is calculated according to step 304D.

[0128] As another example, based on one or more examples of step 304C above, a possible set of weight values ​​is determined as {1,2,3,4,5,6,7}, and a predefined threshold is set to 2.

[0129] As another example, a set of possible weight values ​​{0,1,2,3,4,5,6,7,8} is determined based on one or more examples of step 304C above, and a predefined threshold is set to 2.

[0130] According to the example of step 304D, the VVC standard encoder or VVC standard encoder decoder is configured to compute one or more processors in the computing system to calculate the template matching cost for each of a possible subset of BCW weight values ​​based on inherited BCW weights, which is further based on the POC distance between the current image and the reference image.

[0131] As an example, when using the template matching method to derive the BCW index for bidirectional prediction merging candidates, the POC distance (hereinafter referred to as Poc_diff_L0) between the current image and the reference image from reference image list 0, and the POC distance (hereinafter referred to as Poc_diff_L1) between the current image and the reference image from reference image list 1 are calculated according to step 304D.

[0132] For Poc_diff_L0 equal to Poc_diff_L1, the template matching cost for all weights in a set of possible weight values ​​is calculated according to step 304C. For Poc_diff_L0 less than Poc_diff_L1, the template matching cost for weights less than or equal to the weights in the weighted average bidirectional prediction formula is calculated according to step 304D. That is, the weight applied to P1 (i.e., w) is less than or equal to the weight applied to P0 (i.e., w is 8 in Formula 1, or w is 16 in Formula 2), where P1 is the prediction signal using a reference image from reference image list 1, and P0 is the prediction signal using a reference image from reference image list 0. On the other hand, for Poc_diff_L0 greater than Poc_diff_L1, the template matching cost for weights equal to or greater than the weights in the weighted average bidirectional prediction formula is calculated according to step 304D. That is, the weight applied to P1 is greater than or equal to the weight applied to P0.

[0133] As an example, based on one or more examples from step 304C above, a possible set of weight values ​​is determined as {-2,3,4,5,10}. For Poc_diff_L0 equal to Poc_diff_L1, the template matching cost for {-2,3,4,5,10) is calculated according to step 403C. For Poc_diff_L0 less than Poc_diff_L1, the template matching cost for {-2,3,4} is calculated according to step 304D. For Poc_diff_L0 greater than Poc_diff_L1, the template matching cost for {4,5,10} is calculated according to step 304D.

[0134] According to the example of step 304D, the VVC standard encoder or VVC standard encoder decoder configures one or more processors in the computing system to calculate the template matching cost for each of a subset of possible BCW weight values ​​similar to the inherited BCW weights, the subset being further based on the POC distance between the current image and the reference image.

[0135] When deriving the BCW index for bidirectional prediction merging candidates using the template matching method, in addition to Poc_diff_L0 and Poc_diff_L1, the value diff_w needs to be calculated by subtracting the inherited weight value from each value in a possible set of BCW weight values. For Poc_diff_L0 equal to Poc_diff_L1, the template matching cost is calculated for those weights whose absolute value of diff_w is less than a first predefined threshold. For Poc_diff_L0 less than Poc_diff_L1, the template matching cost is calculated for those weights whose diff_w is less than or equal to 0 and whose absolute value of diff_w is less than a second predefined threshold. For Poc_diff_L0 greater than Poc_diff_L1, the template matching cost is calculated for those weights whose diff_w is greater than or equal to 0 and whose absolute value of diff_w is less than a third predefined threshold. The first, second, and third predefined thresholds can be the same as or different from each other.

[0136] As an example, a set of possible weight values ​​{1,2,3,4,5,6,7} is determined based on one or more examples from step 304C above. The first, second, and third predefined thresholds are set to 2, 3, and 3, respectively. Assume the inherited weight is 4.

[0137] Therefore, for Poc_diff_L0 equal to Poc_diff_L1, the template matching cost for {3,4,5} is calculated according to step 304D. For Poc_diff_L0 less than Poc_diff_L1, the template matching cost for {2,3,4} is calculated according to step 304D. For Poc_diff_L0 greater than Poc_diff_L1, the template matching cost for {4,5,6} is calculated according to step 304D.

[0138] According to another embodiment, when deriving the BCW index of the bidirectional prediction merge candidate using the template matching method, the absolute difference between each value in Poc_diff_L0, Poc_diff_L1, and a possible set of BCW weight values ​​and the inherited weight is calculated. For Poc_diff_L0 equal to Poc_diff_L1, the template matching cost of those weights whose absolute difference is less than a first predefined threshold is calculated. For Poc_diff_L0 not equal to Poc_diff_L1, the template matching cost of those weights whose absolute difference is less than a second predefined threshold is calculated. The first predefined threshold is different from the second predefined threshold.

[0139] As an example, a set of possible weight values ​​{1,2,3,4,5,6,7} is determined based on one or more examples from step 304C above. The first and second predefined thresholds are set to 3 and 2, respectively. Assume the inherited weight is 3.

[0140] Therefore, for Poc_diff_L0 equal to Poc_diff_L1, the template matching cost for {1,2,3,4,5} is calculated according to step 304D. For Poc_diff_L0 not equal to Poc_diff_L1, the template matching cost for {2,3,4} is calculated according to step 304D.

[0141] Furthermore, in step 304E, the VVC standard encoder or decoder configures one or more processors in the computing system to derive merged candidate BCW indices based on template matching costs, while simultaneously adjusting the template matching costs of inherited BCW weights according to values ​​calculated by the VVC standard and ECM specification. As described above, the VVC standard encoder or decoder configures one or more processors in the computing system to calculate the template matching cost for each bidirectional prediction weight (i.e., BCW index) using the SAD between multiple samples of a template from the merged encoded CU and their respective corresponding reference samples. However, since an inherited BCW index from a neighboring block may have higher accuracy than another BCW index, the VVC standard encoder or decoder further configures one or more processors in the computing system to adjust the template matching costs of the inherited BCW indices so that they are selected with priority over the template matching costs of other BCW indices.

[0142] As an example, the VVC standard encoder or decoder configures one or more processors in the computing system to multiply the template matching cost (TMcost) of the inherited BCW index by a weight less than 1. Therefore, the inherited BCW index has a higher probability of being selected. To reduce computational complexity, the VVC standard encoder or decoder configures one or more processors in the computing system to perform bitwise shift and subtraction operations to avoid multiplication operations. For example, setting the weight applied to the template matching cost of the inherited BCW index to 0.90625 can be replaced by TMcost - (TMcost >> 4) - (TMcos >> 5).

[0143] As another example, a VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to subtract an offset value from the template matching cost of the inherited BCW index. The offset value is a non-zero positive number determined based on a quantization parameter (“QP”). For example, the offset value could be set to a Lagrange multiplier λ.

[0144] Considering the higher precision of inherited weights, weights dissimilar to inherited weights may be less promising. According to another embodiment, the template matching cost of a weight dissimilar to its inherited weight is multiplied by a value greater than 1. A weight is determined to be dissimilar to its inherited weight based on the absolute difference between the weight and the inherited weight exceeding a pre-set threshold. As described above, weights dissimilar to their inherited weights can be excluded from the template matching cost calculation.

[0145] As an example, a predefined threshold is set to 1, and a set of possible weight values ​​{-2,3,4,5,10} is determined based on one or more examples from step 304C above. Assume the inherited weight is 3. The template matching cost TMcost for {-2,5,10} multiplied by 1.09375 can be replaced with TMcost + (TMcost >> 4) + (TMcost >> 5).

[0146] According to an example embodiment of this disclosure, a VVC standard encoder or VVC standard decoder configures one or more processors of a computing system to apply bidirectional optical flow (“BDOF”) to bidirectional prediction blocks with equal weights. The prediction samples of the bidirectional prediction blocks are refined using the BDOF technique.

[0147] Furthermore, in step 304F, optionally, the VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to adjust the template matching cost of BCW weights with equal weights according to the values ​​calculated by the VVC standard and ECM specification. Since the application of BDOF may be beneficial for predicting samples, the VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to adjust the template matching cost of BCW indices with equal weights to the weighted average bidirectional prediction formula, prioritizing the selection of template matching costs for other BCW indices. As an example, the VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to multiply the template matching cost of BCW indices with equal weights to the weighted average bidirectional prediction formula by a weight less than 1, or to subtract a positive value from the template matching cost of BCW indices with equal weights to the weighted average bidirectional prediction formula.

[0148] Alternatively, the VVC standard encoder or VVC standard decoder can configure one or more processors in the computing system to apply BDOF to all bidirectional prediction blocks, regardless of their respective merge candidate BCW indices.

[0149] Furthermore, according to an exemplary embodiment of this disclosure, in step 304, wherein the VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to derive the merge candidate BCW index based on template matching cost before applying ARMC-TM to the merge candidate, or to derive the merge candidate BCW index based on template matching cost after applying ARMC-TM to the merge candidate.

[0150] According to some example embodiments, the VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to derive the BCW index of the merge candidates based on the template matching cost before applying ARMC-TM to the merge candidates. When constructing a merge candidate list, the VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to first derive the BCW index of each bidirectional predictive merge candidate in the merge candidate list based on the template matching cost, and then applies ARMC-TM to reorder the merge candidates. The VVC standard encoder or VVC standard decoder further configures one or more processors of the computing system to use the derived BCW indexes to compute the template matching cost value of the bidirectional predictive merge candidates during ARMC-TM.

[0151] According to some example embodiments, the VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to derive the merge candidate BCW index based on template matching cost after applying ARMC-TM to merge candidates, thereby reducing decoding complexity. The VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to first reorder the merge candidates in the merge candidate list by applying ARMC-TM, and then calculates the template matching cost value of the bidirectional predictive merge candidate using the BCW index inherited from adjacent blocks during ARMC-TM. The VVC standard encoder or VVC standard decoder further configures one or more processors of the computing system to select a merge candidate based on the merge index signaled by a merging mechanism; for the selected bidirectional predictive merge candidate, the VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to derive its BCW index based on the template matching cost. In this way, fewer BCW indices can be derived after applying ARMC-TM compared to deriving the BCW index before applying ARMC-TM.

[0152] The merging candidate BCW indexing technique described herein can be applied to any subset of possible merging patterns. The VVC standard encoder or VVC standard decoder configures one or more processors in the computing system to apply this subset in the encoding CU, according to the VVC standard and ECM specification. As an example, a subset of possible merging patterns includes the standard merging pattern, template matching merging pattern, decoder-side motion vector refinement pattern, MMVD pattern, affine merging pattern, affine plus MMVD merging pattern, and CHP pattern, but does not include other merging patterns.

[0153] As another example, the merge candidate BCW indexing technique described herein is applied to a first subset of possible merge patterns for non-low-latency images and a second subset of possible merge patterns for low-latency images. For example, the first subset of possible merge patterns includes standard merge, template matching merge, decoder-side motion vector thinning, and affine merge, excluding other merge patterns; and the second subset of possible merge patterns includes regular merge, template matching merge, decoder-side motion vector thinning, MMVD, affine merge, and affine plus MMVD merge, excluding other merge patterns.

[0154] As another example, the merge candidate BCW indexing technique described in this paper is applied to a subset of possible merge modes for non-low latency images and low latency images, including standard merge mode, template matching merge mode, decoder-side motion vector thinning mode, CHP mode, and affine merge mode, but excluding other merge modes.

[0155] According to some example embodiments, a VVC standard encoder or VVC standard decoder configures one or more processors of the computing system to derive merge candidate BCW indices based on template matching costs, while inheriting BCW indices from non-adjacent spatial merge candidates. For a current CU encoded using decoder-side motion vector refinement mode, if the current CU does not inherit BCW indices from non-adjacent spatial merge candidates, the BCW indices are always set to equal weights when the current CU inherits motion information from non-adjacent spatial merge candidates. However, BCW indices are inherited from spatially adjacent blocks. Therefore, according to example embodiments of this disclosure, for a current CU using decoder-side motion vector refinement mode, it also inherits BCW indices from non-adjacent spatial merge candidates, thereby aligning the two modes.

[0156] When deriving additional BCW weights based on POC differences (as described above), the BCW index can be derived with or without template matching costs for the merge pattern, as described below.

[0157] According to the embodiment, when the BCW index derived using template matching cost is enabled for the merge mode, the default weight is always assigned to the paired merge candidates and the zero merge candidate, regardless of whether additional BCW weights are enabled.

[0158] According to another implementation, when the inherited weight is an additional BCW weight, the method of deriving the BCW index using template matching cost is not applicable to the merge mode.

[0159] According to another embodiment, when the method of deriving the BCW index using template matching cost is enabled in the merge mode, the additional BCW weights are applied only to the low time layer to further reduce bit overhead.

[0160] Those skilled in the art will understand that all the foregoing aspects of this disclosure can be implemented simultaneously in any combination thereof, and all aspects of this disclosure can be combined to implement yet another embodiment of this disclosure.

[0161] Figure 4 An example system 400 is shown for implementing the process and method of deriving the merged candidate BCW indexes described herein.

[0162] The techniques and mechanisms described in this paper can be implemented by multiple instances of System 400 as well as by any other computing device, system, and / or environment. Figure 4The system 400 shown is merely an example of a system and is not intended to impose any limitation on the scope or functionality of any computing device used to perform the above-described processes and / or procedures. Other well-known computing devices, systems, environments, and / or configurations that may be suitable for use with the embodiments include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, game consoles, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above-described systems or devices, implementations using field-programmable gate arrays (“FPGAs”) and application-specific integrated circuits (“ASICs”), etc.

[0163] System 400 may include one or more processors 402 and system memory 404 communicatively coupled to the one or more processors 402. The one or more processors 402 may execute one or more modules and / or processes to enable the one or more processors 402 to perform various functions. In some embodiments, the one or more processors 402 may include a central processing unit (“CPU”), a graphics processing unit (“GPU”), a combination of a CPU and a GPU, or other processing units or components known in the art. Additionally, each of the one or more processors 402 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0164] Depending on the exact configuration and type of system 400, system memory 404 may be volatile (e.g., RAM), non-volatile (e.g., ROM, flash memory, micro hard disk drive, memory card, etc.), or some combination thereof. System memory 404 may include one or more computer-executable modules 406 executable by the one or more processors 402.

[0165] Module 406 may include, but is not limited to, encoder 408 and decoder 410.

[0166] The encoder 408 may be any, some, or all of the aspects of the exemplary embodiments of this disclosure described above, and may be executed by the one or more processors 402 to configure the one or more processors 402 to perform the operations described above.

[0167] Decoder 410 may be any, some, or all of the aspects of the example embodiments of this disclosure described above, and may be executed by the one or more processors 402 to configure the one or more processors 402 to perform the operations described above.

[0168] In some embodiments, a computer program product is provided, the program product comprising: computer program instructions, and the computer program instructions enabling a computer to perform the steps of the methods described in any embodiment of this disclosure.

[0169] In some embodiments, a computer program is provided that enables a computer to perform the steps of the methods described in any embodiment of this disclosure.

[0170] System 400 may additionally include an input / output (I / O) interface 440 for receiving video source data and bitstream data, and for outputting decoded images to a reference image buffer and / or a display buffer. System 400 may also include a communication module 450 that allows the system 400 to communicate with other devices (not shown) via a network (not shown). The network may include the Internet, wired media such as wired networks or direct wired connections, and wireless media such as acoustic, radio frequency (“RF”), infrared, and other wireless media.

[0171] Some or all of the operations described above can be performed by executing computer-readable instructions stored on a computer-readable storage medium as defined below. The term "computer-readable instructions" as used in this invention includes routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on various system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, and combinations thereof.

[0172] Computer-readable storage media may include volatile memory (such as random access memory (“RAM”)) and / or non-volatile memory (such as read-only memory, “ROM”), flash memory, etc. Computer-readable storage media may also include additional removable and / or non-removable storage, including but not limited to flash memory, magnetic storage, optical storage, and / or magnetic tape storage, which can provide non-volatile storage of computer-readable instructions, data structures, program modules, etc.

[0173] Non-transient or non-temporary computer-readable storage media are examples of computer-readable media. Computer-readable media include at least two types of computer-readable media: computer-readable storage media and communication media. Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented in any process or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, phase-change memory (“PRAM”), static random access memory (“SRAM”), dynamic random access memory (“DRAM”), other types of random access memory (“RAM”), read-only memory (“ROM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory or other memory technologies, optical disc read-only memory (“CD-ROM”), digital versatile disc (“DVD”) or other optical storage, cassette tape, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information for access by a computing device. Conversely, communication media may embody computer-readable instructions, data structures, program modules, or other data in modulated data signals, such as carrier waves or other transmission mechanisms. The computer-readable storage medium used in this document should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (such as optical pulses through fiber optic cables), or electrical signals propagating through wires.

[0174] Computer-readable instructions stored on one or more non-transitory or non-temporary computer-readable storage media, when executed by one or more processors, can perform the above-referenced instructions. Figure 1A-3 The described operations. Typically, computer-readable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific abstract data type. The order of the described operations should not be construed as a limitation, and any number of described operations can be combined in any order and / or in parallel to implement the process.

[0175] Although the subject matter has been described in terms of specific structural features and / or methodological behaviors, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or behaviors described. Rather, specific features and behaviors are disclosed as exemplary forms for implementing the claims.

Claims

1. A computer system applied to a decoder, the computer system comprising: One or more processors; and A computer-readable storage medium communicatively coupled to the one or more processors, the computer-readable storage medium storing computer-readable instructions executable by the one or more processors, the computer-readable instructions performing associated operations when executed by the one or more processors including: For a merging candidate list of a merging coding unit (CU), a CU-level weighted bidirectional prediction (BCW) index of the bidirectional prediction merging candidate is derived based on the template matching cost. The deriving of the BCW index of the bidirectional prediction merging candidate based on the template matching cost includes: calculating the template matching cost of each BCW weight value in a subset of possible BCW weight values, where the possible BCW weight values ​​are similar to inherited BCW weights; calculating the absolute difference between each BCW weight value and the inherited BCW weight; for each BCW weight value whose absolute difference is less than a predetermined threshold, calculating its template matching cost; and selecting the BCW weight value that produces the lowest template matching cost among the calculated template matching costs as the BCW index of the bidirectional prediction merging candidate.

2. The computer system according to claim 1, wherein, The template matching cost includes the sum of the absolute differences between multiple samples of a template of the merged encoding unit and their respective corresponding reference samples.

3. The computer system according to claim 1, wherein, The associated operations also include: The weighted average two-way forecast is calculated according to a formula, which includes: Wherein, P0 and P1 are prediction signals, and P... bi-pred The result is obtained by weighted averaging of P0 and P1, where w is the weight and >> is the bitwise right shift operator.

4. The computer system according to claim 3, wherein, The possible set of BCW weight values ​​includes {6, 7, 8, 9, 10}.

5. The computer system according to claim 1, wherein, The template matching cost is calculated for each BCW weight value in the possible subset of BCW weight values, based on the inherited BCW weights which have negative weights relative to the weighted average bidirectional prediction formula.

6. The computer system according to claim 5, wherein, The weighted average two-way prediction formula includes: Wherein, P0 and P1 are prediction signals, and P... bi-pred The result is obtained by weighted averaging of P0 and P1, where w is the weight and >> is the bitwise right shift operator.

7. The computer system according to claim 1, wherein, The BCW index for the bidirectional prediction merging candidate is derived from the template matching cost, including: Based on an inherited BCW weight, a template matching cost is calculated for each BCW weight value in a possible subset of BCW weight values, said subset being further counted based on the respective image sequence number (POC) distances between a current image and multiple reference images; and The BCW weight value that produces the lowest template matching cost among each calculated template matching cost is selected as the BCW index of the bidirectional prediction merging candidate.

8. The computer system according to claim 7, wherein, The respective POC distances include a first POC distance between the current image and a reference image in a first reference image list, and a second POC distance between the current image and a reference image in a second reference image list.

9. The computer system of claim 8, wherein calculating the template matching cost for each BCW weight value in a possible subset of BCW weight values ​​comprises: When the distance to the first POC is less than the distance to the second POC, calculate the template matching cost for each BCW weight value that is less than or equal to an equal weight for the weighted average bidirectional prediction formula; and If the distance to the first POC is greater than the distance to the second POC, calculate the template matching cost for each BCW weight value that is equal to or greater than the equal weight.

10. The computer system according to claim 9, wherein, The weighted average two-way prediction formula includes: Wherein, P0 and P1 are prediction signals, and P... bi-pred The result is obtained by weighted averaging of P0 and P1, where w is the weight and >> is the bitwise right shift operator.

11. The computer system according to any one of claims 1 to 10, wherein, The associated operations also include: Adjust the template matching cost of the inherited BCW weights so that the template matching cost of the inherited BCW weights is preferentially selected relative to the template matching costs of other BCW weights.

12. The computer system according to claim 11, wherein, Adjusting the template matching cost of the inherited BCW weights includes multiplying the template matching cost of the inherited BCW weights by a weight less than 1.

13. The computer system according to claim 11, wherein, Adjusting the template matching cost of the inherited BCW weights includes applying bitwise shift and subtraction operations to scale the template matching cost of the inherited BCW weights by a factor less than 1.

14. The computer system according to any one of claims 1 to 10, wherein, The associated operations also include: The template matching cost of BCW weights is adjusted, including prioritizing the template matching cost of equal weights for the weighted average bidirectional prediction formula relative to other BCW weights.

15. The computer system according to claim 14, wherein, Adjusting the template matching cost of the BCW weights includes multiplying the template matching cost of equal BCW weights by a weight less than 1.

16. A computer system applied to an encoder, the computer system comprising: One or more processors; and A computer-readable storage medium communicatively coupled to the one or more processors, the computer-readable storage medium storing computer-readable instructions executable by the one or more processors, the computer-readable instructions performing associated operations when executed by the one or more processors including: A merge candidate list of merge coding units (CUs) is constructed. A CU-level weighted bidirectional prediction (BCW) index of the bidirectional prediction merge candidates in the merge candidate list is derived based on the template matching cost. Deriving the BCW index of the bidirectional prediction merge candidates based on the template matching cost includes: calculating the template matching cost of each BCW weight value in a subset of possible BCW weight values, where the possible BCW weight values ​​are similar to inherited BCW weights; calculating the absolute difference between each BCW weight value and the inherited BCW weight; for each BCW weight value whose absolute difference is less than a predetermined threshold, calculating its template matching cost; and selecting the BCW weight value that produces the lowest template matching cost among the calculated template matching costs as the BCW index of the bidirectional prediction merge candidate.

17. A non-transitory computer-readable storage medium storing an instruction set and a bit stream, the bit stream being generated by one or more processors when executing the instruction set, the instruction set being executable by the one or more processors to cause the one or more processors to perform a method, the method comprising: For bidirectional predictive merge candidates in the merge candidate list of a merge coding unit, the BCW index of the bidirectional predictive merge candidate is derived based on the template matching cost. Deriving the BCW index of the bidirectional predictive merge candidate based on the template matching cost includes: calculating the template matching cost of each BCW weight value in a subset of possible BCW weight values, where the possible BCW weight values ​​are similar to inherited BCW weights; calculating the absolute difference between each BCW weight value and the inherited BCW weight; for each BCW weight value whose absolute difference is less than a predetermined threshold, calculating its template matching cost; and selecting the BCW weight value that produces the lowest template matching cost among the calculated template matching costs as the BCW index of the bidirectional predictive merge candidate.

18. A computer program product, the computer program product comprising: A method comprising: computer program instructions, wherein the computer program instructions enable a computer to execute the instructions; the method comprising: For a bidirectional predictive merge candidate of a merge candidate list of a merge coding unit, the BCW index of the bidirectional predictive merge candidate is derived based on the template matching cost. Deriving the BCW index of the bidirectional predictive merge candidate based on the template matching cost includes: calculating the template matching cost of each BCW weight value in a subset of possible BCW weight values, where the possible BCW weight values ​​are similar to inherited BCW weights; calculating the absolute difference between each BCW weight value and the inherited BCW weight; for each BCW weight value whose absolute difference is less than a predetermined threshold, calculating its template matching cost; and selecting the BCW weight value that produces the lowest template matching cost among the calculated template matching costs as the BCW index of the bidirectional predictive merge candidate.

Images