Deriving coding unit level weight bi-prediction index for merge candidates

Abstract

An encoder for the VVC standard and a decoder for the VVC standard are provided to achieve derivation of a BCW index based on a template matching cost value. A template matching cost may be calculated for each value in a set of possible BCW weight values, and a BCW weight value capable of generating the lowest template matching cost in each calculated template matching cost may be selected as a BCW index of a bidirectional predictive merge candidate. Optionally, the matching cost of the template can be calculated for each of the possible subsets of BCW weights according to the inherited BCW weights. Further, when a template matching cost in inherited BCW weights is adjusted from values calculated according to the VVC standard and the ECM specification, or a template matching cost of BCW weights having the same weight is adjusted according to values calculated according to the VVC standard and the ECM specification, a merge candidate BCW index may be derived.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present disclosure claims priority to the following applications: U.S. Provisional Application No. 63 / 358,215, filed on July 4, 2022, entitled “Deriving Coding Unit Level Weighted Bidirectional Prediction Indices for Merge Candidates”, U.S. Provisional Application No. 63 / 403,199, filed on September 1, 2022, entitled “Deriving Coding Unit Level Weighted Bidirectional Prediction Indices for Merge Candidates”, and U.S. Provisional Application No. 18 / 215,753, filed on June 28, 2023, entitled “Deriving Coding Unit Level Weighted Bidirectional Prediction Indices for Merge Candidates”, and the entire contents of all of the foregoing provisional applications are incorporated by reference into this application. Technical Field

[0002] Embodiments of the present disclosure relate to video processing, and more particularly, to a method and system for deriving coding unit-level weighted bidirectional prediction indexes for merge candidates. Background Art

[0003] In 2020, the Joint Video Experts Group (“JVET”) of the ITU-T Video Coding Experts Group (“ITU-T VCEG”) and the ISO / IEC Moving Picture Experts Group (“ISO / IEC MPEG”) released the final draft of the next-generation video codec specification “Versatile Video Coding” (“VVC”). This specification further improves video coding performance compared to existing standards such as H.264 / AVC (Advanced Video Coding) and H.265 / HEVC (High Efficiency Video Coding). In addition, at the time of writing this article, the VVC standard is extended by the latest draft of the Enhanced Compression Model (“ECM”), which was proposed at the 29th meeting of JVET in January 2023 as the “Algorithm Description of Enhanced Compression Model 8 (ECM 8)”.

[0004] Inter-frame prediction can use unidirectional prediction or bidirectional prediction. In unidirectional prediction, only one motion vector pointing to one reference image is used to generate the prediction value of the current block; in bidirectional prediction, two motion vectors pointing to respective reference images are used to generate the prediction value of the current block. According to the earlier HEVC standard, a bidirectional prediction signal is generated by averaging two prediction signals obtained from two different reference images and / or using two different motion vectors. According to the VVC standard, the bidirectional prediction mode is extended based on the simple average, allowing a weighted average of the two prediction signals. For simplicity, this technology is called bidirectional prediction with coding unit ("CU") level weight ("BCW").

[0005] According to decoder-side motion vector refinement ("DMVR"), bidirectional prediction can be performed on the current CU so that the motion information of the current CU includes a weighted average of two prediction signals, and the weight index is inferred from neighboring blocks based on the merge candidate index. For merge candidates that meet the DMVR conditions, the adaptive decoder-side motion vector refinement technique further extends multi-pass DMVR to refine the motion vector in only one of the two bidirectional prediction directions.

[0006] According to the current VVC and ECM specifications, the BCW index of a merged coded block is inherited from a neighboring block according to the signaled merge candidate index. However, the inherited BCW index may not be applicable to the merged coded block. Summary of the invention

[0007] Embodiments of the present disclosure relate to methods and systems for deriving bidirectional prediction indexes with coding unit level weights for deriving merge candidates.

[0008] In a first aspect, an embodiment of the present disclosure provides a computer system, comprising: one or more processors, and a computer-readable storage medium communicatively coupled to the one or more processors, wherein the computer-readable storage medium stores computer-readable instructions executable by the one or more processors, and when the computer-readable instructions are executed by the one or more processors, the associated operations are performed, wherein the associated operations include: For a bi-prediction merging candidate in a merging candidate list of a merging coding unit (CU), a CU-level weighted bi-prediction (BCW) index of the bi-prediction merging candidate is derived according to a template matching cost.

[0009] In a second aspect, an embodiment of the present disclosure provides a non-temporary computer-readable storage medium storing an instruction set, which can be executed by one or more processors of a device to enable the device to perform a method, comprising: for a bidirectional prediction merge candidate in a merge candidate list of a merge-coded coding unit (CU), deriving a CU-level weighted bidirectional prediction (BCW) index of the bidirectional prediction merge candidate based on a template matching cost.

[0010] In a third aspect, an embodiment of the present disclosure provides a computer system comprising: computer program instructions, and computer program instructions enabling a computer to execute a method, the method comprising: for a bidirectional prediction merge candidate in a merge candidate list of a merge-coded coding unit (CU), deriving a CU-level weighted bidirectional prediction (BCW) index of the bidirectional prediction merge candidate based on a template matching cost.

[0011] In a fourth aspect, an embodiment of the present disclosure provides a computer program, and the computer program enables a computer to execute a method, the method comprising: for a bidirectional prediction merge candidate in a merge candidate list of a merge-coded coding unit (CU), deriving a CU-level weighted bidirectional prediction (BCW) index of the bidirectional prediction merge candidate based on a template matching cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The detailed description is given with reference to the accompanying drawings. In the drawings, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. The use of the same reference number in different drawings indicates similar or identical items or features.

[0013] Figure 1A and Figure 1B 1. Block diagrams respectively illustrating an example of a video encoding process and a video decoding process according to an embodiment of the present disclosure;

[0014] Figure 2 shows motion prediction of the current image according to bidirectional prediction;

[0015] Figure 3 A flow chart of a method for deriving a BCW index of merge candidates is shown.

[0016] Figure 4 Example systems and methods for implementing the process described herein for deriving a BCW index of merge candidates are shown. DETAILED DESCRIPTION

[0017] According to the VVC video coding standard ("VVC standard") and the motion prediction described therein, a computing system includes at least one or more processors and a computer-readable storage medium communicatively coupled to the one or more processors. The computer-readable storage medium is a non-transitory or non-temporary computer-readable storage medium, as described in the subsequent Figure 4As shown, it stores computer-readable instructions. At least some of the computer-readable instructions stored on the computer-readable storage medium may be executed by one or more processors of the computing system to configure the one or more processors to perform the associated operations of the computer-readable instructions, including at least the operations of the encoder as described by the VVC standard and the operations of the decoder as described by the VVC standard. Some of these encoder operations and decoder operations based on the VVC standard will be described in further detail later, although these subsequent descriptions should not be understood as exhaustive descriptions of encoder operations and decoder operations according to the VVC standard. Subsequently, the "VVC standard encoder" and "VVC standard decoder" will describe the corresponding computer-readable instructions stored on the computer-readable storage medium, which configure one or more processors to perform these corresponding operations (which may be referred to as a "reference implementation" of an encoder or decoder by way of example).

[0018] In addition, according to an example embodiment of the present disclosure, the VVC standard encoder and the VVC standard decoder also include computer-readable instructions stored on a computer-readable storage medium, and the computer-readable instructions can be executed by one or more processors of a computing system to configure one or more processors to perform operations not specified by the VVC standard. The VVC standard encoder should not be understood as being limited to the operation of the reference implementation of the encoder, but rather includes further computer-readable instructions that configure one or more processors of the computing system to perform further operations described herein. The VVC standard decoder should not be understood as being limited to the operation of the reference implementation of the decoder, but rather includes further computer-readable instructions that configure one or more processors of the computing system to perform further operations as described herein.

[0019] Figure 1A and Figure 1B Example block diagrams of a video encoding process 100 and a video decoding process 150 are respectively shown according to example embodiments of the present disclosure.

[0020] In the encoding process 100, the VVC standard encoder configures one or more processors of the computing system to receive one or more input images as input from an image source 102. The input image includes a certain number of pixels sampled by an image capture device (such as a light sensor array) and includes an uncompressed stream of multiple color channels (such as RGB color channels), wherein the multiple color channels store color data of the original resolution of the image, wherein each channel uses a certain number of bits to store the color data of each pixel of the image. The VVC standard encoder configures one or more processors of the computing system to store the uncompressed color data in a compressed format, wherein the color data is stored at a resolution lower than the original resolution of the image, and is encoded as a luma ("Y") channel and two chroma ("U" and "V") channels with a resolution lower than the luma channel.

[0021] The VVC standard encoder encodes an image (the image being encoded is referred to as a "current image" to distinguish it from any other image received from the image source 102) by configuring one or more processors of the computing system to divide the original image into a plurality of units and sub-units according to the partitioning structure. The VVC standard encoder configures one or more processors of the computing system to subdivide the image into a plurality of macroblocks ("MBs"), each of which has a size of 16×16 pixels, and the macroblocks can be further subdivided into a plurality of partitions. The VVC standard encoder configures one or more processors of the computing system to subdivide the image into a plurality of coding tree units ("CTUs"), the luminance and chrominance components of the CTUs can be further subdivided into a plurality of coding tree blocks ("CTBs"), and the CTBs are further subdivided into a plurality of coding units ("CUs"). Alternatively, the VVC standard encoder configures one or more processors of the computing system to subdivide the image into units of N×N pixels, which can then be further subdivided into sub-units. For the purposes of this disclosure, each of these largest subdivision units of an image can generally be referred to as a "block".

[0022] A CU is coded using one block of luma samples and two corresponding blocks of chroma samples, where the image is not monochrome, and is coded using one coding tree.

[0023] The VVC standard encoder configures one or more processors for the computing system to subdivide the block into partitions with multiples of 4×4 pixels. For example, the partitions of the block can have a size of 8×4 pixels, 4×8 pixels, 8×8 pixels, 16×8 pixels, or 8×16 pixels.

[0024] By encoding the color information of blocks of an image and its subdivisions, rather than encoding the color information of pixels of the full-resolution original image, the VVC standard encoder configures one or more processors of a computing system to encode the color information of the image at a lower resolution than the input image, and stores the color information with fewer bits than the input image.

[0025] In addition, the VVC standard encoder encodes the image by configuring one or more processors of the computing system to perform motion prediction on the block of the current image. Motion prediction encoding refers to storing image data of a block of the current image (wherein the block of the original image before encoding is referred to as an "input block") using motion information and multiple prediction units ("PUs") instead of pixel data according to intra-frame prediction 104 or inter-frame prediction 106.

[0026] Motion information refers to data describing the motion of a block structure of an image or a unit or subunit thereof, such as a motion vector and a reference to a block of a current image or a reference image. A PU may refer to a unit or multiple subunits of a block structure in a plurality of block structures corresponding to an image, such as an MB or a CTU, such as an MB or a CTU, where the blocks are divided based on the image data and encoded according to the VVC standard. The motion information corresponding to a PU may describe a motion prediction encoded by a VVC standard encoder as described herein.

[0027] The VVC standard encoder configures one or more processors of a computing system to encode motion prediction information on each block of an image in a coding order between blocks (e.g., a raster scan order, where the first decoded block is the topmost and leftmost block of the image). The encoded block is referred to as a "current block" to distinguish it from any other block of the same image.

[0028] According to intra prediction 104, one or more processors of the computing system are configured to encode the block by referencing motion information and PUs of one or more other blocks of the same image. According to intra prediction coding, one or more processors of the computing system perform intra prediction 104 (also called spatial prediction) calculations by encoding motion information of the current block based on spatial neighboring samples from spatial neighboring blocks of the current block.

[0029] According to inter-frame prediction 106, one or more processors of the computing system are configured to encode the block by referencing motion information and PUs of one or more other images. The one or more processors of the computing system are configured to store one or more previously encoded and decoded images in a reference image buffer for the purpose of inter-frame prediction encoding; these stored images are referred to as reference images.

[0030] The one or more processors are configured to perform inter-frame prediction 106 (also called temporal prediction or motion compensated prediction) calculation by encoding motion information of the current block based on samples from one or more reference images. The inter-frame prediction can be further calculated according to unidirectional prediction or bidirectional prediction: in unidirectional prediction, only one motion vector pointing to one reference image is used to generate the prediction signal of the current block. In bidirectional prediction, two motion vectors each pointing to a corresponding reference image are used to generate the prediction signal of the current block.

[0031] According to the bidirectional prediction implemented in the early HEVC standard, a bidirectional prediction signal is generated by averaging two prediction signals obtained from two different reference images and / or using two different motion vectors. In contrast, the bidirectional prediction mode implemented in the VVC standard is not limited to simple averaging, but can also perform weighted averaging of the two prediction signals, a technique referred to as weighted ("BCW") bidirectional prediction at the coding unit ("CU") level. The two prediction signals P0 and P1 are weighted averaged according to the following formula 1 to obtain P bi-pred , where >> is the bitwise right shift operator. P bi-pred =((8-w)*P0+w*P1+4)>>3

[0032] Formula 1 is mathematically equivalent to:

[0033] Therefore, depending on the value of the weight w applied to Formula 1, P0 and P1 can be weighted equally or unequally. For weight w=4, P0 and P1 can be weighted equally. Therefore, weight w=4 is an "equal weight" applied to Formula 1 (but not necessarily applied to other weighted average bidirectional prediction formulas in this article). In addition, when applied to Formula 1, a weight w that allows both P0 and P1 to be positively weighted is a "positive weight" relative to Formula 1 (where equal weights are also positive weights), and when applied to Formula 1, a weight w that allows one of P0 and P1 to be negatively weighted is a "negative weight" relative to Formula 1.

[0034] According to the bidirectional prediction mode implemented by the VVC standard, a set of 5 possible weight values, w∈(-2,3,4,5,10) are allowed in the weighted average bidirectional prediction. For each bidirectional prediction CU, the weight w is determined in one of two ways: 1) for non-merge coded CUs, the weight index is signaled after the motion vector difference; 2) for merge coded CUs, the weight index is inferred from neighboring blocks based on the merge candidate index.

[0035] According to the bidirectional prediction mode implemented by the VVC standard, BCW is only applied to CUs with 256 or more luma samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency images, all 5 weights are used. For non-low-latency images, a set of only 3 possible weight values ​​(w∈{3,4,5}) is used.

[0036] The VVC standard encoder configures one or more processors of the computing system to encode the CU so that it contains a reference index to identify one or more prediction signals of the current block for reference by the VVC standard decoder. The motion vector and the reference index are sent to the decoder to identify where the prediction signal of the current block comes from. One or more processors of the computing system can encode the CU to include an inter-frame prediction indicator. The inter-frame prediction indicator indicates list 0 prediction, list 1 prediction, or bidirectional prediction, where list 0 prediction refers to a first reference image list called list 0, list 1 prediction refers to a second reference image list called list 1, and bidirectional prediction simultaneously refers to reference image lists called list 0 and list 1, respectively.

[0037] In the case where the inter-frame prediction indicator indicates list 0 prediction or list 1 prediction, one or more processors of the computing system are configured to encode a CU, the CU including a reference index, the index respectively referring to a reference image in a reference image buffer referenced by list 0 or list 1. In the case where the inter-frame prediction indicator indicates bidirectional prediction, one or more processors of the computing system are configured to encode a CU, the CU including a first reference index referring to a first reference image in a reference image buffer referenced by list 0, and including a second reference index referring to a second reference image in a reference image buffer referenced by list 1.

[0038] The VVC standard encoder configures one or more processors of the computing system to encode each current block of the image separately and output a prediction block for each encoding. According to the VVC standard, a CTU can reach l28×128 luminance samples (depending on the chroma format, plus the corresponding chroma samples). The CTU can be further divided into multiple CUs according to a quadtree, a binary tree, or a ternary tree. One or more processors of the computing system are configured to ultimately record a set of coding parameters at the syntax structure of the leaf nodes of the partition structure, such as the coding mode (intra-frame mode or inter-frame mode), motion information for inter-frame coding blocks (reference index, motion vector, etc.), and quantized residual coefficients.

[0039] After outputting the prediction block, the VVC standard encoder configures one or more processors of the computing system to send a set of encoding parameters such as the encoding mode (i.e., intra-frame or inter-frame prediction), intra-frame prediction mode or inter-frame prediction mode, and motion information to the entropy encoder 124 (described below).

[0040] The VVC standard provides semantics for recording the coding parameter set of a CU. For example, with respect to the above coding parameter set, the pre_mode_flag of the CU is set to 0 for the inter-frame coding block and is set to 1 for the intra-frame coding block; the general_merge_flag of the CU is set to indicate whether the merge mode is used in the inter-frame prediction of the CU; the inter_affine_flag and cu_affine_type_flag of the CU are set to indicate whether affine motion compensation is used in the inter-frame prediction of the CU; and the mvp_10_flag and mvp_11_flag are set to indicate the reference image index in list 0 or list 1, respectively. It should be understood that the VVC standard also includes semantics for recording various other information, flags, and options beyond the scope of the present disclosure.

[0041] The VVC standard encoder further implements one or more mode decision and encoder control settings 108, including rate control settings. One or more processors of the computing system are configured to perform mode decision by selecting an optimized prediction mode for the current block according to a rate-distortion optimization method after intra-frame or inter-frame prediction.

[0042] The rate control setting configures one or more processors of the computing system to assign different quantization parameters ("QPs") to different images. The size of the QP determines the scale at which image information is quantized by the one or more processors during the encoding process (as described below), thereby determining the extent to which encoding process 100 discards image information from a sequence of MBs during the encoding process (due to information loss between step spans).

[0043] The VVC standard encoder also implements a subtractor 110. One or more processors of the computing system are configured to perform a subtraction operation by calculating the difference between the input block and the prediction block. Based on the optimized prediction mode, the prediction block is subtracted from the input block. The difference between the input block and the prediction block is called a prediction residual, or "residual" for short.

[0044] Based on the prediction residual, the VVC standard encoder further implements transformation 112. One or more processors of the computing system are configured to perform a transformation operation on the residual through matrix arithmetic operations to calculate a coefficient array (which may be referred to as "residual coefficients", "transform coefficients", etc.), thereby encoding the current block into a transform block ("TB"). The transform coefficient may be a coefficient representing one of a plurality of spatial transformations, such as diagonal flipping, vertical flipping, or rotation, which may be applied to the sub-block.

[0045] It should be appreciated that coefficients may be stored as two components, absolute value and sign, as described in further detail below.

[0046] The sub-blocks of a CU (e.g., PUs and TBs) can be any combination of sub-block sizes as described above. The VVC standard encoder configures one or more processors of the computing system to subdivide the CU into a residual quadtree ("RQT"), i.e., a hierarchical structure of TBs. The RQT provides the order of motion prediction and residual coding for each level of sub-blocks, and recurses down to each level of the RQT.

[0047] The VVC standard encoder further implements quantization 114. One or more processors of the computing system are configured to perform quantization operations on the residual coefficients through matrix arithmetic operations based on the quantization matrix and QP assigned above. The residual coefficients falling within a certain interval are retained, and the residual coefficients falling outside the interval are discarded.

[0048] The VVC standard encoder further implements inverse quantization 116 and inverse transform 118. One or more processors of the computing system are configured to perform inverse quantization and inverse transform operations on the quantized residual coefficients through inverse matrix arithmetic operations of the above-mentioned quantization and transform operations. The inverse quantization and inverse transform operations produce a reconstructed residual.

[0049] The VVC standard encoder further implements an adder 120. One or more processors of the computing system are configured to perform an addition operation by adding the prediction block and the reconstructed residual, and output a reconstructed block.

[0050] The VVC standard encoder also implements a loop filter 122. One or more processors of the computing system are configured to apply a loop filter, such as a deblocking filter, a sample adaptive offset ("SAO") filter, and an adaptive loop filter ("ALF"), to the reconstructed block, and output a filtered reconstructed block.

[0051] The VVC standard encoder further configures one or more processors of the computing system to output the filtered reconstructed block to a decoded picture buffer ("DPB") 200. As described above, DPB 200 stores the reconstructed picture instead of the current picture, and the reconstructed picture is used as a reference picture by one or more processors of the computing system in encoding the picture, referring to the intra-frame prediction described above.

[0052] The VVC standard encoder also implements an entropy encoder 124. One or more processors of the computing system are configured to perform entropy coding, wherein the signs constituting the quantized residual coefficients are encoded by mapping to binary strings (hereinafter referred to as "bins") according to a context-aware binary arithmetic codec ("CABAC"), which can be transmitted in an output bitstream at a compressed bit rate. The signs of the encoded quantized residual coefficients include the absolute values ​​of the residual coefficients (these absolute values ​​are subsequently referred to as "residual coefficient levels").

[0053] Thus, the entropy encoder configures one or more processors in the computing system to encode residual coefficient levels of a block; bypass encoding residual coefficient signs and record the residual coefficient signs with the encoded block; record encoding parameter sets, such as coding modes, intra-frame prediction modes or inter-frame prediction modes, and motion information encoded in the syntax structure of the encoded block (such as a picture parameter set ("PPS") found in a picture header, and a sequence parameter set ("SPS") found in a sequence of multiple pictures); and output the encoded block.

[0054] The VVC standard encoder configures one or more processors of the computing system to output a coded image consisting of coded blocks from the entropy encoder 124. The coded image is output to a transmission buffer, where it is ultimately packaged into a bitstream for output from the VVC standard encoder. The bitstream is written by one or more processors of the computing system to a non-transitory or non-temporary computer-readable storage medium of the computing system for transmission.

[0055] In the decoding process 150 , the VVC standard decoder configures one or more processors of a computing system to receive as input one or more encoded images from a bitstream.

[0056] The VVC standard decoder implements an entropy decoder 152. One or more processors of the configuration computing system perform entropy decoding, wherein, according to CABAC, bins are decoded by inversely mapping symbols to bins, thereby recovering the entropy-encoded quantized residual coefficients. The entropy decoder 152 outputs quantized residual coefficients, outputs bypass-encoded residual coefficient symbols, and outputs syntax structures such as PPS and SPS.

[0057] The VVC standard encoder also implements inverse quantization 154 and inverse transform 156. One or more processors of the computing system are configured to perform inverse quantization and inverse transform operations on the quantized residual coefficients by matrix arithmetic operations, which are inverse operations of the quantization and transform operations described above. The inverse quantization and inverse transform operations generate a reconstructed residual.

[0058] In addition, based on the coding parameter set recorded in the grammatical structures such as PPS and SPS by the entropy encoder 124 (or optionally, received by out-of-band transmission or encoded to the decoder), and the coding mode contained in the coding parameter set, the VVC standard decoder decides whether to apply intra-frame prediction 156 (i.e., spatial prediction) or motion compensated prediction 158 (i.e., temporal prediction) to the reconstructed residual.

[0059] In the event that the encoding parameter set specifies intra prediction, the VVC standard decoder configures one or more processors of the computing system to use the prediction information specified in the encoding parameter set to perform intra prediction 158. The intra prediction 158 thereby generates a prediction signal.

[0060] Where the coding parameter set specifies inter prediction, the VVC standard decoder configures one or more processors of the computing system to perform motion compensated prediction 160 using reference pictures from the DPB 200. The motion compensated prediction 160 thereby generates a prediction signal.

[0061] The VVC standard decoder further implements an adder 162. The adder 162 configures one or more processors of the computing system to perform an addition operation on the reconstructed residual and the prediction signal, thereby outputting a reconstructed block.

[0062] The VVC standard decoder also implements a loop filter 164. One or more processors of the computing system are configured to apply a loop filter (such as a deblocking filter, an SAO filter, and an ALF) to the reconstructed block, thereby outputting a filtered reconstructed block.

[0063] The VVC standard encoder further configures one or more processors of the computing system to output the filtered reconstructed block to DPB 200. As described above, the reconstructed image stored by DPB 200 is used by one or more processors of the computing system as a reference image in encoding the image, rather than storing the current image, as described above with reference to motion compensated prediction.

[0064] The VVC standard decoder further configures one or more processors of the computing system to output the reconstructed image from the DPB to a user-viewable display of the computing system, such as a television display, a personal computing monitor, a smartphone display, or a flat-panel display.

[0065] Therefore, as shown in the above encoding process 100 and decoding process 150, both the VVC standard encoder and the VVC standard decoder implement motion prediction coding that complies with the VVC specification. The VVC standard encoder and the VVC standard decoder each configure one or more processors of the computing system to generate a reconstructed image based on a previous reconstructed image of the DPB according to the motion compensation prediction described by the VVC standard, wherein the previous reconstructed image is used as a reference image in the motion compensation prediction described herein.

[0066] The VVC standard encoder and the VVC standard decoder are respectively configured with one or more processors of the computing system to predict the motion information of the CU in the reconstructed image through various merge modes, such as the conventional merge mode, the template matching merge mode, the decoder-side motion vector refinement ("DMVR") mode, the motion vector difference ("MMVD") mode, the affine merge mode, the affine plus MMVD merge mode, the inter-frame and intra-frame combined prediction ("CHP") mode, etc. Such a CU with motion information predicted by the merge mode is subsequently referred to as a "merged coded CU". The motion information may include multiple motion vectors. The derivation of motion vectors is known to those skilled in the art and need not be elaborated here.

[0067] The motion information of the CU of the reconstructed image may include a motion candidate list. The motion candidate list may be a data structure containing references to multiple motion candidates. The motion candidate may be a block structure or a subunit thereof, such as a pixel or any other appropriate subdivision of the current image block structure, or may be a reference to a motion candidate of another image. The motion candidate may be a spatial motion candidate or a temporal motion candidate. By applying motion vector compensation ("MVC"), the VVC standard decoder configures one or more processors of the computing system to select a motion candidate from the motion candidate list and derive a motion vector of the motion candidate as the motion vector of the CU in the reconstructed image.

[0068] The motion candidate list may be a merge candidate list and may include up to five types of merge candidates, or may include six types of merge candidates according to the ECM. During encoding of the CU, the merge candidate list may be sent to one or more processors of the computing system via a merge candidate index. For example, the spatial merge candidates of the list may be derived by searching for neighboring blocks of the current CU and adding motion information of those neighboring blocks to the merge candidate list.

[0069] By way of example, the temporal merge candidates of the list may be derived by deriving the motion information of the co-located CU belonging to the co-located reference picture.

[0070] The merge candidate list of the current CU encoded according to the merge mode may include the following merge candidates in order: spatial motion vector prediction ("MVP") candidates from spatially neighboring CUs to the current CU; temporary MVP candidates from the same-position CU in the current CU; history-based MVP ("HMVP") candidates from the FIFO table; pairwise average MVP candidates; and zero motion vector.

[0071] In addition, after constructing the merge candidate list, the merge candidates are reordered according to an adaptive reordering method of merge candidates based on template matching (hereinafter referred to as "ARMC-TM"). Based on template matching, the merge candidates are reordered in ascending order of cost value. For simplicity, the merge candidates in the last but not the first subgroup are not reordered. The template matching cost value of the merge candidate is the sum of absolute differences (SAD) between the template sample of the current block and its corresponding reference sample. The template of the current block includes a set of reconstructed samples adjacent to the current block. The reference sample of the template of the current block is located by the motion information of the merge candidate.

[0072] ARMC-TM can be applied to motion prediction through merge modes, including: normal merge mode; CHP mode; adaptive decoder-side motion vector refinement mode; template matching ("TM") merge mode; and affine merge mode, excluding sub-block based temporal motion vector prediction ("SbTMVP").

[0073] Any merge candidate of the merge candidate list may include motion information predicted by bidirectional prediction. In this case, the reference sample of the merge candidate template is also generated by bidirectional prediction; therefore, bidirectional template matching can be performed based on such merge candidates, which are subsequently referred to as "bidirectional matching ('BM') candidates".

[0074] Figure 2 Motion prediction performed on a current image 202 according to bidirectional prediction is shown. The current image 202 includes a current block 202A, which includes a template 202B. According to bidirectional prediction, reference samples of the template refer to two collocated reference images 204 and 206, one from reference list 0 in a first temporal direction and one from reference list 1 in a second temporal direction. Motion information of the current block 202A refers to collocated reference block 204A of collocated reference image 204, and to collocated reference block 206A of collocated reference image 206. Template 202B of the current block 202A refers to reference samples 204B of collocated reference image 204, and to reference samples 206B of collocated reference image 206.

[0075] According to the adaptive decoder-side motion vector refinement technology, two additional merge modes are provided for merge candidates that meet the DMVR conditions: one merge mode is that the motion vector of the current CU is refined by multi-path DMVR in the first time direction instead of in the second time direction, and the other merge mode is that the motion vector of the current CU is refined by multi-path DMVR in the second time direction instead of in the first time direction.

[0076] For the two additional merge modes, a common merge candidate list is constructed only from the merge candidates that satisfy the DMVR condition.Similar to the merge candidate list described above, merge candidates for the common merge candidate list are obtained from spatially adjacent coding blocks, TMVP, non-adjacent blocks, HMVP and paired candidates.

[0077] If the merge candidate list includes Figure 2 For the weighted average of the prediction signal, such a bi-predictive merge candidate inherits the weight index inferred from the neighboring blocks based on the signaled merge candidate index.

[0078] Given this list of merge candidates, a multi-pass DMVR process is applied to the merge candidates to refine the motion vectors. The DMVR process is modified by measuring the distortion by calculating the motion vector difference ("MVD") between the motion vector and the motion vector prediction ("MVP"). The MVD can be calculated by the mean-averaged sum of absolute differences ("MRSAD"), or if the weights are unequal and the bidirectional prediction is weighted with BCW weights, it can be calculated by the mean-averaged sum of absolute transform differences ("MRSATD"); however, during the first pass (i.e., PU level) of the multi-pass DMVR process, MVD0 (MVD in the first temporal direction) or MVD1 (MVD in the second temporal direction) is set to zero. The merge index is encoded in the same manner as the regular merge mode.

[0079] According to an example embodiment of the present disclosure, additional BCW weights may be derived based on picture order number ("POC") differences.

[0080] As an example, if the two reference images are in the same temporal direction relative to the current image (ie, both are from the past or from the future), and the current image is a low-latency image, then the weight pair (-3, 11) may be added.

[0081] As another example, if two reference images are in the same temporal direction relative to the current image (ie, both are from the past or from the future), and the current image is a non-low-latency image, then a weight pair of (-2, 10) may be added.

[0082] As yet another example, for any other case, a weight pair (2, 6) may be added.

[0083] According to any of the above examples, when the POC distances are the same, the larger value from the weight pair may be assigned to the closest POC reference picture or the list 0 reference picture.

[0084] According to any of the above examples, additional BCW weights may be assigned to the pair-wise merge candidates and the zero-merge candidates, instead of equal weights with respect to the weighted average bi-directional prediction equation.

[0085] As described above, according to the VVC standard and the ECM specification, the BCW index of the merged coding block is inherited from the adjacent block according to the merge candidate index notified by the signal. However, the inherited BCW index may not be applicable to the merged coding block. Therefore, the exemplary embodiment of the present disclosure implements the derivation of the BCW index based on the cost value based on template matching.

[0086] In one or more aspects, exemplary embodiments of the present disclosure derive merge candidate BCW indexes based on template matching costs.

[0087] In one or more aspects, exemplary embodiments of the present disclosure derive merged candidate BCW indexes and a reduced set of possible weight values ​​based on template matching costs.

[0088] In one or more aspects, example embodiments of the present disclosure provide for deriving merged candidate BCW indexes based on template matching costs, and a possible extended set of weight values.

[0089] In one or more aspects, exemplary embodiments of the present disclosure derive a merged candidate BCW index based on template matching costs, and an extended set of possible weight values ​​that improves weight accuracy.

[0090] In one or more aspects, exemplary embodiments of the present disclosure derive merged candidate BCW indexes based on template matching costs while adjusting the template matching costs of inherited BCW weights.

[0091] In one or more aspects, example embodiments of the present disclosure provide for applying bidirectional optical flow to bidirectional prediction merging candidates having equal weights with respect to a weighted average bidirectional prediction formula.

[0092] In one or more aspects, exemplary embodiments of the present disclosure derive a merge candidate BCW index according to a template matching cost before applying ARMC-TM to the merge candidate.

[0093] In one or more aspects, exemplary embodiments of the present disclosure derive merge candidate BCW indexes based on template matching costs after applying ARMC-TM to merge candidates.

[0094] In one or more aspects, example embodiments of the present disclosure derive merge candidate BCW indices based on template matching costs for a subset of possible merge patterns.

[0095] In one or more aspects, example embodiments of the present disclosure provide for deriving merge candidate BCW indexes while inheriting BCW indexes from non-adjacent spatial merge candidates.

[0096] Subsequently, the above-mentioned aspects of example embodiments of the present disclosure are described in further detail.

[0097] Figure 3 A flow chart of a method 300 is shown, in which a VVC standard encoder or a VVC standard decoder configures one or more processors of a computing system to derive a merged candidate BCW index based on a template matching cost while utilizing a set of possible weight values ​​according to the VVC standard and the ECM specification.

[0098] It should be understood that the "decoder side" used here does not mean that the method is implemented only by the decoder; on the contrary, the steps of the method can be implemented by the encoder and the decoder in a similar or identical manner. References to "VVC standard encoder or VVC standard decoder" should be understood as steps being performed in a similar or identical manner by one or more processors of the computing system, regardless of whether the processor is configured by the VVC standard encoder or the VVC standard decoder.

[0099] In step 302, as described above, the VVC standard encoder or the VVC standard decoder configures one or more processors in the computing system to build a merge candidate list for merging the coded CU.

[0100] In step 304, for each merge candidate in the merge candidate list, if the merge candidate is a bidirectional prediction candidate, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to derive the BCW index of the bidirectional prediction merge candidate based on the template matching cost. The bidirectional prediction weight of the merge candidate is determined by the weight index of the bidirectional prediction merge candidate (hereinafter referred to as "BCW index"). The VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to derive the BCW index in sub-steps 304A, 304B, 304C, 304D and 304E to be described later.

[0101] In step 304A, for each bidirectional prediction weight (i.e., BCW index), the VVC standard encoder or VVC standard decoder configures one or more processors to calculate the template matching cost using the SAD between the template sample of the merged coded CU and the respective corresponding reference samples. (It should be understood that according to the bidirectional prediction under the VVC standard, the template includes the reconstructed samples adjacent to the left and / or top of the merged coded CU, such as Figure 2 202B. )

[0102] In step 304B, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to generate reference samples of the template by bidirectional prediction using corresponding bidirectional prediction weight values.

[0103] At step 304C, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to calculate a template matching cost for each of a set of possible BCW weight values, and selects the BCW weight value that produces the lowest template matching cost among each calculated template matching cost as the BCW index of the bidirectional prediction merging candidate. According to example embodiments of the present disclosure, various examples of performing step 304C are described below.

[0104] For example, for a bi-directional prediction merging candidate in a non-low-latency image, three template matching cost values ​​are calculated for each BCW weight value in a set of three possible BCW weight values, which represent three different weight values ​​(w∈€{3,4,5}, where w is defined as above with reference to Formula 1).

[0105] Therefore, among the three template matching costs calculated according to the three possible weight values, the VVC standard encoder or the VVC standard decoder configures one or more processors in the computing system to determine the weight value that produces the lowest template matching cost as the BCW index of the bidirectional prediction merging candidate.

[0106] According to example step 304C, a possible set of weight values ​​corresponds to a possible set of weight values ​​in the weighted average bidirectional prediction provided by the VVC standard and / or the ECM specification. That is, for low-delay images, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to calculate the template matching cost from 5 possible weight values ​​(w∈{-2,3,4,5,10}), and for non-low-delay images, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to calculate the template matching cost from 3 possible weight values ​​(w€{3,4,5}). In addition, according to the above formula 1, the two prediction signals P0 and P1 are weighted averaged to obtain P bi-pred .

[0107] According to example step 304C, a set of possible weight values ​​is a reduced set of possible weight values ​​compared to the possible weight values ​​in the weighted average bidirectional prediction provided by the VVC standard and / or the ECM specification. For low-delay images and non-low-delay images, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to calculate the template matching cost from a set of 3 possible weight values ​​(w∈(3,4,5}); that is, the negative weights of Formula 1 are excluded. In addition, according to the above Formula 1, the weighted average of the two prediction signals P0 and P1 is calculated as P bi-pred .

[0108] According to example step 304C, a set of possible weight values ​​is an extended set of possible weight values ​​compared to the possible weight values ​​in the weighted average bidirectional prediction provided by the VVC standard and / or the ECM specification. For example, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to calculate the template matching cost from a set of 7 possible weight values ​​(w∈{1,2,3,4,5,6,7}) for low-delay images and non-low-delay images. This can be achieved by removing the bit overhead used to signal the BCW index in the CU encoding. According to the above formula 1, the two prediction signals P0 and P1 are weighted averaged to obtain P bi-pred .

[0109] According to a further example of step 304C, a possible set of weight values ​​is an extended set of possible weight values ​​compared to the possible weight values ​​in the weighted average bidirectional prediction provided by the VVC standard and / or the ECM specification. For example, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to calculate the template matching cost of the low-delay image and the non-low-delay image from a set of 5 possible weight values ​​(w∈(6,7,8,9,10}). In addition, according to the following formula 2, the weighted average of the two prediction signals P0 and P1 is calculated as P bi-pred , where >> is the bitwise right shift operator. P bi-pred =((16-w)*P0+w*P1+8)>>4

[0110] Formula 2 is mathematically equivalent to:

[0111] Therefore, depending on the value of the weight w applied to Formula 2, P0 and P1 can be weighted equally or unequally. For weight w=8, the weights of P0 and P1 are equal. Therefore, weight w=8 is an "equal weight" that applies to Formula 2 (but not necessarily to the other weighted average bidirectional prediction formulas here). In addition, when applied to Formula 2, a weight w that causes both P0 and P1 to be positively weighted is a "positive weight" relative to Formula 2 (where equal weights are also positive weights), and a weight w that causes one of P0 and P1 to be negatively weighted is a "negative weight" when applied to Formula 2.

[0112] According to a further exemplary embodiment of the present disclosure, in step 304D, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to calculate the template matching cost of each of a possible BCW weight value subset based on the inherited BCW weight value, and selects the BCW weight value that produces the lowest template matching cost in each calculated template matching cost as the BCW index of the bidirectional prediction merging candidate. According to an exemplary embodiment of the present disclosure, various examples of performing step 304D are described subsequently.

[0113] According to the example of step 304D, the VVC standard encoder or the VVC standard encoder-decoder configures one or more processors in the computing system to calculate the template matching cost for each of a possible subset of BCW weight values ​​based on the inherited BCW weight value, where the inherited BCW weight is a positive weight relative to the weighted average bidirectional prediction formula (i.e., w in Formula 1 and Formula 2). For positive inherited weights, the subset of possible weight values ​​for calculating the template matching cost only includes positive weights relative to the weighted average bidirectional prediction formula (for the above Formula 1 and Formula 2, the determination of positive weights is different). Note that for positive weight w, it can be concluded that the weight of P0 (i.e., w is 8 for Formula 1, or w is 16 for Formula 2) and the weight of P1 (i.e., w) are both greater than 0.

[0114] As an example, a possible set of weight values ​​with respect to Formula 1 is determined to be (-2, 3, 4, 5, 10}. For one of the positive inherited weights (i.e., with respect to Formula 1, the inherited weight is {3, 4, 5}), the template matching cost of {3, 4, 5} is calculated according to step 304D without calculating the template matching cost of {-2, 10}. For non-positive inherited weights, as in step 304C, the template matching cost of {-2, 3, 4, 5, 10} is calculated.

[0115] According to the example of step 304D, the VVC standard encoder or the VVC standard encoder decoder configures one or more processors in the computing system to calculate the template matching cost for each of the subset of possible BCW weight values ​​based on the inherited BCW weight, wherein the inherited BCW weight is a negative weight relative to the weighted average bidirectional prediction. For negative inherited weights, the subset of possible weight values ​​for which the template matching cost is calculated only includes negative weights with respect to the weighted average bidirectional prediction formula ((for the above formula 1 and formula 2, the determination of negative weights is different). For negative weights w, it follows that at least one of the weights of P0 and P1 is less than 0.

[0116] As an example, a possible set of weight values ​​is determined to be {-2, 3, 4, 5, 10} according to one or more examples of step 304C above. For a negative inherited weight (for example, for one of {-2, 10} in formula 1), the template matching cost of {-2, 10} is calculated according to step 304D.

[0117] According to a further embodiment extending the foregoing example, even for negative inherited weights, equal weights with respect to the weighted average bi-prediction formula are included in the subset of possible BCW weight values ​​for computing the template matching cost.

[0118] As an example, a possible set of weight values ​​is determined to be {-2, 3, 4, 5, 10} according to one or more examples of step 304C above. When the inherited weight is negative (i.e., with respect to one of {-2, 10} in Formula 1), the template matching cost of {-2, 4, 10} is calculated according to step 304D (because 4 is an equal weight for Formula 1, although 4 is not a negative weight for Formula 1).

[0119] According to the example of step 304D, the VVC standard encoder or the VVC standard encoder-decoder configures one or more processors in the computing system to calculate the template matching cost for each of the subset of possible BCW weight values, and the possible BCW weight values ​​are similar to the inherited BCW weights. When the absolute difference between the weight w and the inherited weight is less than a predefined threshold, the weight w is similar to the inherited weight. The predefined threshold is a non-zero positive integer.

[0120] As an example, a possible set of weight values ​​determined according to one or more examples of step 304C is {-2, 3, 4, 5, 10} and the predefined threshold is set to 2. Therefore, for the inherited weight 4, the template matching cost of (3, 4, 5} is calculated according to step 304D. For the inherited weight 3, the template matching cost of {3, 4} is calculated according to step 304D. When the inherited weight is -2, the template matching cost of {-2} is calculated according to step 304D.

[0121] By way of another example, a possible set of weight values ​​is determined to be {-4, -3, -2, -1, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12}, and the predefined threshold is set to 2. Therefore, for an inherited weight of 4, the template matching cost of {3, 4, 5} is calculated according to step 304D. For an inherited weight of 3, the template matching cost of {2, 3, 4} is calculated according to step 304D. For an inherited weight of -2, the template matching cost of {-3, -2, -1} is calculated according to step 304D.

[0122] As another example, according to one or more examples of step 304C above, a possible set of weight values ​​is determined to be {1, 2, 3, 4, 5, 6, 7}, and the predefined threshold is set to 2.

[0123] As another example, according to one or more examples of step 304C above, a group of possible weight values ​​is determined to be {0, 1, 2, 3, 4, 5, 6, 7, 8}, and the predefined threshold is set to 2.

[0124] According to the example of step 304D, the VVC standard encoder or the VVC standard encoder-decoder configures one or more processors in the computing system to calculate a template matching cost for each of a subset of possible BCW weight values ​​based on the inherited BCW weights, the subset being further based on a POC distance between the current image and the reference image.

[0125] As an example, when the template matching method is used to derive the BCW index for the bidirectional prediction merge candidate, the POC distance between the current image and the reference image from the reference image list 0 (hereinafter referred to as Poc_diff_L0), and the POC distance between the current image and the reference image from the reference image list 1 (hereinafter referred to as Poc_diff_L1) are calculated according to step 304D.

[0126] For Poc_diff_L0 equal to Poc_diff_L1, the template matching cost of all weights in a possible weight value set is calculated according to step 304C. For Poc_diff_L0 less than Poc_diff_L1, the weighted template matching cost less than or equal to the equal weight in the weighted average bidirectional prediction formula is calculated according to step 304D. That is, the weight applied to P1 (i.e., w) is less than or equal to the weight applied to P0 (i.e., w is 8 in Formula 1, or w is 16 in Formula 2), where P1 is the prediction signal using the reference image from reference image list 1, and P0 is the prediction signal using the reference image from reference image list 0. On the other hand, for Poc_diff_L0 greater than Poc_diff_L1, the template matching cost equal to or greater than the equal weight in the weighted average bidirectional prediction formula is calculated according to step 304D. That is, the weight applied to P1 is greater than or equal to the weight applied to P0.

[0127] As an example, a possible set of weight values ​​is determined to be {-2, 3, 4, 5, 10} according to one or more examples of step 304C above. For Poc_diff_L0 equal to Poc_diff_L1, the template matching cost of {-2, 3, 4, 5, 10) is calculated according to step 403C. For Poc_diff_L0 less than Poc_diff_L1, the template matching cost of {-2, 3, 4} is calculated according to step 304D. For Poc_diff_L0 greater than Poc_diff_L1, the template matching cost of {4, 5, 10} is calculated according to step 304D.

[0128] According to the example of step 304D, the VVC standard encoder or the VVC standard encoder-decoder configures one or more processors in the computing system to calculate a template matching cost for each of a subset of possible BCW weight values ​​similar to the inherited BCW weight, wherein the subset is further based on a POC distance between the current image and the reference image.

[0129] When the template matching method is used to derive the BCW index of the bidirectional prediction merge candidate, in addition to Poc_diff_L0 and Poc_diff_L1, it is necessary to subtract the inherited weight value from each value in a possible BCW weight value set to calculate the value diff_w. For Poc_diff_L0 equal to Poc_diff_L1, the template matching cost of those weights whose absolute value of diff_w is less than a first predefined threshold is calculated. For Poc_diff_L0 less than Poc_diff_L1, the template matching cost of those weights whose diff_w is less than or equal to 0 and whose absolute value of diff_w is less than a second predefined threshold is calculated. For Poc_diff_L0 greater than Poc_diff_L1, the template matching cost of those weights whose diff_w is greater than or equal to 0 and whose absolute value of diff_w is less than a third predefined threshold is calculated. The first, second and third predefined thresholds may be the same or different from each other.

[0130] As an example, a set of possible weight values ​​is determined to be {1, 2, 3, 4, 5, 6, 7} according to one or more examples of step 304C above. The first, second and third predefined thresholds are set to 2, 3 and 3 respectively. Assume that the inherited weight is 4.

[0131] Therefore, for Poc_diff_L0 equal to Poc_diff_L1, the template matching cost of {3,4,5} is calculated according to step 304D. For Poc_diff_L0 less than Poc_diff_L1, the template matching cost of {2,3,4} is calculated according to step 304D. For Poc_diff_L0 greater than Poc_diff_L1, the template matching cost of {4,5,6} is calculated according to step 304D.

[0132] According to another embodiment, when the BCW index of the bidirectional prediction merge candidate is derived using the template matching method, the absolute difference between Poc_diff_L0, Poc_diff_L1 and each value in a possible BCW weight value set and the inherited weight is calculated. For Poc_diff_L0 equal to Poc_diff_L1, the template matching cost of those weights whose absolute difference is less than a first predefined threshold is calculated. For Poc_diff_L0 not equal to Poc_diff_L1, the template matching cost of those weights whose absolute difference is less than a second predefined threshold is calculated. The first predefined threshold is different from the second predefined threshold.

[0133] As an example, a group of possible weight values ​​is determined to be {1, 2, 3, 4, 5, 6, 7} according to one or more examples of step 304C above. The first and second predefined thresholds are set to 3 and 2, respectively. Assume that the inherited weight is 3.

[0134] Therefore, if Poc_diff_L0 is equal to Poc_diff_L1, the template matching cost of {1, 2, 3, 4, 5} is calculated according to step 304D. If Poc_diff_L0 is not equal to Poc_diff_L1, the template matching cost of {2, 3, 4} is calculated according to step 304D.

[0135] In addition, at step 304E, the VVC standard encoder or decoder configures one or more processors of the computing system to derive a merged candidate BCW index based on the template matching cost, and adjust the template matching cost of the inherited BCW weight according to the value calculated by the VVC standard and the ECM specification. As described above, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to calculate the template matching cost for each bidirectional prediction weight (i.e., BCW index) using the SAD between multiple samples of a template of the merged coded CU and their respective corresponding reference samples. However, since the inherited BCW index from the neighboring block may have a higher accuracy than another BCW index, the VVC standard encoder or the VVC standard decoder further configures one or more processors of the computing system to adjust the template matching cost of the inherited BCW index so that it is selected in priority to the template matching cost of other BCW indexes.

[0136] As an example, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to multiply the template matching cost (TMcost) of the inherited BCW index by a weight less than 1. Therefore, the inherited BCW index has a higher probability of being selected. In order to reduce computational complexity, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to perform bitwise shift operations and subtraction operations to avoid multiplication operations. For example, the weight of the template matching cost applied to the inherited BCW index is set to 0.90625, which can be replaced by TMcost-(TMcost>>4)-(TMcos>>5).

[0137] By way of another example, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to subtract an offset value from the template matching cost of the inherited BCW index. The offset value is a non-zero positive number determined according to a quantization parameter ("QP"). For example, the offset value may be set to a Lagrangian multiplier λ.

[0138] Weights that are not similar to the inherited weights may be less promising, given the higher precision of the inherited weights. According to another embodiment, the template matching cost of the weights that are not similar to the inherited weights is multiplied by a value greater than 1. The weight is determined to be not similar to the inherited weight based on the absolute difference between the weight and the inherited weight being greater than a pre-set threshold. As described above, weights that are not similar to the inherited weights may be excluded from calculating the template matching cost.

[0139] As an example, the predefined threshold is set to 1, and a set of possible weight values ​​is determined according to one or more examples of step 304C above: {-2, 3, 4, 5, 10}. Assume that the inherited weight is 3. The template matching cost TMcost of {-2, 5, 10} is multiplied by 1.09375, which can be replaced by TMcost+(TMcost>>4)+(TMcost>>5).

[0140] According to an exemplary embodiment of the present disclosure, a VVC standard encoder or a VVC standard decoder configures one or more processors of a computing system to apply bidirectional optical flow ("BDOF") to bidirectional prediction blocks with equal weights. Through the BDOF technology, the prediction samples of the bidirectional prediction blocks are refined.

[0141] In addition, at step 304F, optionally, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to adjust the template matching cost of the BCW weights with the same weight according to the value calculated by the VVC standard and the ECM specification. Since the application of BDOF may be beneficial to the prediction sample, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to adjust the template matching cost of the BCW index having equal weight to the weighted average bidirectional prediction formula, and the template matching cost of other BCW indexes is selected in priority. As an example, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to multiply the template matching cost of the BCW index having equal weight to the weighted average bidirectional prediction formula by a weight less than 1, or subtract a positive value from the template matching cost of the BCW index having equal weight to the weighted average bidirectional prediction formula.

[0142] Alternatively, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to apply BDOF to all bidirectionally predicted blocks regardless of their respective merge candidate BCW indexes.

[0143] In addition, according to an exemplary embodiment of the present disclosure, in step 304, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to derive the merge candidate BCW index according to the template matching cost before applying ARMC-TM to the merge candidate, or derive the merge candidate BCW index according to the template matching cost after applying ARMC-TM to the merge candidate.

[0144] According to some example embodiments, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to derive a BCW index of the merge candidate based on the template matching cost before applying ARMC-TM to the merge candidate. When constructing a merge candidate list, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to first derive a BCW index of each bidirectional prediction merge candidate in the merge candidate list based on the template matching cost, and then apply ARMC-TM to reorder the merge candidates. The VVC standard encoder or the VVC standard decoder further configures one or more processors of the computing system to use the derived BCW index to calculate the template matching cost value of the bidirectional prediction merge candidate during ARMC-TM.

[0145] According to some example embodiments, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to derive the BCW index of the merge candidate according to the template matching cost after applying ARMC-TM to the merge candidate, thereby reducing decoding complexity. The VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to first reorder the merge candidates in the merge candidate list by applying ARMC-TM, and then calculates the template matching cost value of the bidirectional prediction merge candidate using the BCW index inherited from the neighboring block during ARMC-TM. The VVC standard encoder or the VVC standard decoder further configures one or more processors of the computing system, and then selects the merge candidate according to the merge index notified by the signal; for the selected bidirectional prediction merge candidate, the VVC standard encoder or the VVC standard decoder configures one or more processors of the computing system to derive its BCW index according to the template matching cost. In this way, fewer BCW indexes can be derived after applying ARMC-TM than when the BCW index is derived before applying ARMC-TM.

[0146] The merge candidate BCW indexing technique as described herein can be applied to any subset of possible merge modes, and the VVC standard encoder or VVC standard decoder configures one or more processors in the computing system to apply the subset in the encoded CU according to the VVC standard and the ECM specification. As an example, a subset of possible merge modes includes standard merge mode, template matching merge mode, decoder side motion vector refinement mode, MMVD mode, affine merge mode, affine plus MMVD merge mode, and CHP mode, and does not include other merge modes.

[0147] By way of another example, the merge candidate BCW indexing technique as described herein is applied to a first subset of possible merge modes for non-low-latency images, and a second subset of possible merge modes for low-latency images. For example, the first subset of possible merge modes includes a standard merge mode, a template matching merge mode, a decoder-side motion vector refinement mode, and an affine merge mode, excluding other merge modes; and the second subset of possible merge modes includes a regular merge mode, a template matching merge mode, a decoder-side motion vector refinement mode, an MMVD mode, an affine merge mode, and an affine plus MMVD merge mode, excluding other merge modes.

[0148] As another example, the merge candidate BCW indexing technique described in this article is applied to a subset of possible merge modes for non-low-latency images and low-latency images, including standard merge mode, template matching merge mode, decoding-side motion vector refinement mode, CHP mode, and affine merge mode, but excluding other merge modes.

[0149] According to some example embodiments, a VVC standard encoder or a VVC standard decoder configures one or more processors of a computing system to derive a merge candidate BCW index based on a template matching cost while inheriting the BCW index from a non-adjacent spatial merge candidate. For a current CU encoded using a decoder-side motion vector refinement mode, if the current CU does not inherit the BCW index of a non-adjacent spatial merge candidate, the BCW index is always set to equal weight when the current CU inherits motion information from a non-adjacent spatial merge candidate. However, the BCW index is inherited from a spatially adjacent block. Therefore, according to an example embodiment of the present disclosure, for a current CU using a decoder-side motion vector refinement mode, it also inherits the BCW index from a non-adjacent spatial merge candidate, thereby aligning the two modes.

[0150] When additional BCW weights are derived based on POC differences (described above), the BCW index may be derived with or without template matching costs for merge mode, as described below.

[0151] According to an embodiment, when BCW index derived using template matching cost is enabled for merge mode, default weights are always assigned to pairs of merge candidates and zero merge candidates, regardless of whether additional BCW weights are enabled.

[0152] According to another embodiment, when the inherited weight is an additional BCW weight, the method of deriving the BCW index using the template matching cost is not applicable to the merge mode.

[0153] According to yet another embodiment, when the method of deriving BCW index using template matching cost is enabled for merge mode, additional BCW weights are only applied to the low temporal layer to further reduce the bit overhead.

[0154] Those skilled in the art will understand that all of the above aspects of the present disclosure may be implemented simultaneously in any combination thereof, and all aspects of the present disclosure may be implemented in combination as yet another embodiment of the present disclosure.

[0155] Figure 4 An example system 400 is shown for implementing the processes and methods described herein for derivation of merge candidate BCW indexes.

[0156] The techniques and mechanisms described herein may be implemented by multiple instances of the system 400, as well as by any other computing devices, systems, and / or environments. Figure 4The system 400 shown is only one example of a system, and is not intended to impose any limitation on the scope of use or functionality of any computing device used to perform the above-described processes and / or processes. Other well-known computing devices, systems, environments, and / or configurations that may be suitable for use with the embodiments include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, game consoles, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, implementations using field programmable gate arrays ("FPGAs") and application specific integrated circuits ("ASICs"), and the like.

[0157] The system 400 may include one or more processors 402 and a system memory 404 communicatively coupled to the one or more processors 402. The one or more processors 402 may execute one or more modules and / or processes to enable the one or more processors 402 to perform various functions. In some embodiments, the one or more processors 402 may include: a central processing unit ("CPU"), a graphics processing unit ("GPU"), a CPU and a GPU, or other processing units or components known in the art. In addition, each of the one or more processors 402 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0158] The system memory 404 may be volatile (e.g., RAM), non-volatile (such as ROM, flash memory, micro hard drive, memory card, etc.), or some combination thereof, depending on the exact configuration and type of system 400. The system memory 404 may include one or more computer executable modules 406 that may be executed by the one or more processors 402.

[0159] Module 406 may include, but is not limited to, an encoder 408 and a decoder 410 .

[0160] The encoder 408 may be a VVC standard encoder that implements any, some or all aspects of the exemplary embodiments of the present disclosure described above, and may be executed by the one or more processors 402 to configure the one or more processors 402 to perform the above operations.

[0161] The decoder 410 may be a VVC standard encoder implementing any, some or all aspects of the exemplary embodiments of the present disclosure described above, which may be executed by the one or more processors 402 to configure the one or more processors 402 to perform the operations described above.

[0162] In some embodiments, a computer program product is provided, the program product comprising: computer program instructions, and the computer program instructions enable a computer to execute the steps of the method described in any embodiment of the present disclosure.

[0163] In some embodiments, a computer program is provided, which enables a computer to execute the steps of the method described in any embodiment of the present disclosure.

[0164] The system 400 may additionally include an input / output (I / O) interface 440 for receiving video source data and bitstream data, and for outputting decoded images to a reference image buffer and / or a display buffer. The system 400 may also include a communication module 450 that allows the system 400 to communicate with other devices (not shown) over a network (not shown). The network may include the Internet, wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, radio frequency ("RF"), infrared, and others.

[0165] Some or all of the operations of the above method can be performed by executing computer-readable instructions defined as follows stored on a computer-readable memory. The term "computer-readable instructions" used in the present invention includes routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on various system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, and combinations thereof.

[0166] Computer-readable storage media may include volatile memory (such as random access memory ("RAM")) and / or non-volatile memory (such as read-only memory ("ROM"), flash memory, etc.). Computer-readable storage media may also include additional removable storage and / or non-removable storage, including but not limited to flash memory, magnetic storage, optical storage, and / or tape storage, which may provide non-volatile storage of computer-readable instructions, data structures, program modules, etc.

[0167] Non-transient or non-temporary computer-readable storage media are examples of computer-readable media. Computer-readable media include at least two types of computer-readable media, namely computer-readable storage media and communication media. Computer-readable storage media include volatile and non-volatile, removable and non-removable media, which are implemented in any process or technology for storing information, such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media include, but are not limited to, phase change memory ("PRAM"), static random access memory ("SRAM"), dynamic random access memory ("DRAM"), other types of random access memory ("RAM"), read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), flash memory or other memory technology, compact disc read-only memory ("CD-ROM"), digital versatile disk ("DVD") or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. In contrast, communication media can embody computer-readable instructions, data structures, program modules or other data in modulated data signals, such as carrier waves or other transmission mechanisms. The computer-readable storage media employed herein should not be interpreted as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (such as light pulses through fiber optic cables), or electrical signals propagating through wires.

[0168] The computer readable instructions stored on one or more non-transitory or non-temporary computer readable storage media, when executed by one or more processors, may perform the above-referenced Figure 1A-3 Typically, computer-readable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described should not be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0169] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. A computer system comprising: one or more processors; and A computer-readable storage medium communicatively coupled to the one or more processors, the computer-readable storage medium storing computer-readable instructions executable by the one or more processors, the computer-readable instructions, when executed by the one or more processors, performing associated operations including: For a bi-prediction merging candidate in a merging candidate list of a merging coding unit (CU), a CU-level weighted bi-prediction (BCW) index of the bi-prediction merging candidate is derived according to a template matching cost.

2. The computer system according to claim 1, wherein: The template matching cost includes the sum of absolute differences between a plurality of samples of a template of the merged coded CU and their respective corresponding reference samples.

3. The computing system of claim 1, wherein: Deriving the BCW index of the bidirectional prediction merging candidate according to the template matching cost includes: calculating a template matching cost for each BCW weight value in a set of possible BCW weight values; and The BCW weight value that generates the lowest template matching cost among each calculated template matching cost is selected as the BCW index of the bidirectional prediction merging candidate.

4. The computer system according to claim 3, wherein: The associated operations also include: The weighted average bidirectional prediction is calculated according to a formula, the formula comprising: P bi-pred =((16-w)*P0+w*P1+8)>>4 5. The computer system according to claim 4, wherein: The possible BCW weight value set includes {6, 7, 8, 9, 10}.

6. The computer system according to claim 1, wherein: Deriving the BCW index of the bidirectional prediction merging candidate according to the template matching cost includes: calculating a template matching cost for each BCW weight value in a subset of possible BCW weight values ​​based on the inherited BCW weight; The BCW weight value that generates the lowest template matching cost among each calculated template matching cost is selected as the BCW index of the bidirectional prediction merging candidate.

7. The computer system according to claim 6, wherein: The template matching cost is calculated for each BCW weight value in a subset of possible BCW weight values ​​based on the inherited BCW weight being a negative weight relative to a weighted average bi-directional prediction formula.

8. The computer system according to claim 7, wherein: The weighted average bidirectional prediction equation includes: P bi-pred =((8-w)*P0+w*P1+4)>>3 9. The computer system according to claim 1, wherein: Deriving the BCW index of the bidirectional prediction merging candidate according to the template matching cost includes: Based on an inherited BCW weight, calculating a template matching cost for each BCW weight value in a subset of possible BCW weight values, the subset further counted based on respective picture order number (POC) distances between a current image and a plurality of reference images; and The BCW weight value that generates the lowest template matching cost among each calculated template matching cost is selected as the BCW index of the bi-directional prediction merging candidate.

10. The computer system according to claim 9, wherein: The respective POC distances include a first POC distance between the current image and a reference image of a first reference image list, and a second POC distance between the current image and a reference image of a second reference image list.

11. The computer system of claim 10, wherein calculating the template matching cost for each BCW weight value in a subset of possible BCW weight values ​​comprises: In the case where the first POC distance is less than the second POC distance, calculating, for the weighted average bidirectional prediction formula, a template matching cost for each BCW weight value that is less than or equal to an equal weight; and In a case where the first POC distance is greater than the second POC distance, a template matching cost for each BCW weight value that is equal to or greater than the equal weight is calculated.

12. The computer system according to claim 11, wherein: The weighted average bidirectional prediction formula includes: P bi-pred =((8-w)*P0+w*P1+4)>>3 13. The computer system according to claim 11, wherein: Deriving the BCW index of the bidirectional prediction merging candidate according to the template matching cost includes: calculating a template matching cost for each BCW weight value in a subset of possible BCW weight values ​​that is similar to the inherited BCW weight; The calculated BCM weight value that produces the lowest template matching cost in each of the template matching costs is selected as the BCW index of the bidirectional prediction merging candidate.

14. The computer system of claim 13, wherein the template matching cost is calculated for each BCW weight value in a subset of possible BCW weight values; calculating a respective absolute difference between each BCW weight value and the inherited BCW weight; For each BCW weight value whose absolute difference is less than a predetermined threshold, the template matching cost is calculated.

15. The computer system according to any one of claims 1 to 14, wherein: The associated operations also include: The template matching cost of adjusting the inherited BCW weight is selected in preference to the template matching cost of other BCW weights.

16. The computer system of claim 15, wherein: Adjusting the template matching cost of the inherited BCW weight includes multiplying the template matching cost of the inherited BCW weight by a weight less than one.

17. The computer system according to claim 15, wherein: Adjusting the template matching cost of the inherited BCW weights includes applying a bitwise shift operation and a subtraction operation to scale the template matching cost of the inherited BCW weights by a factor less than one.

18. The computer system according to any one of claims 1 to 14, wherein: The associated operations also include: Adjusting the template matching costs of the BCW weights includes prioritizing template matching costs of other BCW weights over equal weights for the weighted average bi-directional prediction formula.

19. The computer system according to claim 18, wherein: Adjusting the template matching cost of the BCW weight includes multiplying the template matching cost of equal BCW weight by a weight less than one.

20. A non-transitory computer-readable storage medium, wherein: The computer-readable storage medium stores a set of instructions, which can be executed by one or more processors of a device to cause the device to perform a method, the method comprising: For a bi-predictive merging candidate in a merging candidate list of a merge-coded CU, a BCW index of the bi-predictive merging candidate is derived according to a template matching cost.

21. A computer program system comprising: Computer program instructions, and the computer program instructions enable a computer to perform a method, the method comprising: For a bi-predictive merging candidate in a merging candidate list of a merge-coded CU, a BCW index of the bi-predictive merging candidate is derived based on a template matching cost.

22. A computer program, wherein the computer program enables a computer to perform a method comprising: For a bi-predictive merging candidate in a merging candidate list of a merge-coded CU, a BCW index of the bi-predictive merging candidate is derived according to a template matching cost.

Images