Method and apparatus for inter prediction using template matching in video coding system

Abstract

The invention discloses a method and a device for determining an inter-frame direction according to template matching cost. An L0 matching cost between a first template corresponding to a first neighborhood region of a first reference block and a current template corresponding to a current neighborhood region of a current block is determined. An L1 matching cost is determined between a second template corresponding to a neighboring area of the second reference block and the current template. An inter-frame direction of an MVP candidate of a current block is determined based on first information including an L0 matching cost and an L1 matching cost, wherein the inter-frame direction corresponds to a bi-directional prediction, an L0 uni-directional prediction, or an L1 uni-directional prediction. The MVP candidates are inserted into an AMVP (adaptive MVP) list or a merge list. And encoding or decoding the current block by using second information including the MVP list or the merge list.

Description

[0001] Cross-references

[0002] This application is a non-provisional application and claims priority to U.S. Provisional Patent Application No. 63 / 368,382 filed on July 14, 2022, No. 63 / 368,778 filed on July 19, 2022, and No. 63 / 478,704 filed on January 6, 2023. The above U.S. Provisional Patent Applications are incorporated herein by reference in their entirety. Technical Field

[0003] The present invention relates to a video coding system and more particularly to determining an inter-mode direction using a template matching cost in a video coding system. Background Art

[0004] Versatile video coding (VVC) is the latest international video coding standard jointly developed by the ITU-T Video Coding Experts Group (VCEG) and the Joint Video Experts Group (JVE T) of the ISO / IEC Moving Picture Experts Group (MPEG). The standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Information technology-Coded representation of immersive media-Part 3:Versatile Video Coding, released in February 2021. VVC improves the coding efficiency by adding more coding tools on the basis of its predecessor HEVC (High Efficiency Video Coding), and can also process various types of video sources, including 3-dimensional (3D) video signals.

[0005] Figure 1AAn exemplary adaptive Inter / Intra video coding system including loop processing is illustrated. For intra prediction, prediction data is derived from previously coded video data in the current picture. For inter prediction 112, motion estimation (ME) is performed on the encoder side and motion compensation (MC) is performed based on the results of ME to provide prediction data derived from other pictures and motion data. Switch 114 selects intra prediction 110 or inter prediction 112 and the selected prediction data is provided to adder 116 to form a prediction error, also called a residual. The prediction error is then processed by transform (T) 118 and subsequent quantization (Q) 120. The transformed and quantized residual is then encoded by entropy encoder 122 to be included in the video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged with side information (e.g., motion and coding modes associated with intra-frame prediction and inter-frame prediction) and other information (e.g., parameters associated with the loop filter applied to the underlying image area). Side information associated with intra-frame prediction 110, inter-frame prediction 112, and in-loop filter 130 is provided to entropy encoder 122, as shown in FIG. Figure 1A As shown. When using inter-frame prediction mode, one or more reference pictures must also be reconstructed at the encoder end. Therefore, the transformed and quantized residual is processed by inverse quantization (IQ) 124 and inverse transform (IT) 126 to recover the residual. The residual is then added back to the prediction data 136 at reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in a reference picture buffer 134 and used to predict other frames.

[0006] like Figure 1AAs shown, the input video data undergoes a series of processes in the encoding system. Due to the series of processes, the reconstructed video data from REC 128 may be subject to various impairments. Therefore, the loop filter 130 is often applied to the reconstructed video data before it is stored in the reference picture buffer 134 to improve the video quality. For example, a deblocking filter (DF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF) may be used. It may be necessary to merge the loop filter information into the bitstream so that the decoder can correctly recover the required information. Therefore, the loop filter information is also provided to the entropy encoder 122 to be merged into the bitstream. In Figure 1A In , a loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in a reference picture buffer 134 . Figure 1A The system in is intended to illustrate an exemplary structure of a typical video encoder. It may correspond to a High Efficiency Video Codec (HEVC) system, VP8, VP9, ​​H.264, VVC, or any other video codec standard.

[0007] like Figure 1B As shown, the decoder can use similar or identical functional blocks as the encoder, except for transform 118 and quantization 120, because the decoder only needs inverse quantization 124 and inverse transform 126. Instead of the entropy encoder 122, the decoder uses an entropy decoder 140 to decode the video bitstream into quantized transform coefficients and required codec information (such as ILPF information, intra-frame prediction information, and inter-frame prediction information). The intra-frame prediction 150 on the decoder side does not need to perform a pattern search. Instead, the decoder only needs to generate an intra-frame prediction based on the intra-frame prediction information received from the entropy decoder 140. In addition, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) 140 based on the inter-frame prediction information received from the entropy decoder without motion estimation.

[0008] According to VVC, similar to HEVC, the input picture is partitioned into non-overlapped square block regions called CTUs (Codec Tree Units). Each CTU can be divided into one or more smaller-sized Codec Units (CUs). The generated CU partitions can be square or rectangular. In addition, VVC divides CTUs into Prediction Units (PUs) as units for applying prediction processing (e.g., inter-frame prediction, intra-frame prediction, etc.).

[0009] The VVC standard incorporates various new codec tools to further improve the codec efficiency beyond the HEVC standard. Among the various new codec tools, some of these codec tools related to the present invention are briefly described below.

[0010] Inter-frame prediction overview

[0011] According to Section 3.4 of JVET-T2002 (Jianle Chen, et.al., “Algorithm description for Versatile Video Coding and Test Model 11 (VTM 11)”, Joint Video Experts Team (JVET) of ITU-T SG 16WP 3and ISO / IEC JTC 1 / SC 29, 20th Meeting, bytleconference, 7–16 October 2020, Document: JVET-T2002), for each inter-prediction CU, motion parameters consisting of motion vector, reference picture index and reference picture list usage index and additional information are used for the generation of inter-prediction samples. Motion parameters can be sent explicitly or implicitly. When a CU is encoded and decoded in skip mode, the CU is associated with one PU and has no significant residual coefficients, no coded motion vector increment or reference picture index. Merge mode means that the motion parameters of the current CU are obtained from neighboring CUs, including spatial and temporal candidates, as well as additional scheduling introduced in VVC. Merge mode can be used for any inter-predicted CU. An alternative to merge mode is explicit transmission of motion parameters, where the motion vector of each CU, the corresponding reference picture index and reference picture list usage flag of each reference picture list, and other required information are explicitly sent.

[0012] In addition to the inter-frame coding and decoding functions in HEVC, VVC also includes many new and improved inter-frame prediction coding and decoding tools, as listed below:

[0013] —Expanded Merger Forecast

[0014] —Merge mode with MVD (MMVD)

[0015] —Symmetric MVD (SMVD) transmission

[0016] —Affine motion compensated prediction

[0017] —Subblock-based temporal motion vector prediction (SbTMVP)

[0018] - Adaptive motion vector resolution (AMVR) - Motion field storage: 1 / 16 luminance sample MV storage and 8x8 motion field compression

[0019] —Bi-prediction with CU-level weight (BCW)

[0020] —Bi-directional optical flow (BDOF)

[0021] —Decoder side motion vector refinement (DMVR)

[0022] —Geometric partitioning mode (GPM)

[0023] —Combined inter and intra prediction (CIIP)

[0024] The following description provides details of those inter prediction methods specified in VVC.

[0025] Expanded Consolidated Forecast

[0026] In VVC, the merge candidate list is constructed by including the following five categories of candidates in sequence:

[0027] 1) Spatial MVP from spatially adjacent CUs

[0028] 2) Time MVP from the same CU

[0029] 3) History-based MVP from FIFO table

[0030] 4) Paired Average MVP

[0031] 5) Zero MV.

[0032] The size of the merge list is sent in the sequence parameter set (SPS) header and the maximum allowed size of the merge list is 6. For each CU encoded in merge mode, the index of the best merge candidate is encoded using truncated unary binarization. The first bin of the merge index is encoded and decoded using context, and bypass encoding and decoding is used for the remaining bins.

[0033] This section provides the derivation process of merge candidates for each category. As done in HEVC, VVC also supports parallel derivation of merge candidate lists (or merge candidate lists) for all CUs in a certain size region.

[0034] The derivation of spatial merge candidates in VVC is the same as in HE VC, except that the positions of the first two merge candidates are swapped. Figure 2 Up to four merge candidates (B0, A0, B1 and A1) of the current CU 210 are selected from the candidates at the positions shown. The derived order is B0, A0, B1, A1 and B2. Position B2 is only considered when one or more neighboring CUs at positions B0, A0, B1 and A1 are not available (for example, belong to another fragment or tile) or are intra-coded. After the candidate at position A1 is added, the addition of the remaining candidates is checked for redundancy to ensure that candidates with the same motion information are excluded from the list, thereby improving coding efficiency.

[0035] Merge Mode with MVD (MMVD)

[0036] In addition to the merge mode in which the implicitly derived motion information is directly used for the prediction sample generation of the current CU, a merge mode with motion vector difference (MMVD) is introduced in VVC. The MMVD flag is sent immediately after the regular merge flag is sent to specify whether the MMVD mode is used for the CU.

[0037] In MMVD, after a merge candidate (referred to in this disclosure as a basic merge candidate) is selected, it is further refined by sending MVD information. Further information includes a merge candidate flag, an index for specifying the magnitude of motion, and an index for indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected for use as the MV basis. The MMVD candidate flag is sent to specify which one to use between the first and second merge candidates.

[0038] The distance index specifies the motion magnitude information and indicates a predetermined offset from the start points (312 and 322) of the L0 reference block 310 and the L1 reference block 320. Figure 3 As shown, the offset is added to the horizontal component or the vertical component of the starting MV, where small circles of different styles correspond to different offsets from the center. The relationship between the distance index and the predetermined offset is specified in Table 1.

[0039] Table 1 - Relationship between distance index and predetermined offset

[0040]

[0041]

[0042] The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions as shown in Table 1. It should be noted that the meaning of the MVD symbol can change according to the information of the starting MV. When the starting MV is an un-prediction MV or a bi-directional prediction MV, where both lists point to the same side of the current picture (i.e., both referenced POCs are greater than the POC of the current picture, or both are less than the POC of the current picture), the symbol in Table 2 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bi-directional prediction MV, the two MVs point to different sides of the current picture (i.e., one referenced POC is greater than the POC of the current picture, and the other referenced POC is less than the POC of the current picture), and the difference in POC in List 0 is greater than the POC in List 1, the symbol in Table 2 specifies that the sign of the MV offset added to the List 0 MV component of the starting MV has the opposite value to the sign of the List 1 MV. Otherwise, if the difference in POC in List 1 is greater than List 0, the symbol in Table 2 specifies that the sign of the MV offset added to the List 1 MV component of the starting MV has the opposite value to the sign of the List 0 MV.

[0043] The MVD is scaled according to the difference in POC in each direction. If the difference in POC is the same in both lists, no scaling is required. Otherwise, if the POC difference in list 0 is greater than that in list 1, the MVD of list 1 is scaled by defining the POC difference for L0 as td and the POC difference for L1 as tb. If the POC difference for L1 is greater than that for L0, the MVD of list 0 is scaled in the same way. If the starting MV is unidirectionally predicted, the MVD is added to the available MVs.

[0044] Table 2 - MV offset symbols specified by direction index

[0045] Directions to IDX 00 01 10 11 x-axis + - N / A N / A y-axis N / A N / A + -

[0046] Geometric Partitioning Mode (GPM)

[0047] In VVC, geometric partition mode (GPM) is supported for inter-frame prediction, as described in JVET-W2002 (Adrian Browne et al., Algorithm description for Versatile Video Coding and Test Model 14 (VTM14), ITU-T / ISO / IEC Joint Video Exploration Team (JVET), 23rd Meeting, by teleconference, July 7-16, 2021, document: document JVET-M2002). The geometric partition mode is signaled using a CU-level flag as a merge mode. Other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. For each possible CU size, the geometric partition mode supports a total of 64 partitions, w×h=2 m ×2 m , where m,n∈{3...6} does not include 8x64 and 64x8. The GPM mode can be applied to skip or merge CUs with sizes within the above limits and with at least two normal merge modes.

[0048] When using this mode, the CU is split into two parts by a geometrically positioned straight line at a certain angle. In VVC, GPM uses a total of 20 angles and 4 offset distances, which is reduced from 24 angles in earlier drafts. The position of the split line is mathematically derived based on the angle and offset parameters of the specific partition. In VVC, there are a total of 64 partitions, such as Figure 4 As shown in Figure 1, the partitions are grouped according to angle, and the dotted lines indicate redundant partitions. Each part of the geometric partition in the CU uses its own motion for inter-frame prediction; each partition only allows uni-prediction, that is, each part has a motion vector and a reference index. Figure 4 , each row corresponds to the boundary of a partition. The partitions are grouped according to their angles. For example, partition group 410 consists of three vertical GPM partitions (i.e., 90°). Partition group 420 consists of four inclined GPM partitions with a small angle to the vertical direction. Moreover, partition group 430 consists of three vertical GPM partitions similar to group 410, but with the opposite direction (i.e., 270°). A unidirectional prediction motion constraint is applied to ensure that each CU only requires two motion compensated predictions, the same as traditional bidirectional prediction. The unidirectional prediction motion for each partition is derived using the process described later.

[0049] If geometry partition mode is used for the current CU, a geometry partition index indicating the selected partition mode (angle and offset) of the geometry partition as well as two merge indices (one for each partition) are further signaled. The number of maximum GPM candidate sizes is explicitly signaled in the SPS (sequence parameter set) and the syntax binarization of the GPM merge indices is specified. After predicting each part of the geometry partition, the sample values ​​along the edges of the geometry partition are adjusted using a hybrid process with adaptive weights using the process described later. This is the prediction signal for the entire CU and, as with other prediction modes, the transform and quantization process will be applied to the entire CU. Finally, the motion field of the CU predicted using the geometry partition mode is stored using the process described later.

[0050] Combined Inter and Intra Prediction (CIIP)

[0051] In VVC, when a CU is encoded and decoded in merge mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and if both the CU width and the CU height are less than 128 luma samples, an additional flag is sent to indicate whether the combined inter / intra prediction (CIIP) mode is applied to the current CU. As its name suggests, CIIP prediction combines the inter prediction signal with the intra prediction signal. CIIP mode P inter The inter prediction signal in is derived using the same inter prediction process as applied to the normal merge mode; and the intra prediction signal P intra The intra-frame and inter-frame prediction signals are then combined using a weighted average, where the weight values ​​wt are calculated based on the top and left neighboring blocks of the current CU 510 (e.g. Figure 5 The codec mode is calculated as follows:

[0052] If the top neighboring block is available and is intra-coded, isIntraTop is set to 1, otherwise isIntraTop is set to 0;

[0053] If the left adjacent block is available and is intra-coded, isIntraLeft is set to 1, otherwise isIntraLeft is set to 0;

[0054] If (isIntraLeft+isIntraTop) is equal to 2, wt is set to 3;

[0055] Otherwise, if (isIntraLeft+isIntraTop) is equal to 1, wt is set to 2;

[0056] Otherwise, wt is set to 1.

[0057] The CIIP forecast is formed as follows:

[0058] P CIIP =((4-wt)*P imter +wt*P intra +2)>>2 (1)

[0060] Adaptive Motion Vector Resolution (AMVR)

[0061] In HEVC, when use_integer_mv_flag in the slice header is equal to 0, the motion vector difference (MVD) (between the CU's motion vector and the predicted motion vector) is sent in units of one quarter of a brightness sample. In VVC, the CU-level adaptive motion vector resolution (AMVR) scheme is introduced. AMVR allows the CU's MVD to be encoded and decoded with different precisions. Depending on the current CU mode (normal AMVP mode or affine AVMP mode), the current CU's MVD can be adaptively selected as follows:

[0062] Conventional AMVP modes: quarter-brightness samples, half-brightness samples, integer-brightness samples, or quad-brightness samples.

[0063] Affine AMVP mode: quarter luma samples, integer luma samples, or 1 / 16 luma samples.

[0064] If the current CU has at least one non-zero MVD component, the CU-level MVD resolution indication is conditionally sent.If all MVD components (ie, horizontal and vertical MVD for reference list L0 and reference list L1) are zero, then quarter luma sample MVD resolution is inferred.

[0065] For a CU with at least one non-zero MVD component, a first flag is sent to indicate whether quarter luma sample MVD precision is used for the CU. If the first flag is 0, no further transmission is required, and quarter luma sample MVD precision is used for the current CU. Otherwise, a second flag is sent to indicate that half luma samples or other MVD precision (integer or four luma samples) are used for a regular AMVP CU. In the case of half luma samples, a 6-tap interpolation filter is used for the half luma sample position instead of the default 8-tap interpolation filter. Otherwise, a third flag is sent to indicate whether integer luma samples or four luma sample MVD precision is used for a regular AMVP CU. In the case of an affine AMVP CU, the second flag is used to indicate whether integer luma samples or 1 / 16 luma sample MVD precision is used. To ensure that the reconstructed MV has the expected precision (quarter luma samples, half luma samples, integer luma samples, or four luma samples), the motion vector predictor of the CU will be rounded to the same precision as the MVD before being added to the MVD. The motion vector predictor is rounded towards zero (ie, a negative motion vector predictor is rounded towards positive infinity, and a positive motion vector predictor is rounded towards negative infinity).

[0066] The encoder uses the RD check to determine the motion vector resolution of the current CU. In VTM11, in order to avoid always performing four CU-level RD checks for each MVD resolution, the RD check of MVD precision other than quarter luma samples is only called conditionally. For the conventional AVMP mode, the RD cost of the quarter luma sample MVD precision and the integer luma sample MV precision is first calculated. Then, the RD cost of the integer luma sample MVD precision is compared with the RD cost of the quarter luma sample MVD precision to determine whether it is necessary to further check the RD cost of the four luma sample MVD precision. When the RD cost of the quarter luma sample MVD precision is much smaller than the RD cost of the integer luma sample MVD precision, the RD check of the four luma sample MVD precision is skipped. Then, if the RD cost of the integer luma sample MVD precision is significantly greater than the best RD cost of the previously tested MVD precision, the check of the half luma sample MVD precision is skipped. For the affine AMVP mode, if the affine inter mode is not selected after checking the rate-distortion costs of the affine merge / skip mode, merge / skip mode, quarter luma sample MVD precision regular AMVP mode, and quarter luma sample MVD precision affine AMVP mode, the 1 / 16 luma sample MV precision and 1 pixel MV precision affine inter modes are not checked. In addition, the affine parameters obtained in the quarter luma sample MV precision affine inter mode are used as the starting search points for the 1 / 16 luma sample and quarter luma sample MV precision affine inter modes.

[0067] Bi-Prediction with CU-level Weight (BCW)

[0068] In HEVC, the bidirectional prediction signal P bi-pred Generated by averaging two prediction signals P0 and P1 obtained from two different reference pictures and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging, allowing a weighted average of the two prediction signals.

[0069] P bi-pred =((8-w)*P0+w*P1+4)>>3 (2)

[0071] Weighted average bidirectional prediction allows five weights, w∈{-2,3,4,5,10}. For each bidirectionally predicted CU, the weight w is determined in one of two ways: 1) For non-merged CUs, the weight index is sent after the motion vector difference; 2) For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applicable to CUs with 256 or more luma samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency pictures, all 5 weights are used. For non-low-latency pictures, only 3 weights (w∈{3,4,5}) are used. At the encoder, a fast search algorithm is used to find the weight index without significantly increasing the complexity of the encoder. These algorithms are summarized below. The detailed information is disclosed in the VTM software and document JVET-L0646 (Yu-Chi Su, et.al., “CE4-related: Generalized bi-prediction improvements combined from JVET-L0197 and JVET-L0296”, Joint Video Experts Team (JVE T) of ITU-T SG 16WP 3and ISO / IEC JTC 1 / SC 29, 12th Meeting: Macao, CN, 3–12 Oct. 2018, Document: JVET-L0646).

[0072] When combined with AMVR, unequal weights are only conditionally checked for 1-pixel and 4-pixel motion vector precision if the current picture is a low-latency picture.

[0073] When combined with affine, affine ME will be performed for unequal weights if and only if the affine mode is selected as the current best mode.

[0074] Unequal weights are only checked conditionally when the two reference pictures in bidirectional prediction are the same.

[0075] Unequal weights are not searched when certain conditions are met, which depend on the POC distance between the current picture and its reference pictures, the codec QP and the temporal level.

[0076] The BCW weight index is encoded using a context codec bin and a subsequent bypass codec bin. The first context codec bin indicates whether equal weights are used; if unequal weights are used, an additional bin is sent using the bypass codec to indicate which unequal weights are used.

[0077] Weighted prediction (WP for short) is a codec tool supported by the H.264 / AVC and HEVC standards for efficient encoding and decoding of video content with fading. Support for WP has also been added to the VVC standard. WP allows weighting parameters (weights and offsets) to be sent for each reference picture in each reference picture list L0 and L1. Then, during motion compensation, the weights and offsets of the corresponding reference pictures are applied. WP and BCW are designed for different types of video content. In order to avoid interaction between WP and BCW, which complicates the VVC decoder design, if the CU uses WP, the BCW weight index is not sent, and the weight w is inferred to be 4 (i.e., equal weights are applied). For the merged CU, the weight index is inferred from the neighboring blocks based on the merge candidate index. This can be applied to both the normal merge mode and the inherited affine merge mode. For the constructed affine merge mode, the affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index of the CU using the constructed affine merge mode is simply set equal to the BCW index of the first control point MV.

[0078] In VVC, CIIP and BCW cannot be used jointly for a CU. When a CU is coded in CIIP mode, the BCW index of the current CU is set to 2, (ie, equal weight of w=4). Equal weight means the default value of the BCW index.

[0079] Symmetric MVD (SMVD) codec

[0080] In VVC, in addition to normal uni-prediction and bi-prediction mode MVD signaling, a symmetric MVD mode for bi-prediction MVD signaling is applied. In symmetric MVD mode, the motion information including reference picture indexes of both list 0 and list 1 and MVD of list 1 is not signaled but derived.

[0081] The decoding process of the symmetric MVD mode is as follows:

[0082] 1. At the slice level, the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 are derived as follows:

[0083] If mvd_l1_zero_flag is 1, BiDirPredFlag is set equal to 0.

[0084] Otherwise, if the nearest reference picture in list-0 and the nearest reference picture in list-1 form a forward and backward reference picture pair or a backward and forward reference picture pair, BiDirPredFlag is set to 1, and both list-0 and list-1 reference pictures are short-term reference pictures. Otherwise BiDirPredFlag is set to 0.

[0085] 2. At CU level, if the CU is bidirectionally predicted and BiDirPredFlag is equal to 1, the symmetric mode flag is explicitly signaled to indicate whether the symmetric mode is used.

[0086] When the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag and MV D0 are explicitly signaled. The reference index of list-0 and list-1 are set equal to the reference picture pair, respectively. MVD1 is set equal to (-MVD0). The final motion vector is shown in the following equation.

[0087]

[0088] In the encoder, symmetric MVD motion estimation starts with initial MV evaluation. A set of initial MV candidates includes MVs obtained from unidirectional prediction search, MVs obtained from bidirectional prediction search, and MVs from the AMVP list. The one with the lowest rate-distortion cost is selected as the initial MV for symmetric MVD motion search.

[0089] Figure 6 An example of a symmetric MVD pattern is shown, where frames 600, 610, and 620 correspond to the current picture, list-0 reference picture, and list-1 reference picture, respectively. Block 602 corresponds to the current block, and blocks 614 and 622 correspond to reference blocks evaluated according to the initial MV. The final MV is determined by searching the initial MV, and the lowest cost position is selected as the final reference block (612 and 624). The MVD labels are 616 and 626.

[0090] Affine Motion Compensated Prediction

[0091] In HEVC, only the translational motion model is applied to motion compensation prediction (MCP). In the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensation prediction is applied. Fig. 7A As shown in FIG. 1B , the affine motion field of block 710 is given by Fig. 7A The motion information of the two control points (4 parameters) in Figure 7B The three control point motion vectors (6 parameters) are described in .

[0092] For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

[0093]

[0094] For the 6-parameter affine motion model, the motion vector at a sample position (x, y) in the block is derived as:

[0095]

[0096] Where (mv 0x ,mv 0y ) is the motion vector of the upper left control point, (mv 1x ,mv 1y ) is the motion vector of the upper right corner control point, (mv 2x ,mv 2y ) is the motion vector of the lower left control point.

[0097] To simplify motion compensated prediction, block-based affine transform prediction is applied. To derive the motion vector for each 4×4 luminance sub-block, the motion vector of the center sample of each sub-block is Figure 8 As shown, it is calculated according to the above equation and rounded to 1 / 16 fractional precision. Then, the motion compensated interpolation filter is applied to generate a prediction for each subblock with a derived motion vector. The subblock size of the chroma component is also set to 4×4. The MV of the 4×4 chroma subblock is calculated as the average of the MVs of the top left and bottom right luminance subblocks in the same 8x8 luminance region.

[0098] For translational motion inter prediction, there are also two affine motion inter prediction modes: affine merge mode and affine AMVP mode.

[0099] Affine Merge Predictions

[0100] AF_MERGE mode can be applied to CUs with width and height greater than or equal to 8. In this mode, the control point MV (CPMV) of the current CU is generated based on the motion information of the spatially adjacent CUs. There can be up to five CPMVP (CPMV prediction) candidates, and the index is sent to indicate which candidate to use for the current CU. The following three types of CPVM candidates are used to form the affine merge candidate list:

[0101] – Inherited affine merge candidates inferred from the CPMV of neighboring CUs

[0102] – Constructed affine mergecandidate (CPMVP) derived from the translation MV of the adjacent CU

[0103] –Zero MV

[0104] In VVC, there are at most two inherited affine candidates, which come from the affine motion models of neighboring blocks, one from the left neighboring CU and one from the upper neighboring CU. Figure 2 The blocks shown are the same. For the left predictor, the scanning order is A0->A1, and for the upper predictor, the scanning order is B0->B1->B2. Only the first inherited candidate on each side is selected. No pruning check is performed between two inherited candidates. When a neighboring affine CU is identified, its control point motion vectors are used to derive the CPMVP candidate in the affine merge list of the current CU. Fig. 9 As shown, if the lower left neighboring block A of the current block 910 is encoded and decoded in affine mode, the motion vectors v2, v3 and v4 of the upper left corner, upper right corner and lower left corner of the CU 920 containing block A are obtained. When block A is encoded and decoded using a 4-parameter affine model, the two CPMVs (i.e., v0 and v1) of the current CU are calculated based on v2 and v3. When block A is encoded and decoded using a 6-parameter affine model, the three CPMVs of the current CU are calculated based on v2, v3 and v4.

[0105] Constructing affine candidates means constructing candidates by combining the adjacent translation motion information of each control point. Fig. 9 As shown, the motion information of the control point is derived from the specified spatial and temporal neighboring blocks of the current block 910. CPMV k (k=1, 2, 3, 4) indicates the kth control point. For CPMV1, B2->B3->A2 blocks are checked and the MV of the first available block is used. For CPMV2, B1->B0 blocks are checked, and for CPMV3, A1->A0 blocks are checked. If TMVP is available, it is used as CPMV4.

[0106] After obtaining the MVs of the four control points, an affine merge candidate is constructed based on the motion information. The following combination of control point MVs is used to construct in order:

[0107] {CPMV1,CPMV2,CPMV3},{CPMV1,CPMV2,CPMV4},{CPMV1,CPMV3,CPMV4},

[0108] {CPMV2,CPMV3,CPMV4},{CPMV1,CPMV2},{CPMV1,CPMV3}

[0109] Combinations of 3 CPMVs construct 6-parameter affine merge candidates and combinations of 2 CPMVs construct 4-parameter affine merge candidates. To avoid motion scaling processing, the relevant combination of control point MVs is discarded if the reference indexes of the control points are different.

[0110] After the inherited affine merge candidates and the constructed affine merge candidates are checked, if the list is still not full, a zero MV is inserted at the end of the list.

[0111] Affine AMVP prediction

[0112] Affine AMVP mode can be applied to CUs with width and height greater than or equal to 16. A CU-level affine flag is sent in the bitstream to indicate whether affine AMVP mode is used, and then another flag is sent to indicate whether 4-parameter affine or 6-parameter affine is used. In this mode, the difference between the CPMV of the current CU and its predicted sub-CPMVP is sent in the bitstream. The affine AVMP candidate list size is 2, which is generated in sequence by the following four CPVM candidates:

[0113] – Inherited affine AMVP candidates inferred from the CPMV of neighboring CUs

[0114] – The constructed affine AMVP candidate CPMVP is derived using the translation MV of the adjacent CU

[0115] –Translation MV from neighboring CU

[0116] – Zero MV

[0117] The order in which inherited affine AMVP candidates are checked is the same as that of inherited affine merge candidates. The only difference is that for AVMP candidates, only affine CUs with the same reference picture as the current block are considered. When an inherited affine motion predictor is inserted into the candidate list, the pruning process is not applied.

[0118] The AMVP candidates constructed from Fig.10The specified spatial neighboring blocks of the current block 1010 shown in are derived. The same check order as in the affine merge candidate construction is used. In addition, the reference picture index of the neighboring block is also checked. In the check order, the first block using inter-frame coding and having the same reference picture in the current CU is used. When the current CU is encoded using 4-parameter affine mode, and both mv0 and mv1 are available, they are added to the affine AMVP list as a candidate. When the current CU is encoded using 6-parameter affine mode, and all three CPMVs are available, they are added to the affine AMVP list as a candidate. Otherwise, the constructed AMVP candidate is set to unavailable.

[0119] If the number of affine AMVP candidates in the affine AMVP list is still less than 2 after inserting the valid inherited affine AMVP candidates and the constructed AMVP candidates, then mv0, mv1, and mv2 are added as translation MVs to predict all control point MVs of the current CU when available. Finally, if the affine AMVP list is still not full, zero MVs are used to fill the affine AMVP list.

[0120] Subblock-based Temporal Motion Vector Prediction (SbTMVP) in VVC

[0121] VVC supports the sub-block-based temporal motion vector prediction (SbTMVP) method. Similar to the temporal motion vector prediction (TM VP) in HEVC, SbTMVP uses the motion field in the collocated picture to improve the motion vector prediction and merge mode of the CU in the current picture. The same collocated pictures used by TMVP are used for SbTM VP. The differences between SbTMVP and TMVP are mainly in the following two aspects:

[0122] TMVP predicts motion at the CU level, but SbTMVP predicts motion at the sub-CU level;

[0123] TMVP obtains the temporal motion vector from a collocated block in a collocated picture (the collocated block is the bottom-right or center block relative to the current CU), and SbTMVP applies a motion offset before obtaining the temporal motion information from the collocated picture, where the motion offset is obtained from the motion vector of one of the spatially neighboring blocks of the current CU.

[0124] Fig.11A -B shows the SbTMVP process. SbTMVP predicts the motion vector of the sub-CU in the current CU in two steps. In the first step, check Fig.11AThe spatial neighbor A1 in is selected. If A1 has a motion vector that uses the collocated picture as its reference picture, then that motion vector is selected as the motion offset to be applied. If no such motion is identified, then the motion offset is set to (0,0).

[0125] In the second step, the motion offset identified in step 1 is applied (i.e., added to the coordinates of the current block) to obtain the sub-CU level motion information (motion vector and reference index) from the collocated picture, as Fig. 11B shown. Fig. 11B The example in assumes that the motion offset is set to the motion of block A1, where frame 1120 corresponds to the current picture and frame 1130 corresponds to the reference picture (i.e., the collocated picture). Then, for each sub-CU, the motion information of its corresponding block in the collocated picture (the minimum motion grid covering the center sample) is used to derive the motion information of the sub-CU. After determining the motion information of the collocated sub-CU, it is converted into the motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference picture of the temporal motion vector to the picture of the current CU. Fig. 11B In the example of FIG. 11, the arrow in each sub-block of the collocated picture 1130 corresponds to the motion vector of the collocated sub-block (the thick arrow for L0 MV and the thin arrow for L1 MV). For the current picture 1120, the arrow in each sub-block corresponds to the scaled motion vector of the current sub-block (the thick arrow for L0 MV and the thin arrow for L1 MV).

[0126] In VVC, a sub-block based merge list containing a combination of SbTMVP candidates and affine merge candidates is used for signaling of sub-block based merge mode. SbTMVP mode is enabled / disabled by a sequence parameter set (SPS) flag. If SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry in the sub-block based merge candidate list, followed by the affine merge candidates. The size of the sub-block based merge list in VVC is signaled in the SPS, and the maximum allowed size of the sub-block based merge list is 5.

[0127] The sub-CU size used in SbTMVP is fixed to 8x8. Like the affine merge mode, the SbTMV P mode is only applicable to CUs whose width and height are greater than or equal to 8.

[0128] The encoding process flow of the additional SbTMVP merge candidates is the same as that of other merge candidates, that is, for each CU in a P or B slice, an additional RD check is performed to decide whether to use the SbTMVP candidate.

[0129] Bi-directional optical flow (BDOF)

[0130] The Bidirectional Optical Flow (BDOF) tool is included in VVC. BDOF (formerly known as BIO) has been included in JEM. Compared to the JEM version, BDOF in VVC is a simpler version that requires much less computation, especially in terms of the number of multiplications and the size of the multipliers.

[0131] BDOF is used to refine the bidirectional prediction signal of a CU at the 4×4 sub-block level. BDOF is applied to a CU if all of the following conditions are met:

[0132] -CU is encoded using "true" bi-prediction mode, that is, one of the two reference pictures is located before the current picture in display order, and the other reference picture is located after the current picture in display order.

[0133] - The distances from the two reference images to the current image (i.e., POC difference) are the same.

[0134] -Both reference images are short-term reference images.

[0135] -CU does not use affine mode or SbTMVP merge mode for encoding and decoding.

[0136] -CU has more than 64 luma samples.

[0137] -CU height and CU width are both greater than or equal to 8 luma samples

[0138] -BCW weight index means equal weight.

[0139] - Currently CU is not enabled for WP.

[0140] - Currently CU does not use CIIP mode.

[0141] BDOF is only applicable to the luma component. As the name suggests, BDOF mode is based on the concept of optical flow and assumes that the motion of objects is smooth. For each 4×4 sub-block, the motion refinement (v x ,v y ). Motion refinement is then used to adjust the bidirectional prediction sample values ​​in the 4x4 sub-block.

[0142] Decoder Side Motion Vector Refinement (DMVR) in VVC

[0143] In order to increase the accuracy of the MV in the merge mode, decoder-side motion vector refinement based on bilateral-matching (BM) is applied in VVC. In the bidirectional prediction operation, the refined MV is searched around the initial MVs (1232 and 1234) in the reference picture list L0 1212 and the reference picture list L1 1214 for the current block 1220 and the current picture 1210. The collocated blocks 1222 and 1224 in L0 and L1 are determined based on the initial MVs 1232 and 1234) and the position of the current block 1220 in the current picture, as shown in FIG. Fig.12 As shown. The BM method calculates the distortion between two candidate blocks (1242 and 1244) in the reference picture list L0 and list L1. Two candidate MVs (1252 and 1254) are derived by adding two opposite offsets (1262 and 1264) to the two initial MVs (1232 and 1234) to determine the positions of the two candidate blocks (1242 and 1244). Fig.12 As shown, the SAD between the candidate blocks (1242 and 1244) based on each MV candidate around the initial MV (1232 or 1234) is calculated. The MV candidate (1252 or 1254) with the lowest SAD becomes the refined MV and is used to generate a bidirectional prediction signal.

[0144] In VVC, the application of DMVR is restricted and only applies to CUs encoded and decoded in the following modes and features:

[0145] -CU level merge mode with bi-predictive MV

[0146] - One reference image is in the past and the other is in the future relative to the current image

[0147] - The distances (POC differences) from the two reference images to the current image are the same

[0148] - Both reference images are short-term reference images

[0149] -CU has more than 64 luminance samples

[0150] -CU height and CU width are both greater than or equal to 8 luma samples

[0151] -BCW Weight Index indicates equal weighting

[0152] -Current block is not WP enabled

[0153] - The current block does not use CIIP mode

[0154] The refined MV derived by the DMVR process is used to generate inter-frame prediction samples and is also used for temporal motion vector prediction for future picture encoding and decoding, while the original MV is used in the deblocking process and is also used for spatial motion vector prediction for future CU encoding and decoding.

[0155] Boundary matching for sign prediction

[0156] The Joint Video Experts Group (JVET) of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11 is currently exploring the next generation of video codec standards. The Enhanced Compression Model 2 (ECM 2) adopts some promising new codec tools (M. Coban et al., "Algorithm description of Enhanced Compression Model 2 (ECM 2)", Joint Video Experts Group of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11, 23rd Meeting, Teleconference, July 7-16, 2021, document JVET-W2025) to further improve VVC. The adopted new tools have been implemented in the reference software ECM-2.0 (ECM Reference Software ECM-2.0, available at https: / / vcgit.hhi.fraunhofer.de / ecm / ECM [online]). In particular, a new method for jointly predicting the set of transform coefficient level signs in a residual transform block has been developed (JVET-D0031, Felix Henry et al., "Residual Coefficient Sign Prediction", Joint Video Experts Group of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11, 4th Meeting: Chengdu, China, October 15-21, 2016, document JVET-D0031). In ECM 2, to derive the best sign prediction hypothesis for the residual transform block, the cost function is defined as spanning Fig.13 13, where block 1310 corresponds to a transformed block, circle 1320 corresponds to adjacent samples, and circle 1330 corresponds to reconstructed samples associated with the symbol candidate for block 1310. The cost function is defined as the absolute second derivative in the residual domain of the rows and left columns above.

[0157] derivatives) as follows:

[0158]

[0159] In the above equation, R is the reconstructed neighbor, P is the prediction of the current block, and r is the residual hypothesis. The term (-R -1 +2R0-P1), and only the residual assumption is subtracted.

[0160] Select the transform coefficient with the largest qIdx value in the upper left 4x4 region. The qIdx value is the transform coefficient level after compensating for the effects of multiple quantizers in DQ. A larger qIdx value will produce a larger dequantized transform coefficient level. The derivation of qIdx is as follows

[0161] qIdx=(abs(level)<<1)-(state&1);

[0162] where level is the transform coefficient level parsed from the bitstream and state is a variable maintained by the encoder and decoder in DQ.

[0163] The symbol prediction area is extended to a maximum of 32x32. The symbol of the upper left corner MxN block is predicted. The values ​​of M and N are calculated as follows:

[0164] M=min(w,maxW),

[0165] N=min(h,maxH),

[0166] Where w and h are the width and height of the transform block. The maximum region for symbol prediction is not always set to 32x32. The encoder sets the maximum region (maxW, maxH) based on the configuration, sequence class, and QP, and signals the region in the SPS.

[0167] The maximum number of predicted symbols remains the same. Symbol prediction also applies to LFNST blocks. For LFNST blocks, a maximum of 4 coefficients in the top left 4x4 region are allowed for symbol prediction.

[0168] Template matching for MV refinement

[0169] Template matching (TM) is a decoder-side MV derivation method used to refine the motion information of the current CU by finding the closest match between the template in the current picture 1410 (i.e., the top neighboring block 1414 and / or the left neighboring block 1416 of the current CU 1412) and the blocks in the reference picture 1420 (i.e., blocks 1424 and 1426 of the same size as the template). Fig.14In the pixel search range 1422, a better MV is searched around the initial motion 1430 in the range [–8, +8] around the position 1428 in the reference image 1420 pointed to by the initial MV 1430. The template matching method in JVET-J0021 is used with the following modifications: the search step size is determined according to the AMVR mode, and the TM can be cascaded with the bilateral matching process in the merge mode.

[0170] In AMVP mode, MVP candidates are determined based on template matching errors to select the MVP candidate with the smallest difference between the current block template and the reference block template. TM is then performed only for this specific MVP candidate for MV refinement. TM refines this MVP candidate using an iterative diamond search starting from full-pixel MVD accuracy (or 4-pixel AMVR mode) within a [–8, +8] pixel search range. The AMVP candidate can be further refined by using a cross search with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode) followed by half-pixel and quarter-pixel searches according to the AMVR mode specified in Table 3. This search process ensures that the MVP candidate still maintains the same MV accuracy as indicated by the AMVR mode after the TM process. During the search process, if the difference between the previous minimum cost and the current minimum cost in the iteration is less than or equal to a threshold (the threshold is equal to the block area), the search process terminates.

[0171] Table 3. Search patterns of AMVR and merge mode.

[0172]

[0173] In merge mode, a similar search method is applied to the merge candidate indicated by the merge index. As shown in Table 3, TM can be performed all the way to 1 / 8 pixel MVD accuracy, or skipped beyond half-pixel MVD accuracy, depending on whether an alternative interpolation filter is used based on the merged motion information (for AMVR in half-pixel mode). In addition, when TM mode is enabled, template matching can be used as an independent process between block-based and sub-block-based bilateral matching (BM) methods or an additional MV refinement process, depending on whether BM can be enabled according to its enabled status.

[0174] MVD Sign Prediction

[0175] In JVET-X0132 (YanZhang et al., "Non-EE2: On MVD sign prediction", Joint Video Experts Group (JVET) of ITU-T SG16WP 3 and ISO / IEC JTC 1 / SC, 24th Meeting, Teleconference, October 6-15, 2021, Document: JVET-X0132), if the motion vector difference contains a non-zero component, the motion vector difference sign prediction can be applied in the regular inter mode. The possible MVD symbol combinations are sorted according to the template matching cost, and the index corresponding to the true MVD symbol is derived and encoded and decoded using the context model. On the decoder side, the derivation of the MVD symbol is as follows:

[0176] - resolve the size of the MVD components,

[0177] - Parse the MVD symbol prediction index for context codec,

[0178] - Create MV candidates by building possible combinations between signs and absolute MVD values ​​and add them to the MV predictor,

[0179] - derive MVD symbol prediction cost for each derived MV based on template matching cost and ranking,

[0180] - use the MVD symbol prediction index to pick the true MVD symbol, and

[0181] - Add the true MVD to the MV predictor of the final MV.

[0182] A bilinear filter is used for reference template generation. The template matching cost is measured by the SAD between the neighboring samples of the current CU and its corresponding reference samples, such as Fig.15 As shown. Fig.15, the template (1514 and 1516) of the current PU 1512 in the current picture 1510 is shown. The motion vector 1530 of PU 1512 points to the position P in the reference picture 1520. Four MVD candidates based on the absolute value of the MVD (assuming that both the X and Y components of the MVD are non-zero) are generated for the four possible sign combinations. The positions of the four reference blocks associated with the four MVD candidates are labeled A, B, C, and D. For each MVD candidate, the template matching cost is measured. For example, for MVD candidate A, the matching cost between the template (i.e., 1514 and 1516) of the current block (i.e., PU 1512) and the corresponding template (i.e., 1524 and 1526) of MVD candidate A is calculated. The matching costs of all four MVD candidates are calculated, and the one with the lowest cost is selected as the final MVD. Therefore, the sign associated with the final MVD candidate is implicitly determined. The samples in the template have been reconstructed for the current PU so that the decoder can perform the same process as the encoder.

[0183] Several other codec tools such as SMVD (Symmetric Motion Vector Difference) mode, affine mode, and MMVD mode have been introduced during VVC standardization. In ECM, the affine MMVD mode is also introduced. All these tools either explicitly signal the MVD symbol as part of the MVD or implicitly signal the MVD symbol through the direction index. This shows that MVSD can also be applied to these codec tools. When MVD symbol prediction is applied to affine and affine MMVD modes, a sub-block based template is used, and the template matching cost of each sub-block is accumulated as the final cost. The template matching cost derivation for the affine and affine MMVD modes is shown in Figures 16A-B. As Fig.16A As shown, the current block template (i.e. Fig.16A The templates A0, A1, ... and L0, L1, ...) in TM are the same as those in the conventional TM applied to the translational motion model, but the reference block template (i.e., Fig. 16B A0, A1, ... and L0, L1, ... in include multiple 4x4 sub-blocks, which are pointed to by MVs derived from the CPMVs of adjacent sub-blocks at the CU boundary. The search process of the affine TM starts from CPMV0, while keeping other CPMVs unchanged, and performs searches in the horizontal and vertical directions, and performs diagonal searches only when the zero vector is not the best difference vector found from the horizontal / vertical search. Then, if a 6-parameter model is used, the affine TM repeats the same search process for CPMV1 and CPMV2. Based on the refined CPMV, if the zero vector is not the best difference vector of the previous iteration and the search process has less than 3 iterations, the entire search process starts again from CPMV0.

[0184] In case of bidirectional prediction in SMVD, MMVD and Affine MMVD modes, to reduce complexity, only templates from reference list 0 are used to derive the template matching cost. When the MVD sign prediction index is encoded or decoded, different contexts are used depending on the size of the MVD component. Predefined thresholds are used for comparison with the MVD component amplitudes. Different thresholds are used for shifted inter mode and affine mode.

[0185] Adaptive reference image reordering

[0186] In JVET-Y0139 (Han Huang et al., "Non-EE2: On the extended number of activereference pictures and reference picture reordering", Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29, 25th Meeting, Teleconference, January 12-21, 2022, Document: JVET-Y0139), the proposed reference picture reordering method is based on template matching cost. For unidirectional prediction AMVP mode, the reference pictures in list 0 and list 1 are interleaved to produce a joint list. For each hypothesis of the reference pictures in the joint list, motion information can be derived accordingly and template matching can be performed to calculate the cost. The joint list is reordered in ascending order of template matching cost. The index of the selected reference picture in the reordered joint list is signaled in the bitstream. For bidirectional prediction AMVP mode, a list of reference picture pairs from list 0 and list 1 is generated and similarly reordered based on template matching cost. The index of the selected pair is signaled in the bitstream. Summary of the invention

[0187] A method and apparatus for inter-frame prediction in a video coding and decoding system are disclosed. According to the method, input data associated with a current block is received, wherein the input data includes pixel data of the current block to be encoded at the encoder side or encoded data associated with the current block to be decoded at the decoder side, and wherein an MVP (motion vector prediction) candidate of the current block includes a first MV predictor pointing to a first reference block in an L0 reference picture and a second MV predictor pointing to a second reference block in an L1 reference picture. An L0 matching cost between a first template corresponding to one or more first neighboring regions of the first reference block and a current template corresponding to one or more current neighboring regions of the current block is determined. An L1 matching cost between a second template corresponding to one or more second neighboring regions of the second reference block and the current template is determined. An inter-frame direction of an MVP candidate of the current block is determined based on first information including an L0 matching cost and an L1 matching cost, wherein the inter-frame direction corresponds to bidirectional prediction, L0 unidirectional prediction, or L1 unidirectional prediction. The MVP candidate is inserted into an AMVP (adaptive MVP) list or a merge list. The current block is encoded or decoded by using second information including the MVP list or the merge list.

[0188] In one embodiment, a bi-prediction matching cost between a mixed template and a current template is calculated, and wherein the mixed template is derived by mixing the first template and the second template. In one embodiment, the bi-prediction matching cost, the L0 matching cost, and the L1 matching cost are calculated based on the sum of absolute differences (SAD) or the sum of squared differences (SSD).

[0189] In one embodiment, the bi-prediction matching cost is weighted by a factor less than 1 for the matching cost comparison between the L0 matching cost, the L1 matching cost, and the bi-prediction matching cost.

[0190] In one embodiment, the inter direction of the MVP candidate for the current block is determined on a per-sample basis. In one embodiment, if the difference between the corresponding L0 reference sample of the target sample and the corresponding L1 reference sample of the target sample is greater than a threshold, the target sample in the current block is a candidate for changing from bidirectional prediction to unidirectional prediction.

[0191] In one embodiment, if the L0 matching cost is the smallest of the L0 matching cost, the L1 matching cost, and the bidirectional prediction matching cost, the target sample is changed to L0 unidirectional prediction, or if the L1 matching cost is the smallest of the L0 matching cost, the L1 matching cost, and the bidirectional prediction matching cost, the target sample is changed to L1 unidirectional prediction. In one embodiment, if the bidirectional prediction matching cost is the smallest of the L0 matching cost, the L1 matching cost, and the bidirectional prediction matching cost, the target sample remains using bidirectional prediction.

[0192] In one embodiment, the threshold is fixed or determined according to a set of candidate thresholds. In another embodiment, the threshold is adaptively determined for the current block. In one embodiment, the threshold is adaptively selected from a set of candidate thresholds based on the L0 matching cost, L1 matching cost and bidirectional prediction matching cost calculated for each of the set of candidate thresholds, and the threshold corresponds to the target candidate threshold that achieves the lowest matching cost in the candidate threshold set. In one embodiment, the threshold is adaptively selected from a set of candidate thresholds based on the increment number associated with each of the candidate threshold sets, and the increment number is calculated as the increment from the current total number of samples where the absolute difference between the L0 predictor and the L1 predictor is less than the current candidate threshold to the next total number of samples where the absolute difference between the L0 predictor and the L1 predictor is less than the next candidate threshold, and the threshold corresponds to the target candidate threshold with the largest increment number in the set of candidate thresholds. In one embodiment, the increment number associated with each of the set of candidate thresholds is calculated for the current template, the first template and the second template respectively to determine the three corresponding thresholds of the current template, the first template and the second template, and if the three corresponding thresholds are the same, the inter-frame direction of the MVP candidate of the current block is determined based on the first information including the L0 matching cost and the L1 matching cost. If the three corresponding thresholds are not the same, the method of determining the inter direction of the MVP candidate of the current block based on the first information including the L0 matching cost and the L1 matching cost is not applied.

[0193] In one embodiment, the threshold value depends on the QP (quantization parameter) of the current block, the block size of the current block, one or more template matching costs of the current block, the number of samples in the current block whose differences between corresponding L0 and L1 reference samples fall within the respective threshold intervals, or a combination thereof. In one embodiment, the smaller the QP (quantization parameter) of the current block, the smaller the threshold value is set to.

[0194] In one embodiment, the current block is processed through BDOF (Bidirectional Optical Flow) or MP-DMVR (Multi-channel Decoder-side Motion Vector Refinement). In one embodiment, the MVP candidate of the current block corresponds to an inter-frame AMVP candidate, an inter-frame merge candidate, an affine AMVP candidate, an affine merge candidate, or an S bTMvp (sub-block based temporal motion vector prediction) candidate.

[0195] In one embodiment, the current block corresponds to a luminance block or a chrominance block. In one embodiment, the thresholds between the luminance block and the chrominance block are different.

[0196] In one embodiment, RD (rate-distortion) costs of the current block with and without per-sample inter-direction determination are calculated to decide whether to apply per-sample inter-direction determination of the current block.

[0197] In one embodiment, three RD costs are calculated, a decision is selected for a current block as to whether to use per-sample inter-direction determination and a corresponding inter-direction associated with a minimum RD cost, and wherein a first RD cost corresponds to encoding the current block using bidirectional prediction, and a second RD cost and a third RD cost correspond to encoding and decoding the current block using L0 unidirectional prediction and L1 unidirectional prediction, respectively, for samples in the current block whose difference between corresponding L0 and L1 reference samples is greater than a threshold. In one embodiment, two bits are signaled to indicate whether to use per-sample inter-direction determination and a corresponding inter-direction.

[0198] In one embodiment, two RD costs are calculated and a decision is determined for a current block as to whether to use inter direction determination based on a per-sample basis, and wherein a first RD cost corresponds to encoding the current block using bidirectional prediction, and a second RD cost corresponds to encoding or decoding the current block using L0 unidirectional prediction or L1 unidirectional prediction according to an L0 matching cost and an L1 matching cost for samples in the current block having a difference between corresponding L0 and L1 reference samples greater than a threshold. In one embodiment, one bit is signaled to indicate whether to use inter direction determination based on a per-sample basis.

[0199] In one embodiment, the inter-frame direction of the MVP candidate is determined on a per-block basis for the current block. In one embodiment, the MVP candidate corresponds to a conventional merge candidate, a GPM (geometric partitioning mode) candidate, an MMVD (merged motion vector difference) candidate, a BM (bilateral matching) candidate, or an affine candidate, a CIIP candidate. BRIEF DESCRIPTION OF THE DRAWINGS

[0200] Figure 1A An exemplary adaptive inter / intra video coding system incorporating in-loop processing is described.

[0201] Figure 1B Shows Figure 1A The corresponding decoder of the encoder in .

[0202] Figure 2 Neighboring blocks used to derive spatial merging candidates for VVC are shown.

[0203] Figure 3 The distance offset from the starting MV in horizontal and vertical directions according to the merge mode with MVD (MMVD) is shown.

[0204] Figure 4 An example of 64 partitions used in the VVC standard is shown, where the partitions are grouped according to their angles and the dashed lines indicate redundant partitions.

[0205] Figure 5An example of derivation of weight values ​​for combined inter and intra prediction (CIIP) according to the coding modes of the top and left neighboring blocks is shown.

[0206] Figure 6 An example of a symmetric MVD pattern is shown.

[0207] Fig. 7A An example of the affine motion field of a block described by the motion information (4 parameters) of two control points is shown.

[0208] Figure 7B An example of the affine motion field of a block described by the motion information of three control point motion vectors (6 parameters) is shown.

[0209] Figure 8 An example of block-based affine transform prediction is shown, where the motion vector of each 4×4 luma sub-block is derived from the control point MV.

[0210] Fig. 9 An example of deriving inherited affine candidates based on the control point MVs of neighboring patches is shown.

[0211] Fig.10 An example of constructing affine candidates by combining translational motion information of each control point from both spatial and temporal neighbors is shown.

[0212] FIG. 11A shows an example of sub-block based temporal motion vector prediction (SbTMVP) in VVC, where the availability of motion information of spatial neighboring blocks is checked.

[0213] FIG. 11B shows an example of an SbTMVP for deriving a sub-CU motion field by applying motion shifts from spatial neighbors and scaling motion information from a corresponding collocated sub-CU.

[0214] Fig.12 An example of decoding-side motion vector refinement is shown.

[0215] Fig.13 The cost function calculation for deriving the best sign prediction hypothesis for a residual transform block according to the Enhanced Compression Model 2 (ECM 2) is shown.

[0216] Fig.14 An example of template matching for refining an initial MV by searching a region around the initial MV is shown.

[0217] Fig.15 An example of template matching cost derivation for MVD (motion vector difference) sign prediction is shown.

[0218] Fig.16A An example of the template of the current block in the derivation of the affine template matching cost is shown.

[0219] Fig. 16B An example of a template of a reference sub-block in the derivation of the affine template matching cost is shown.

[0220] Fig.17 A flowchart of an exemplary video coding and decoding system for determining the inter-frame direction of MVP candidates according to template matching cost according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0221] It will be readily appreciated that the components of the present invention as generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of the embodiments of the systems and methods of the present invention as shown in the figures is not intended to limit the scope of the claimed invention, but is merely representative of selected embodiments of the present invention. References throughout this specification to "one embodiment," "an embodiment," or similar language mean that a particular feature, structure, or characteristic described in conjunction with that embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing throughout this specification do not necessarily all refer to the same embodiment.

[0222] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. However, those skilled in the relevant art will recognize that the present invention may be practiced without one or more specific details, or practiced using other methods, components, etc. In other cases, well-known structures or detailed descriptions of operations are not shown or are not shown to avoid obscuring aspects of the present invention. The illustrated embodiments of the present invention will be best understood with reference to the accompanying drawings, in which like parts are represented by like numbers throughout. The following description is intended to be exemplary only, and simply illustrates certain selected embodiments of the apparatus and methods consistent with the present invention as claimed herein.

[0223] Below, various inter-frame prediction improvements according to the present invention are disclosed. In the present invention, a sample-based inter-frame mode determination method using template matching cost is disclosed. The sample-based inter-frame mode determination can adaptively change from a bidirectional prediction mode to a unidirectional prediction mode on a per-sample basis.

[0224] Implicit sample-based inter direction determination

[0225] In the Enhanced Compression Model (ECM) currently being developed for the next generation of video codec technology, the inter-frame direction of a CU can be determined by the rate-distortion (RD) cost on the encoder side, the MVD and L0 and L1 reference indexes can be signaled for the inter-frame AMVP mode (i.e., SMVD, AMVR, etc.), or the inter-frame direction can be determined by the inherited CU of the inter-frame merge mode. However, both methods are CU-based. That is, the entire CU should follow the same interaction direction, which loses flexibility. According to the present invention, in some cases (e.g., object occlusion), samples in the foreground area can refer to samples in one of the predictors through unidirectional prediction, and bidirectional prediction mixing is only applied to samples in the background area. Therefore, the present invention proposes a sample-based inter-frame direction determination method, which can independently determine the inter-frame direction of each sample using certain criteria to improve the codec gain. The inter-frame direction refers to whether the inter-frame prediction mode is unidirectional (using L0 or L1) or bidirectional (i.e., both L0 and L1) inter-frame prediction.

[0226] In one embodiment, for a bidirectional prediction CU, if the difference between the corresponding L0 and L1 reference samples is greater than a threshold, the sample in the current CU will be changed to unidirectional prediction. This threshold can be a fixed factor or determined through any adaptive method. In order to determine whether to change the sample to L0 or L1 unidirectional prediction, or continue to use bidirectional prediction (i.e., keep bidirectional prediction), the template matching cost of the bidirectional prediction template (the result of the bidirectional prediction mixture of the L0 and L1 templates), the L0 unidirectional prediction template, and the L1 unidirectional prediction template are calculated. The template with the smallest cost will be the best template, and the corresponding inter-frame direction will be the best inter-frame direction. For samples whose corresponding difference between the L0 and L1 reference samples is greater than the threshold, the bidirectional prediction mixture is replaced with a unidirectional prediction following the best inter-frame direction. If the L0 unidirectional prediction template or the L1 unidirectional prediction template has the smallest cost. For samples whose corresponding difference between the L0 and L1 reference samples is greater than the threshold, if the bidirectional prediction matching cost is the smallest among all matching costs, the sample remains bidirectionally predicted.

[0227] In one embodiment, if the difference between the corresponding L0 and L1 reference samples is greater than a threshold (denoted as Tf) selected from a set of thresholds (denoted as {T0, T1, ...}), the samples in the bidirectionally predicted CU will be changed to unidirectionally predicted. The threshold Tf is determined by the CU size, the QP value, the average of the differences between the L0 and L1 predictors, or the number of samples where the difference between the L0 and L1 predictors falls within a threshold interval (e.g., the number of samples where the difference falls within the interval [Ta, Tb)). For larger QP values, the threshold Tf may be larger and may be the threshold Ta where most samples fall within the threshold interval. Any other information may be considered to determine the threshold. The threshold may be determined by using information from the codec, such as by reconstructing neighboring samples and reference templates of the L0 and / or L1 predictors. The threshold may also be signaled in the bitstream at the TB / TU / CB / CU / CTU / CTU group / CTU row / slice / picture / sequence level.

[0228] In one embodiment, the template matching cost of the bidirectional prediction template (the result of the bidirectional prediction of the L0 and L1 templates), the L0 template and the L1 template can be calculated by the sum of absolute differences (abbreviated as SAD) or the sum of squared differences (abbreviated as SSD) or the sum of absolute transformed differences (abbreviated as SATD). The area of ​​accumulating the sum of the differences is determined by a threshold Tt (i.e., only samples whose difference between the corresponding L0 and L1 template samples is greater than Tt will be used to calculate the template matching cost), and the threshold can be a fixed factor or the same as Tf or determined according to the number of samples whose difference between the L0 and L1 templates falls within each threshold interval.

[0229] As described above, the threshold can be determined by any adaptive method. For example, the threshold can be adaptively selected from a set of candidate thresholds, such as 8 predefined thresholds {32, 64, 96, 128, ..., 256}. The TM cost associated with L0, L1 and bidirectional prediction can be calculated for each candidate threshold. The direction between the target candidate thresholds that causes the minimum TM cost is selected as the target threshold. Another example of adaptive threshold determination includes counting the total number of absolute differences between corresponding L0 and L1 samples in the template that are less than the selected candidate threshold. Determine the number of increments from the current total number of the current candidate threshold to the next total number of the next candidate threshold. The threshold of the current CU is selected from the candidate threshold with the largest number of increments. In another example, the increment number associated with each of the set of candidate thresholds is first calculated for the current template, the first template, and the second template, respectively, to determine three corresponding thresholds for the current template, the first template, and the second template. If the three corresponding thresholds are the same, the method of determining the inter-frame direction of the MVP candidate of the current block based on the first information including the L0 matching cost and the L1 matching cost is applied; if the three corresponding thresholds are not the same, the method of determining the inter-frame direction of the MVP candidate of the current block based on the first information including the L0 matching cost and the L1 matching cost is not applied.

[0230] In one embodiment, if the threshold Tf of the current CU does not match the threshold Tt of the template, the sample-based inter-direction determination method will not be applied to the current CU.

[0231] In one embodiment, if the threshold Tf of the current CU and the threshold TLt of the left template do not match, but the threshold Tf of the current CU and the threshold TRt of the right template match, the sample-based inter-frame direction determination method will be applied to calculate the template matching cost using only the current CU of the right template, and vice versa.

[0232] In one embodiment, the thresholds for luma samples and chroma samples in a bi-directionally predicted CU may be different and may be determined separately.For each CU, the sample-based inter direction determination method may be applied only to luma, only to chroma, or to both luma and chroma.

[0233] In one embodiment, the sample-based inter-frame direction determination method is applied only when the template quality is guaranteed. That is, if the difference between the current template and each bidirectional prediction template, L0 template or L1 template is greater than a threshold, the sample-based inter-frame direction determination process is skipped. In addition, if the sum of the residuals in the current template area is greater than a threshold, which indicates that the reconstruction in the current template area is not accurate enough, the sample-based inter-frame direction determination process is skipped.

[0234] In one embodiment, the bi-directional prediction template matching cost is weighted by a factor less than 1 so that the inter direction determination process tends to choose bi-directional prediction over L0 or L1 uni-directional prediction.

[0235] In one embodiment, the sample-based inter-frame direction decision method can be applied to inter-frame AMVP, inter-frame merge, affine AMVP, affine merge, SbTMvp and all other inter-frame modes.

[0236] In one embodiment, the sample-based inter-frame direction determination method can be applied together with BDOF. The L0 and L1 predictors after BDOF refinement are stored, and the difference between the refined L0 and L1 predictors is used to determine whether to change the inter-frame direction of each sample in the current CU. The sample-based inter-frame direction determination method can be applied before and / or after sample-based BDOF.

[0237] In one embodiment, the sample-based inter-frame direction decision method can be applied together with MP-DMVR / DMVR / any decoder-side MV derivation / refinement tool. The L0 and L1 predictors refined by MP-DMVR / DMVR / any decoder-side MV derivation / refinement tool are stored, and the difference between the refined L0 and L1 predictors is used to determine whether to change the inter-frame direction of each sample in the current CU. The sample-based inter-frame direction determination method can also be applied before and / or after each pass or one pass of MP-DMVR / DMVR / any decoder-side MV derivation / refinement (i.e., there are 3 passes for MP-DMVR). For example, the MV used to generate the L0 / L1 template can be the MV before MP-DMVR / DMVR / any decoder-side MV derivation / refinement tool or the MV after MP-DMVR / DMVR / any decoder-side MV derivation / refinement tool.

[0238] Signaling based on the sample's inter-frame direction

[0239] Although sample-based inter-frame direction can improve encoding and decoding performance, the performance improvement may not always be achieved for all CUs. Therefore, a method for sample-based inter-frame direction determination through explicit indication is disclosed. According to this method, the RD cost is used at the encoder to determine whether the predictor refined by the sample-based inter-frame direction determination method is better, thereby reducing the bits to be sent to the decoder to determine whether the sample-based inter-frame direction determination method is applied to the current CU.

[0240] In one embodiment, three RD costs are calculated: one for the bi-directional prediction predictor and two for the predictor that changes the sample to L0 or L1 uni-directional prediction if the difference between the L0 and L1 predictors is greater than a threshold. The predictor with the smallest RD cost will be the best predictor and the corresponding inter direction will be the best inter direction. Therefore, two bits are signaled to the decoder to determine the best inter direction and whether the sample-based inter direction determination method should be applied to the current CU. In one example, the decision to use L0, L1 or bi-directional prediction can be reordered by the corresponding TM cost. The syntax of the signaling is to select the best decision in the reordered list.

[0241] In one embodiment, two RD costs are calculated for the bi-predictive predictor and the predictor refined by sample-based inter direction determination using templates. The predictor with the smallest RD cost will be the best predictor. Therefore, only one bit is signaled to the decoder to determine whether the sample-based inter direction determination method is applied to the current CU.

[0242] The selection of L0 / L1 / Bi or the decision whether to apply sample-based inter direction may also be signaled in the bitstream at TB / TU / CB / CU / CTU / CTU group / CTU row / slice / picture / sequence level.

[0243] Merge candidate list inter-frame direction reordering

[0244] According to this method, the inter-frame direction or predictor of each merge candidate in each merge candidate list (i.e., including normal merge, GPM, MMVD, TM, BM, Affine, CIIP, etc.) can be refined through the CU-based inter-frame direction determination method.

[0245] In one embodiment, the inter-frame direction of each merge candidate in each candidate list is reordered according to the template matching costs of the bidirectional prediction template, the L0 template, and the L1 template. The final inter-frame direction will be the inter-frame direction with the minimum template matching cost.

[0246] In one embodiment, the predictor of each merge candidate in each candidate list is refined by using sample-based inter-frame direction determination of the template. The final predictor is selected from the refined merge candidates.

[0247] In one embodiment, the proposed method is set to be applied or not applied to a true bidirectional prediction block, where the reference pictures of the block correspond to one from the past picture and one from the future picture. In another embodiment, the proposed method is set to be applied or not applied to a general bidirectional prediction block, where the reference pictures of the block are all from the past pictures.

[0248] BCW nonlinear operation in generating the final predictor

[0249] According to this method, a nonlinear operation is introduced when generating the BCW final predictor of the following equation. In the initial design, the final predictor is derived according to equation (2):

[0250] P_ bi-pred =Clip3(minValue,maxValue,((8-w)*P_0+w*P_1+4)>>3), (6)

[0251] Among them, minValue and maxValue are based on P0, P l It is the combination of and w.

[0252] In one embodiment, minValue and maxValue are determined by the following equations.

[0253] minValue=((w>4)?P1:P0)–thA;

[0254] maxValue=((w>4)?P1:P0)+thA;

[0255] In another embodiment, minValue and maxValue are determined by the following equations.

[0256] minValue=((w<4)?P1:P0)–thA;

[0257] maxValue=((w<4)?P1:P0)+thA.

[0258] In one embodiment, thA is signaled at the CU level, CTU level, slice level, picture level, or sequence level. In another embodiment, thA is implicitly derived based on QP and some predefined values. For example, when QP increases, thA also increases. In another example, when QP increases, thA decreases. In another example, thA depends on the selected reference picture, such as the temporal distance of two reference frames, the slice QP of the reference picture, etc. In another example, thA depends on the motion vector. For example, if two motion vectors are close to each other, thA uses a larger value. Otherwise, a smaller value is used as thA.

[0259] In one embodiment, these nonlinear operations are applied only when certain specific weight pairs are selected. For example, only when a weight pair "8-w" is selected that is greater than 4, a nonlinear operation is introduced to generate the final predictor. For others, the original equation (6) is used.

[0260] In one embodiment, the proposed method is applied only to CUs whose size is greater than or equal to some predefined size. In another embodiment, the proposed method is applied only to CUs whose size is less than some predefined size. The size can be width only, height only, both, or the product of width and height.

[0261] In one embodiment, the selection between P0 and P1 is determined based on neighboring reconstructed samples. In one example, template matching can be applied to select one of P1 and P0. In another example, block boundary matching is used to select one of P1 and P0. The difference between the neighboring reconstructed samples and each predictor is calculated. Then, based on these two differences, the selection of P1 and P0 can be determined. The selection can be made at the sample level, the segment level, or the CU level. If the selection is made at the sample level, the selection may be different for each sample. If the selection is made at the CU level, the selection is the same for all samples in a CU.

[0262] BCW asymmetric weight set

[0263] According to this method, an asymmetric BCW weight set is proposed. In the initial design, the design of BCW weights is symmetric. That is, if a weight pair {w0, w1} of L0 and L1 is supported in the BCW weight set, the symmetric weight part {w1, w0} of L0 and L1 is also supported. However, in some cases, this design may introduce some redundancy. For example, if the reference frames in L0 and L1 are the same, we can swap the MVs, the corresponding reference pictures, and the selected BCW weights in L0 and L1 and get the same final predictor. To eliminate this redundancy, an asymmetric weight set of BCW weights is proposed. That is, if a weight pair {w0, w1} of L0 and L1 is supported in the BCW weight set, the symmetric weight part {w1, w0} of L0 and L1 will not be supported. That is, the sum of the two weights in any two weight pairs cannot be the same as the sum of a pair of weights. That is, the sum of the two weights in any two weight pairs cannot be the same as the sum of a pair of weights. For example, the BCW weights were originally {-2, 3, 4, 5, 10}. The sum of the two weights of the first and last weight pair will be -2+10=8. The sum of the two weights of the second and fourth weight pair will be 3+5=8. Therefore, they are symmetric weighted pairs. In the proposed method, the BCW weight set will be like {-1,1,4,5,10}, {-1,2,4,5,10}, {-3,1,4,5,10}, {-3,2,4,5,10} or {-3,1,4,5,10}. The sum of the two weights of any two weight pairs is not equal to 8.

[0264] Interaction between BCW, MVD sign prediction, and adaptive reference picture reordering

[0265] According to this method, the process of MVD symbol prediction and adaptive reference pictures is independent of BCW. That is, MVD symbol prediction and adaptive reference pictures can always be applied to the current CU regardless of the choice of BCW. However, the choice of BCW is designed for the current MV pair, and it may be detrimental to MVD symbol prediction and adaptive reference picture reordering because these codec tools improve signaling by considering only one of the current MV pair. To improve this, we propose the following method.

[0266] In one embodiment, the use of BCW and MVD symbol prediction (and / or adaptive reference picture reordering) are mutually exclusive. That is, when a pair of unequal weights is used in BCW, MVD symbol prediction (and / or adaptive reference picture reordering) cannot be used.

[0267] In one embodiment, the use of BCW and MVD symbol prediction (and / or adaptive reference picture reordering) is mutually exclusive only when some specific unequal weight pairs are selected. For example, when one of the weight pairs selected in the BCW is negative, the use of BCW and MVD symbol prediction (and / or adaptive reference picture reordering) is mutually exclusive. Otherwise, BCW and MVD symbol prediction (and / or adaptive reference picture reordering) can be applied together. In another embodiment, the use of BCW and MVD symbol prediction (and / or adaptive reference picture reordering) is mutually exclusive when the difference between the two weights in the weight pair selected in the BCW is greater than a predefined threshold. Otherwise, BCW and MVD symbol prediction (and / or adaptive reference picture reordering) can be applied together.

[0268] In another embodiment, CABAC context selection and / or context modeling for MVD symbol prediction (and / or adaptive reference picture reordering) depends on the selection of BCW. For example, there are two sets of CABAC contexts for MVD symbol prediction and adaptive reference picture reordering. One for BCWs of equal weight and another for BCWs of unequal weight.

[0269] In another embodiment, the use of BCW and MVD symbol prediction (and / or adaptive reference picture reordering) can be applied together. This embodiment uses bidirectional template pairing. The encoder or decoder uses reference samples from two or more inputs (e.g., two different pictures or the same picture with different blocks) to generate a bidirectional prediction template. The template matching cost is calculated by using neighboring reconstructed samples and the bidirectional prediction template. When generating the bidirectional prediction template, the inherited BCW weighting is taken into account. The bidirectional prediction template taking into account the BCW weighting is used for the MVD symbol prediction (and / or adaptive reference picture reordering) process.

[0270] Any of the above-mentioned proposed methods for determining the inter-frame direction based on the template matching cost can be implemented in the codec and / or the decoder. For example, any of the proposed methods can be implemented in the inter-frame coding module (e.g. Figure 1A Intra-frame prediction 112 in ) or motion compensation module (e.g. Figure 1B Since MVP may involve merging / AMVP candidate construction, the present invention or a part of the present invention may also be implemented in a merge candidate derivation module of an encoder or decoder. Alternatively, any of the proposed methods may be implemented as a circuit coupled to an inter-frame coding module and / or an entropy coding module and / or a merge candidate derivation module at an encoder and / or a decoder.

[0271] Fig.17A flowchart of an exemplary video coding system for determining the inter-frame direction of MVP candidates according to a template matching cost according to an embodiment of the present invention is shown. The steps shown in the flowchart can be implemented as executable program codes on one or more processors (e.g., one or more CPUs) on the encoder side. The steps shown in the flowchart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to execute the steps in the flowchart. According to the method, in step 1710, input data associated with a current block is received, wherein the input data includes pixel data of the current block to be encoded on the encoder side or encoded data associated with the current block to be decoded on the decoder side. And wherein the motion vector prediction (Motion Vector Prediction, abbreviated as MVP) candidate of the current block includes a first MV predictor pointing to a first reference block in an L0 reference picture and a second MV predictor pointing to a second reference block in an L1 reference picture. In step 1720, the L0 matching cost between a first template corresponding to one or more first neighboring regions of the first reference block and a current template corresponding to one or more current neighboring regions of the current block is determined. In step 1730, the L1 matching cost between a second template corresponding to one or more second neighboring regions of the second reference block and the current template is determined. In step 1740, an inter direction of an MVP candidate of the current block is determined based on the first information including the L0 matching cost and the L1 matching cost, the inter direction corresponding to bidirectional prediction, L0 unidirectional prediction or L1 unidirectional prediction. In step 1750, the MVP candidate is inserted into an AMVP (Adaptive MVP) list or a merge list. In step 1760, the current block is encoded or decoded by using the second information including the MVP list or the merge list.

[0272] The flowchart shown is intended to illustrate an example of video coding according to the present invention. Without departing from the spirit of the present invention, those skilled in the art may modify each step, rearrange the steps, split the steps, or combine the steps to implement the present invention. In the present disclosure, specific syntax and semantics have been used to illustrate examples to implement embodiments of the present invention. Without departing from the spirit of the present invention, those skilled in the art may implement the present invention by replacing syntax and semantics with equivalent syntax and semantics.

[0273] The above description is provided to enable those skilled in the art to practice the present invention provided in the context of a specific application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described, but to conform to the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are illustrated to provide a thorough understanding of the present invention. However, those skilled in the art will appreciate that the present invention may be implemented.

[0274] The embodiments of the present invention as described above can be implemented in various hardware, software codes or a combination of the two. For example, one embodiment of the present invention can be one or more circuits integrated into a video compression chip or a program code integrated into the video compression software to perform the processing described herein. The embodiments of the present invention can also be a program code to be executed on a digital signal processor (DSP) to perform the processing described herein. The present invention can also be related to many functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array (FPGA). These processors can be configured to perform specific tasks according to the present invention by executing machine-readable software codes or firmware codes that define the specific methods embodied by the present invention. The software code or firmware code can be developed in different stylized languages ​​and different formats or styles. It is also possible to compile software code for different target platforms. However, the different code formats, styles and languages ​​of the software code and other ways of configuring the code to perform tasks according to the present invention will not depart from the spirit and scope of the present invention.

[0275] The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples should be considered in all respects as illustrative and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the foregoing description. All changes that fall within the meaning and range of equivalents of the claims should be included within their scope.

Claims

1. A video encoding and decoding method, the method comprising: Receiving input data associated with a current block, wherein the input data includes pixel data of the current block to be encoded at an encoder side or coded data associated with the current block to be decoded at a decoder side, and wherein motion vector prediction candidates for the current block include a first MV predictor pointing to a first reference block in an L0 reference picture and a second MV predictor pointing to a second reference block in an L1 reference picture; Determining an L0 matching cost between a first template corresponding to one or more first neighboring regions of the first reference block and a current template corresponding to one or more current neighboring regions of the current block; Determining an L1 matching cost between a second template corresponding to one or more second neighboring regions of the second reference block and the current template; Determine an inter-frame direction of the MVP candidate of the current block based on the first information including the L0 matching cost and the L1 matching cost, wherein the inter-frame direction corresponds to bidirectional prediction, L0 unidirectional prediction or L1 unidirectional prediction; Insert the candidate MVP into the AMVP list or the merged list; and The current block is encoded or decoded by using second information including the AMVP list or the merge list.

2. The method according to claim 1, characterized in that A bi-prediction matching cost between a hybrid template and the current template is calculated, and wherein the hybrid template is derived by mixing the first template and the second template.

3. The method according to claim 2, characterized in that The bi-prediction matching cost, L0 matching cost and L1 matching cost are calculated based on the sum of absolute differences or the sum of squared differences.

4. The method according to claim 2, characterized in that: The bidirectional prediction matching cost is weighted by a factor less than 1 for matching cost comparison among the L0 matching cost, the L1 matching cost, and the bidirectional prediction matching cost.

5. The method according to claim 2, characterized in that: The inter-frame direction of the MVP candidate is determined for the current block based on each sample.

6. The method according to claim 5, characterized in that If the difference between the corresponding L0 reference sample of the target sample and the corresponding L1 reference sample of the target sample is greater than a threshold, the template sample in the current block is a candidate for changing from the bidirectional prediction to the unidirectional prediction.

7. The method according to claim 6, characterized in that If the L0 matching cost is the smallest of the L0 matching cost, the L1 matching cost and the bidirectional prediction matching cost, the target sample is changed to the L0 unidirectional prediction, or if the L1 matching cost is the smallest of the L0 matching cost, the L1 matching cost and the bidirectional prediction matching cost, the target sample is changed to the L1 unidirectional prediction.

8. The method according to claim 6, characterized in that If the bidirectional prediction matching cost is the smallest one among the L0 matching cost, the L1 matching cost and the bidirectional prediction matching cost, the target sample keeps using the bidirectional prediction.

9. The method according to claim 6, characterized in that The threshold is either fixed or determined from a set of candidate thresholds.

10. The method according to claim 6, characterized in that The threshold is adaptively determined for the current block.

11. The method according to claim 10, characterized in that The threshold is adaptively selected from the set of candidate thresholds based on the L0 matching cost, the L1 matching cost and the bidirectional prediction matching cost calculated for each of a set of candidate thresholds, and the threshold corresponds to a target candidate threshold that achieves the lowest matching cost in the set of candidate thresholds.

12. The method according to claim 10, characterized in that The threshold is adaptively selected from a set of candidate thresholds based on an increment number associated with each of the set of candidate thresholds, and the increment number is calculated as an increment from a current total number of samples where the absolute difference between the L0 predictor and the L1 predictor is less than the current candidate threshold to a next total number of samples where the absolute difference between the L0 predictor and the L1 predictor is less than the next candidate threshold, and the threshold corresponds to a target candidate threshold with the largest increment number in the set of candidate thresholds.

13. The method according to claim 12, characterized in that The incremental number associated with each of the group of candidate thresholds is calculated for the current template, the first template and the second template respectively to determine three corresponding thresholds of the current template, the first template and the second template, and if the three corresponding thresholds are the same, the method of determining the inter-frame direction of the MVP candidate of the current block based on the first information including the L0 matching cost and the L1 matching cost is applied; if the three corresponding thresholds are not the same, the method of determining the inter-frame direction of the MVP candidate of the current block based on the first information including the L0 matching cost and the L1 matching cost is not applied.

14. The method according to claim 6, characterized in that The threshold depends on a quantization parameter of the current block, a block size of the current block, one or more template matching costs of the current block, a number of samples in the current block whose differences between corresponding L0 and L1 reference samples fall within respective threshold intervals, or a combination thereof.

15. The method according to claim 6, characterized in that The smaller the quantization parameter of the current block is, the smaller the threshold is set to.

16. The method according to claim 6, characterized in that The current block is processed via bidirectional optical flow or multi-channel decoder-side motion vector refinement.

17. The method according to claim 6, characterized in that The MVP candidate of the current block corresponds to an inter-frame AMVP candidate, an inter-frame merge candidate, an affine AMVP candidate, an affine merge candidate, or a sub-block based temporal motion vector prediction candidate.

18. The method according to claim 6, characterized in that The current block corresponds to a luminance block or a chrominance block.

19. The method according to claim 18, characterized in that The threshold value is different between the luma block and the chroma block.

20. The method according to claim 6, characterized in that A rate-distortion cost of the current block with and without the per-sample inter-direction determination is calculated to determine whether to apply the per-sample inter-direction determination to the current block.

21. The method according to claim 20, characterized in that Three RD costs are calculated, and a decision on whether to use the per-sample inter direction and the corresponding inter direction associated with the minimum RD cost are selected for the current block, and a first RD cost corresponds to encoding and decoding the current block using the bidirectional prediction, and a second RD cost and a third RD cost correspond to encoding and decoding the current block using the L0 unidirectional prediction and the L1 unidirectional prediction, respectively, for samples in the current block whose corresponding L0 and L1 reference samples have a difference greater than a threshold.

22. The method according to claim 21, characterized in that Two bits are signaled to indicate whether the per-sample based inter-direction determination and the corresponding inter-direction are used.

23. The method according to claim 20, characterized in that Two RD costs are calculated and a decision is determined for the current block as to whether to use the per-sample inter direction determination, and wherein a first RD cost corresponds to encoding and decoding the current block using the bidirectional prediction, and a second RD cost corresponds to encoding and decoding the current block using the L0 uniprediction or the L1 uniprediction according to the L0 matching cost and the L1 matching cost for samples in the current block where the difference between corresponding L0 and L1 reference samples is greater than the threshold.

24. The method according to claim 23, characterized in that One bit is signaled to indicate whether the per-sample based inter-frame direction determination is to be used.

25. The method according to claim 2, characterized in that The inter-frame direction of the MVP candidate is determined on a per-block basis for the current block.

26. The method according to claim 25, characterized in that The MVP candidate corresponds to a regular merge candidate, a geometric partition mode candidate, a merged motion vector difference candidate, a bilateral matching candidate or an affine candidate, a CIIP candidate.

27. An apparatus for video encoding and decoding, the apparatus comprising one or more electronic devices or processors configured to: Receive input data associated with the current block, where The input data includes pixel data of the current block to be encoded at the encoder side or encoded data associated with the current block to be decoded at the decoder side, and wherein the motion vector prediction candidate of the current block includes a first MV predictor pointing to a first reference block in the L0 reference picture and a second MV predictor pointing to a second reference block in the L1 reference picture; Determining an L0 matching cost between a first template corresponding to one or more first neighboring regions of the first reference block and a current template corresponding to one or more current neighboring regions of the current block; Determining an L1 matching cost between a second template corresponding to one or more second neighboring regions of the second reference block and the current template; Determine an inter-frame direction of the MVP candidate of the current block based on the first information including the L0 matching cost and the L1 matching cost, wherein the inter-frame direction corresponds to bidirectional prediction, L0 unidirectional prediction or L1 unidirectional prediction; Insert the candidate MVP into the AMVP list or the merged list; and The current block is encoded or decoded by using second information including the AMVP list or the merge list.

Images