Method and apparatus for decoder-side motion derivation in video coding system

Abstract

The invention discloses a video encoding and decoding method and device. According to the method, at least one of a first MVP and a second MVP of a current block is determined. At least one of a first MVD associated with the first MVP and a second MVD associated with the second MVP from the at least one predefined set of MVD candidates is determined based on the matching cost. Each matching cost is determined between neighboring samples of the current block and prediction samples from corresponding neighboring samples of each reference block associated with a candidate of the at least one set of predefined MVD candidates. And encoding or decoding the current block by using motion information including at least one of a first final MV associated with the first MVP and the first MVD and a second final MV associated with the second MVP and the second MVD.

Description

[0001] Cross-references

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 336,378 filed on April 29, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to a video coding system. In particular, the present invention relates to reducing the signaling overhead associated with the MV Difference (MVD) of a Motion Vector Predictor (MVP) by using template matching. Background Art

[0004] Versatile Video Coding (VVC) is the latest international video coding standard jointly developed by the Joint Video Experts Group (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). The standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Information technology - Coded representation of immersive media - Part 3: Versatile video coding, published in February 2021. VVC is based on its predecessor HEVC (High Efficiency Video Coding), by adding more codec tools to improve codec efficiency, and can also handle various types of video sources, including 3D video signals.

[0005] Figure 1A An exemplary adaptive inter / intra video coding system including a loop process is described. For intra prediction, prediction data is derived from previously encoded video data in the current picture. For inter prediction 112, motion estimation (ME) is performed on the encoder side and motion compensation (MC) is performed based on the results of ME to provide prediction data derived from other pictures and motion data. Switch 114 selects intra prediction 110 or inter prediction 112 and the selected prediction data is provided to adder 116 to form a prediction error, also known as a residual. The prediction error is then processed by transform (T) 118 and subsequent quantization (Q) 120. The transformed and quantized residual is then encoded by entropy encoder 122 to be included in a video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged with auxiliary information (such as motion and coding modes associated with intra prediction and inter prediction) and other information (such as parameters associated with loop filters applied to underlying image areas). Auxiliary information associated with intra prediction 110, inter prediction 112, and in-loop filter 130 is provided to entropy encoder 122, such as Figure 1AAs shown. When using inter-frame prediction mode, one or more reference pictures must also be reconstructed at the encoder end. Therefore, the transformed and quantized residual is processed by inverse quantization (IQ) 124 and inverse transform (IT) 126 to recover the residual. The residual is then added back to the prediction data 136 at reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in a reference picture buffer 134 and used to predict other frames.

[0006] like Figure 1A As shown, the input video data undergoes a series of processes in the encoding system. Due to the series of processes, the reconstructed video data from REC128 may be subject to various impairments. Therefore, the loop filter 130 is often applied to the reconstructed video data before it is stored in the reference picture buffer 134 to improve the video quality. For example, a deblocking filter (DF), sample adaptive offset (SAO), and adaptive loop filter (ALF) may be used. It may be necessary to merge the loop filter information into the bitstream so that the decoder can correctly recover the required information. Therefore, the loop filter information is also provided to the entropy encoder 122 to be merged into the bitstream. Figure 1A In , a loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in a reference picture buffer 134 . Figure 1A The system in is intended to illustrate an exemplary structure of a typical video encoder. It may correspond to the High Efficiency Video Coding (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0007] like Figure 1B As shown, the decoder can use similar or identical functional blocks as the encoder, except for transform 118 and quantization 120, because the decoder only needs inverse quantization 124 and inverse transform 126. Instead of entropy encoder 122, the decoder uses entropy decoder 140 to decode the video bitstream into quantized transform coefficients and required encoding information (such as ILPF information, intra-frame prediction information, and inter-frame prediction information). Intra-frame prediction 150 on the decoder side does not need to perform mode search. Instead, the decoder only needs to generate intra-frame prediction based on the intra-frame prediction information received from entropy decoder 140. In addition, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the inter-frame prediction information received from entropy decoder 140 without motion estimation.

[0008] According to VVC, similar to HEVC, the input picture is divided into non-overlapping square block areas called CTUs (Coding Tree Units). Each CTU can be divided into one or more coding units (CUs) of smaller size. The generated CU partitions can be square or rectangular. In addition, VVC divides CTUs into prediction units (PUs) as units for applying prediction processes such as inter-frame prediction, intra-frame prediction, etc.

[0009] In video coding, the MVD (MV difference) between the final MV and the MVP (MV predictor) is signaled. The system can use more MVDs to improve coding performance. However, more MVDs will require more signaling overhead. In the present invention, template matching is used to help reduce the signaling overhead associated with the MVD of one or more MVPs (MV predictors). Summary of the invention

[0010] A method and apparatus for video encoding and decoding are disclosed. According to the method, data associated with a current block is received at an encoder side or encoded data associated with a current block to be decoded is received at a decoder side. The current block is encoded using unidirectional prediction or bidirectional prediction. At least one of a first motion vector predictor (MVP) and a second MVP of the current block is determined. At least one of a first MVD (MV difference) associated with the first MVP and a second MVD associated with the second MVP is determined from at least one set of predefined MVD candidate sets based on a matching cost. The derivation of the matching cost depends on whether the current block is encoded and decoded using unidirectional prediction or bidirectional prediction. If the current block is encoded using unidirectional prediction, each matching cost is determined between one or more neighboring samples of the current block and a plurality of prediction samples, wherein the one or more prediction costs come from one or more corresponding neighboring samples of each reference block pointed to by the unidirectional prediction candidate MV. The unidirectional prediction candidate MV is a prediction candidate MV based on a candidate in the at least one predefined MVD candidate set and one of the first MVP and the second MVP. If the current block is encoded and decoded using bidirectional prediction, each matching cost is determined between one or more neighboring samples of the current block and one or more prediction samples. The one or more prediction samples are from one or more corresponding neighboring samples of each reference block pointed to by a bidirectional prediction candidate MV, wherein the bidirectional prediction candidate MV is a prediction candidate MV based on at least a first MVP, a second MVP, and a candidate in the at least one predefined MVD candidate set. The current block is encoded or decoded by using motion information including at least one of a first final MV associated with the first MVP and the first MVD and a second final MV associated with the second MVP and the second MVD.

[0011] In one embodiment, in response to encoding and decoding the current block using unidirectional prediction, a unidirectional prediction candidate MV achieving a minimum matching cost is selected to derive at least one of a first final MV and a second final MV.

[0012] In one embodiment, in response to encoding and decoding the current block using bidirectional prediction, a bidirectional prediction candidate MV that achieves a minimum matching cost is selected to derive at least one of the first final MV and the second final MV.

[0013] In one embodiment, in response to encoding and decoding the current block using unidirectional prediction, the at least one set of predefined MVD candidate sets corresponds to only one set of predefined MVD candidate sets for deriving the unidirectional prediction candidate MVs in list 0 or list 1.

[0014] In one embodiment, in response to encoding and decoding the current block using bi-directional prediction, the at least one set of predefined MVD candidates corresponds only to a set of predefined MVD candidates used to derive the bi-directional prediction candidate MV.

[0015] In one embodiment, in response to encoding and decoding the current block using bidirectional prediction, the at least one set of predefined MVD candidate sets corresponds to two separate predefined MVD candidate sets, which are used to derive list 0 MV in the bidirectional prediction MV and list 1 MV in the bidirectional prediction candidate MV.

[0016] In one embodiment, in response to encoding and decoding the current block using unidirectional prediction or bidirectional prediction, one or more candidates in the at least one set of predefined MVD candidate sets are derived from an initial MVD.

[0017] In one embodiment, the initial MVD of List 0 or List 1 is signaled or parsed. In another embodiment, the at least one set of predefined MVD candidates includes one or more candidate members determined based on one or more signs of the initial MVD, one or more values ​​of the initial MVD, or both. In one embodiment, the one or more signs of the initial MVD correspond to plus and minus signs. In another embodiment, the one or more values ​​of the initial MVD correspond to k*(initial MVD) or 0, and where k corresponds to N or 1 / N, and N is a positive integer. In another embodiment, the one or more values ​​of the initial MVD correspond to (initial MVD) ± b, and where b corresponds to an integer or a fraction.

[0018] In one embodiment, the at least one set of predefined MVD candidates includes one or more candidate members determined based on one or more symbols of the initial MVD, and wherein the symbol of the target MVD candidate is defined according to the at least one predefined MVD candidate set and based on the matching cost. In one embodiment, the value of the target MVD candidate value is predefined. In another embodiment, the value of the target MVD candidate is signaled or parsed. In yet another embodiment, one or more grammars related to the value of the target MVD candidate are signaled or parsed at a block level, an SPS level, a PPS level, an APS level, a PH level, a SH level, or a combination thereof.

[0019] In one embodiment, the matching cost corresponds to distortion between the one or more neighboring samples of the current block and one or more corresponding neighboring samples of each reference block, and wherein the distortion is measured using one or more metrics including SATD, SAD, MSE or SSE. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1A An exemplary adaptive inter / intra video coding system incorporating loop processing is shown.

[0021] Figure 1B Shows Figure 1A The corresponding decoder of the encoder in .

[0022] Figure 2 Neighboring blocks used to derive spatial merging candidates for VVC are shown.

[0023] Figure 3 The possible candidate pairs considered for redundancy checking in VVC are described.

[0024] Figure 4 An example of temporal candidate derivation is shown, where scaled motion vectors are derived according to the POC (Picture Order Count) distance.

[0025] Figure 5 The position of the temporal candidate selected between candidates C0 and C1 is shown.

[0026] Figure 6 The distance offset from the starting MV in the horizontal and vertical directions according to the MVD merge mode (MMVD) is marked.

[0027] Fig. 7A An example of an affine motion field of a block described by the motion information (4 parameters) of two control points is shown.

[0028] Figure 7B An example of an affine motion field of a block described by motion information of three control point motion vectors (6 parameters) is shown.

[0029] Figure 8 An example of block-based affine transform prediction is shown, where the motion vector for each 4×4 luma sub-block is derived from the control point MV.

[0030] Fig. 9 An example of deriving inherited affine candidates based on control point MVs of neighboring blocks is shown.

[0031] Fig.10 Examples of locations where candidates for constructing an affine merge pattern are marked.

[0032] Fig.11An example of constructing an affine candidate by combining the translational motion information of each control point from its spatial neighbors and time is shown.

[0033] Fig.12 An example of weight value derivation for combined inter and intra prediction (CIIP) according to the coding modes of top and left neighboring blocks is shown.

[0034] Fig.13 An example of 64 partitions used in the VVC standard is shown, where the partitions are grouped according to their angles and the dashed lines represent redundant partitions.

[0035] Fig.14 An example of unidirectional prediction MV selection for geometric partitioning mode is marked.

[0036] Fig.15 An example of the warp weight ω0 using the geometric partitioning pattern is marked.

[0037] Fig.16 An example of the GPM mixing process based on a discrete ramp function for the mixing region around the boundary is shown.

[0038] Fig.17 An example of a GPM mixing process for GPM mixing in ECM 4.0 is shown.

[0039] Fig.18 Examples of symmetric MVD patterns are marked.

[0040] FIG. 19 shows an example of available IPM candidates: Parallel angle patterns relative to GPM block boundaries (parallel patterns, Fig.19A ), the vertical angle mode relative to the GPM block boundary (vertical mode, Fig.19B ) and planar mode ( Fig.19C ).

[0041] Fig.19D An example of a GPM with intra and intra prediction is shown, where the intra prediction is restricted to reduce the signaling overhead and hardware decoder cost of the IPM.

[0042] Fig. 20A The syntax encoding for a spatial GPM (SGPM) before using the simplification method is shown.

[0043] Fig. 20B An example of a simplified syntax encoding of a spatial GPM (SGPM) is shown.

[0044] Fig.21 An example of the shape of a template used to generate the candidate list is shown.

[0045] Fig. 22An example of template matching for refining an initial MV by searching a region around the initial MV is shown.

[0046] Fig.23 A flowchart of an exemplary video encoding system for selecting an MVD from a set of MVD candidates using template matching according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0047] It will be readily appreciated that the components of the present invention as generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of the embodiments of the systems and methods of the present invention as shown in the figures is not intended to limit the scope of the claimed invention, but is merely representative of selected embodiments of the present invention. References throughout this specification to "one embodiment," "an embodiment," or similar language mean that a particular feature, structure, or characteristic described in conjunction with that embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing throughout this specification do not necessarily all refer to the same embodiment.

[0048] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. However, those skilled in the relevant art will recognize that the present invention may be practiced without one or more specific details, or using other methods, components, etc. In other cases, well-known structures or detailed descriptions of operations are not shown or are not shown to avoid obscuring aspects of the present invention. The illustrated embodiments of the present invention will be best understood with reference to the accompanying drawings, in which like parts are represented by like numbers throughout. The following description is intended to be exemplary only and simply illustrates certain selected embodiments of the apparatus and methods consistent with the present invention as claimed herein.

[0049] The VVC standard incorporates various new coding tools to further improve the coding efficiency over the HEVC standard. Among the various new coding tools, some coding tools related to the present invention are summarized as follows.

[0050] Inter-frame prediction overview

[0051] According to Section 3.4 of JVET-T2002, (Jianle Chen et al., “Algorithmic Description of Generic Video Coding and Test Model 11 (VTM11)”, Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29, 20th Meeting, Teleconference, October 7–16, 2020, Archives: JVET-T2002), for each inter-predicted CU, the motion parameters contain the motion vector, reference picture index, and reference picture list usage flag, as well as additional information required by the new coding features of VVC for inter-predicted sample generation. The motion parameters can be signaled in an explicit or implicit manner. When a CU is coded in skip mode, the CU is associated with one PU and has no significant residual coefficients, no coded motion vector increments or reference picture indexes. To specify the merge mode, the motion parameters of the current CU are obtained from neighboring CUs, including spatial and temporal candidates, and the additional scheduling introduced in VVC. The merge mode can be applied to any inter-predicted CU, not just skip mode. An alternative to merge mode is explicit transmission of motion parameters, where the motion vectors, the corresponding reference picture index for each reference picture list and the reference picture list usage flag, along with other required information, are explicitly sent per CU.

[0052] In addition to the inter-frame coding features in HEVC, VVC also includes many new and improved inter-frame prediction coding tools, listed below:

[0053] –Extended merge prediction

[0054] –Merge mode with MVD(MMVD)

[0055] –Symmetric MVD (SMVD) signalling

[0056] –Affine motion compensated prediction

[0057] – Subblock-based temporal motion vector prediction (SbTMVP)

[0058] –Adaptive motion vector resolution (AMVR)

[0059] – Motion field storage: 1 / 16 luminance sample MV storage and 8x8 motion field compression

[0060] –Bi-prediction with CU-level weight (BCW)

[0061] –Bi-directional optical flow(BDOF)

[0062] –Decoder side motion vector refinement (DMVR)

[0063] –Geometric partitioning mode (GPM)

[0064] – Combined inter and intra prediction (CIIP)

[0065] The following description provides details of those inter prediction methods specified in VVC.

[0066] Extended Merger Forecast

[0067] In VVC, the merge candidate list is constructed by including the following five categories of candidates in sequence:

[0068] 1) Spatial MVP from spatially adjacent CUs

[0069] 2) Time MVP from collocated CU

[0070] 3) History-based MVP from FIFO table

[0071] 4) Paired Average MVP

[0072] 5) Zero MV.

[0073] The size of the merge list is signaled in the sequence parameter set (SPS) header and the maximum allowed size of the merge list is 6. For each CU encoded in merge mode, the index of the best merge candidate is encoded using truncated unary binarization (TU). The first binary string of the merge index (hereafter referred to as bin) is context encoded and bypass encoding is used for the remaining bins.

[0074] This section provides the derivation process of merge candidates for each category. As done in HEVC, VVC also supports parallel derivation of merge candidate lists (or merge candidate lists) for all CUs in a certain size region.

[0075] Spatial candidate derivation

[0076] The derivation of spatial merge candidates in VVC is the same as in HEVC, except that the positions of the first two merge candidates are swapped. Figure 2 Up to four merge candidates (B0, A0, B1, and A1) for the current CU 210 are selected from the candidates at the positions shown. The derived order is B0, A0, B1, A1, and B2. Position B2 is only considered when one or more neighboring CUs at positions B0, A0, B1, A1 are not available (for example, belong to another slice or tile) or are intra-coded. After adding the candidate at position A1, a redundancy check is performed on the addition of the remaining candidates to ensure that candidates with the same motion information are excluded from the list, thereby improving coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only the possible candidate pairs are considered. Figure 3 The pairs are linked by arrows in , and a candidate is added to the list only if the corresponding candidates used for redundancy checking do not have the same motion information.

[0077] Time candidate derivation

[0078] In this step, only one candidate is added to the list. Specifically, in the derivation of the temporal merge candidate for the current CU 410, based on the candidate belonging to Figure 4 The collocated CU 420 of the collocated reference picture shown derives the scaled motion vector. The reference picture list and the reference index of the collocated CU used to derive the reference picture are explicitly signaled in the slice header. Figure 4 As shown by the dotted line in , a scaled motion vector 430 of the temporal merge candidate is obtained, which is scaled from the motion vector 440 of the collocated CU using the POC (Picture Order Count) distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal merge candidate is set equal to zero.

[0079] The position of the temporal candidate is selected between candidates C0 and C1, such as Figure 5If the CU at position C0 is not available, intra-coded, or outside the current CTU row, position C1 is used. Otherwise, position C0 is used to derive the temporal merge candidate.

[0080] History-based Merge Candidate Derivation

[0081] History-based MVP (HMVP) merge candidates are added to the merge list after spatial MVP and TMVP. In this method, the motion information of the previously coded block is stored in a table and used as the MVP of the current CU. A table with multiple HMVP candidates is maintained during the encoding / decoding process. The table will be reset (cleared) when a new CTU row is encountered. Whenever there is a non-sub-block inter-coded CU, the associated motion information will be added to the last entry of the table as a new HMVP candidate.

[0082] The HMVP table size S is set to 6, which indicates that a maximum of 5 history-based MVP (HMVP) candidates can be added to the table. When inserting a new motion candidate into the table, a constrained first-in, first-out (FIFO) rule is used, where a redundancy check is first applied to find if the same HMVP exists in the table. If found, the same HMVP is removed from the table and all subsequent HMVP candidates are moved forward, and the same HMVP is inserted into the last entry of the table.

[0083] HMVP candidates can be used in the merge candidate list construction process. The latest HMVP candidates in the table are checked in turn and inserted into the candidate list after the TMVP candidate. Redundancy check is applied to HMVP candidates to spatial or temporal merge candidates.

[0084] In order to reduce the number of redundant check operations, the following simplifications are introduced:

[0085] 1. The last two entries in the table perform redundancy checks on the A1 and B1 space candidates, respectively.

[0086] 2. Once the total number of available merge candidates reaches the maximum number of allowed merge candidates - 1, the merge candidate list construction process of HMVP is terminated.

[0087] Pairwise average merge candidate derivation

[0088] Paired average candidates are generated by averaging predefined candidate pairs in the existing merge candidate list using the first two merge candidates. The first merge candidate is defined as p0Cand and the second merge candidate can be defined as p1Cand, respectively. The average motion vector is calculated for each reference list separately according to the availability of motion vectors of p0Cand and p1Cand. If both motion vectors are available in one list, they are averaged and their reference pictures are set to one of p0Cand even if the two motion vectors point to different reference pictures; if only one motion vector is available, one is used directly; if no motion vector is available, this list is kept invalid. In addition, if the half-pixel interpolation filter index of p0Cand and p1Cand is different, it is set to 0.

[0089] When the merge list is not full after adding pairwise average merge candidates, zero MVPs are inserted at the end until the maximum number of merge candidates is encountered.

[0090] Merge estimated areas

[0091] The merge estimation region (MER) allows independent derivation of the merge candidate lists of CUs in the same merge estimation region (MER). Candidate blocks in the same MER as the current CU are not included in the generation of the merge candidate list of the current CU. In addition, only when

[0092] (xCb+cbWidth)>>Log2ParMrgLevel is greater than xCb>>Log2ParMrgLevel and (yCb+cbHeight)>>Log2ParMrgLevel is greater than (yCb>>Log2ParMrgLevel), where (xCb, yCb) is the top-left luminance sample position of the current CU in the picture,

[0093] (cbWidth, cbHeight) is the CU size. The MER size is selected at the encoder side and signaled in the sequence parameter set (SPS) as log2_parallel_merge_level_minus2.

[0094] Merge Mode with MVD (MMVD)

[0095] In addition to the merge mode in which the implicitly derived motion information is directly used for prediction sample generation of the current CU, a merge mode with motion vector difference (MMVD) is introduced in VVC. The MMVD flag is sent immediately after the regular merge flag is sent to specify whether the MMVD mode is used for the CU.

[0096] In MMVD, after a merge candidate is selected, it is further refined by signaling MVD information. Further information includes a merge candidate identifier, an index for specifying the magnitude of motion, and an index for indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the MV basis. An MMVD candidate identifier is issued to specify which one to use between the first and second merge candidates.

[0097] The distance index specifies the motion magnitude information and indicates a predefined offset from the start point (612 and 622) of the L0 reference block 610 and the L1 reference block 620. Figure 6 As shown, the offset is added to the horizontal component or vertical component of the starting MV, where small circles of different styles correspond to different offsets from the center. The relationship between the distance index and the predefined offset is specified in Table 1.

[0098] Table 1 – Relationship between distance index and predefined offsets

[0099]

[0100] The direction index indicates the direction of the MVD relative to the starting point. The direction index can represent four directions as shown in Table 2. It should be noted that the meaning of the MVD symbol can change according to the information of the starting MV. When the starting MV is a non-predicted MV or a bidirectionally predicted MV, both lists point to the same side of the current picture (that is, the POCs of both references are greater than the POC of the current picture, or are less than the POC of the current picture), and the symbol in Table 2 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectionally predicted MV, the two MVs point to different sides of the current picture (that is, the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), and the difference in POC in list 0 (which can be abbreviated as list0 or L0) is greater than the POC in list 1 (which can be abbreviated as list1 or L1), the symbol in Table 2 specifies that the sign of the MV offset of the MV component of list 0 added to the starting MV has an opposite value to the sign of the MV in list 1. Otherwise, if the difference in POC in List 1 is greater than List 0, then the sign in Table 2 specifies that the sign of the MV offset added to the List 1 MV component of the starting MV has the opposite value to that of the List 0 MV.

[0101] The MVD is scaled by the difference in the POC in each direction. If the difference in the POC is the same in both lists, no scaling is required. Otherwise, if the difference in the POC in list 0 is greater than that in list 1, the MVD of list 1 is scaled by defining the POC difference of L0 as td and the POC difference of L1 as tb, as Figure 5If the POC difference of L1 is greater than L0, the MVD of list 0 is scaled in the same way. If the starting MV is unidirectionally predicted, the MVD is added to the available MVs.

[0102] Table 2 – MV offset symbols specified by direction index

[0103]

[0104] Affine Motion Compensated Prediction

[0105] In HEVC, only the translational motion model is applied to motion compensation prediction (MCP). In the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensation prediction is applied. As shown in the figure. In Figure 7A-B, the affine motion domain of block 710 is Fig. 7A The motion information of the two control points (4 parameters) in Figure 7B The three control point motion vectors (6 parameters) are described in .

[0106] For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

[0107]

[0108] For the 6-parameter affine motion model, the motion vector at a sample position (x, y) in the block is derived as:

[0109]

[0110] Where (mv 0x , mv 0y ) is the motion vector of the upper left control point, (mv 1x , mv 1y ) is the motion vector of the upper right corner control point, (mv 2x , mv 2y ) is the motion vector of the bottom-left corner control point.

[0111] To simplify motion compensated prediction, block-based affine transform prediction is applied. To derive the motion vector for each 4×4 luminance sub-block, the motion vector of the center sample of each sub-block is Figure 8 As shown, it is calculated according to the above equation and rounded to 1 / 16 fractional precision. Then, a motion compensated interpolation filter is applied to generate a prediction for each subblock with a derived motion vector. The subblock size of the chroma component is also set to 4×4. The MV of the 4×4 chroma subblock is calculated as the average of the MVs of the upper left and lower right luminance subblocks in the collocated 8x8 luminance area.

[0112] For translational motion inter prediction, there are also two affine motion inter prediction modes: affine merge mode and affine AMVP mode.

[0113] Affine merge prediction (AF_MERGE)

[0114] AF_MERGE mode can be applied to CUs with width and height greater than or equal to 8. In this mode, the CPMV (control point MV) of the current CU is generated based on the motion information of the spatial neighboring CUs. There can be up to five CPMVP (CPMV prediction) candidates, and an index is signaled to indicate which one to use for the current CU. The following three types of CPVM candidates are used to form the affine merge candidate list:

[0115] – Inherited affine merge candidates inferred from the CPMV of neighboring CUs

[0116] – Construct affine merge candidate CPMVP derived from the translation MV of the adjacent CU

[0117] –Zero MV

[0118] In VVC, there are at most two inherited affine candidates, which come from the affine motion models of neighboring blocks, one from the left neighboring CU and one from the upper neighboring CU. Figure 2 The same as shown in Figure 1. For the left predictor, the scan order is A0->A1, and for the upper predictor, the scan order is B0->B1->B2. Only the first inherited candidate of each side is selected. No pruning check is performed between two inherited candidates. When a neighboring affine CU is identified, its control point motion vectors are used to derive the CPMVP candidate in the affine merge list of the current CU. Fig. 9 As shown, if the lower left neighboring block A of the current block 910 is encoded in affine mode, the motion vectors v2, v3 and v4 of the upper left corner, upper right corner and lower left corner of the CU 920 including the block A are obtained. When the block A is encoded using the 4-parameter affine model, two CPMVs (i.e., v0 and v1) of the current CU are calculated based on v2 and v3. When the block A is encoded using the 6-parameter affine model, three CPMVs of the current CU are calculated based on v2, v3 and v4.

[0119] Constructing affine candidates means constructing candidates by combining the adjacent translation motion information of each control point. Fig.10 As shown, the motion information of the control point is derived from the specified spatial neighbors and temporal neighbors of the current block 1010. CPMV k(k=1, 2, 3, 4) indicates the kth control point. For CPMV1, check B2->B3->A2 blocks and use the MV of the first available block. For CPMV2, check B1->B0 blocks, and for CPMV3, check A1->A0 blocks. If TMVP is available, use it as CPMV4.

[0120] After obtaining the MVs of the four control points, an affine merge candidate is constructed based on the motion information. The following combinations of control point MVs are used to construct in order:

[0121] {CPMV1,CPMV2,CPMV3},{CPMV1,CPMV2,CPMV4},{CPMV1,CPMV3,CPMV4},

[0122] {CPMV2,CPMV3,CPMV4},{CPMV1,CPMV2},{CPMV1,CPMV3}

[0123] The combination of 3 CPMVs constructs a 6-parameter affine merge candidate and the combination of 2 CPMVs constructs a 4-parameter affine merge candidate. To avoid the motion scaling process, the relevant combination of control point MVs is discarded if the reference indexes of the control points are different.

[0124] After checking inherited affine merge candidates and constructed affine merge candidates, if the list is still not full, a zero MV is inserted at the end of the list.

[0125] Affine AMVP prediction

[0126] Affine AMVP mode can be applied to CUs with width and height both greater than or equal to 16. A CU-level affine flag is signaled in the bitstream to indicate whether to use affine AMVP mode, and then another flag is signaled to indicate whether to use 4-parameter affine or 6-parameter affine. In this mode, the difference between the current CU's CPMV and its predictor CPMVP is signaled in the bitstream. The affine AVMP candidate list size is 2, and is generated from the following four CPVM candidates in sequence:

[0127] – Inherited affine AMVP candidates inferred from the CPMV of neighboring CUs

[0128] – The constructed affine AMVP candidate CPMVP is derived using the translation MV of the adjacent CU

[0129] –Translation MV from neighboring CU

[0130] –Zero MV

[0131] The order of checking inherited affine AMVP candidates is the same as that of checking inherited affine merge candidates. The only difference is that for AVMP candidates, only affine CUs with the same reference picture as the current block are considered. When inserting inherited affine motion prediction into the candidate list, the cropping process is not applied.

[0132] The constructed AMVP candidates are Fig.10 Derived from the specified spatial neighbors shown in . The same checking order as in the affine merge candidate construction is used. In addition, the reference picture indexes of the neighboring blocks are also checked. In the checking order, the first block that uses inter coding and has the same reference picture as in the current CU. When the current CU is encoded using a 4-parameter affine mode and both mv0 and mv1 are available, they are added to the affine AMVP list as one candidate. When the current CU is encoded using a 6-parameter affine mode and all three CPMVs are available, they are added to the affine AMVP list as one candidate. Otherwise, the constructed AMVP candidate is set to unavailable.

[0133] If the number of affine AMVP candidate lists is still less than 2 after inserting valid inherited affine AMVP candidates and constructed AMVP candidates, mv0, mv1 and mv2 are added as translation MVs to predict all control point MVs of the current CU when available. Finally, if the affine AMVP list is still not full, it is filled with zero MVs.

[0134] Affine motion information storage

[0135] In VVC, the CPMV of an affine CU is stored in a separate buffer. The stored CPMV is only used to generate inherited PMVP for the most recently encoded CU in affine merge mode and affine AMVP mode. The sub-block MV derived from the CPMV is used for motion compensation, merging of translation MVs / MV derivation of AMVP lists, and deblocking.

[0136] To avoid an extra CPMV picture row buffer, the affine motion data inherited from the CU of the above CTU is treated differently to inheritance from a normal neighboring CU. If the candidate CU for affine motion data inheritance is in the above CTU row, the bottom left and bottom right sub-block MVs in the row buffer are used for affine MVP derivation instead of the CPMVs. In this way, the CPMVs are only stored in the local buffer. If the candidate CU is 6-parameter affine coded, the affine model degenerates to a 4-parameter model. Fig.11 As shown, along the top CTU boundary, the bottom left and bottom right sub-block motion vectors of the CU are used for affine inheritance of the CU in the bottom CTU. Fig.11, lines 1110 and 1112 represent the x and y coordinates of the picture with the origin (0, 0) at the upper left corner. Legend 1120 shows the meaning of various motion vectors, where arrow 1122 represents the CPMV for affine inheritance in the local buffer, arrow 1124 represents the sub-block vector for MC / Merge / Skip MVP / Deblocking / TMVP in the local buffer and affine inheritance in the row buffer, and arrow 1126 represents the sub-block vector for MC / Merge / Skip MVP / Deblocking / TMVP.

[0137] Adaptive Motion Vector Resolution (AMVR)

[0138] In HEVC, when use_integer_mv_flag in the slice header is equal to 0, the motion vector difference (MVD) (between the CU's motion vector and the predicted motion vector) is signaled in units of quarter-luminance samples. In VVC, a CU-level adaptive motion vector resolution (AMVR) scheme is introduced. AMVR allows the MVD of a CU to be encoded with different precisions. Depending on the mode of the current CU (normal AMVP mode or affine AVMP mode), the MVD of the current CU can be adaptively selected as follows:

[0139] – Normal AMVP mode: quarter brightness samples, half brightness samples, integer brightness samples, or quad brightness samples.

[0140] – Affine AMVP mode: quarter luma samples, integer luma samples, or 1 / 16 luma samples.

[0141] If the current CU has at least one non-zero MVD component, the CU-level MVD resolution indication is conditionally signaled.If all MVD components (ie, horizontal and vertical MVDs for reference list L0 and reference list L1) are zero, a quarter luma sample MVD resolution is inferred.

[0142] For CUs with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MVD precision is used for the CU. If the first flag is 0, no further signaling is required and quarter luma sample MVD precision is used for the current CU. Otherwise, a second flag is issued to indicate that half luma samples or other MVD precision (integer or four luma samples) are used for normal AMVP CUs. In the case of half luma samples, a 6-tap interpolation filter is used for the half luma sample positions instead of the default 8-tap interpolation filter. Otherwise, a third flag is signaled to indicate whether integer luma samples or four luma sample MVD precision is used for normal AMVP CUs. In the case of affine AMVP CUs, the second flag is used to indicate whether integer luma samples or 1 / 16 luma sample MVD precision is used. To ensure that the reconstructed MV has the expected precision (quarter luma samples, half luma samples, integer luma samples, or four luma samples), the motion vector predictor of the CU is rounded to the same MVD precision before being added to the MVD. Motion vector prediction values ​​are rounded towards zero (i.e., negative motion vector prediction values ​​are rounded towards positive infinity, and positive motion vector prediction values ​​are rounded towards negative infinity).

[0143] The encoder uses RD check to determine the motion vector resolution of the current CU. In order to avoid always performing four CU-level RD checks for each MVD resolution, the RD check for MVD precision other than quarter luma samples is only called conditionally in VTM11. For normal AVMP mode, the RD cost of quarter luma sample MVD precision and integer luma sample MV precision is first calculated. Then, the RD cost of integer luma sample MVD precision is compared with the RD cost of quarter luma sample MVD precision to decide whether it is necessary to further check the RD cost of four luma sample MVD precision. When the RD cost of quarter luma sample MVD precision is much smaller than the RD cost of integer luma sample MVD precision, the RD check of four luma sample MVD precision is skipped. Then, if the RD cost of integer luma sample MVD precision is significantly greater than the best RD cost of the previously tested MVD precision, the check of half luma sample MVD precision is skipped. For the affine AMVP mode, if the affine inter mode, affine AMVP mode is not selected after checking the rate-distortion cost of the affine merge / skip mode, merge / skip mode, quarter luma sample MVD accuracy normal AMVP mode, and quarter luma sample MVD accuracy, the 1 / 16 luma sample MV accuracy and 1 pixel MV accuracy affine inter modes are not checked. In addition, the affine parameters obtained in the quarter luma sample MV accuracy affine inter mode are used as the starting search points for the 1 / 16 luma sample and quarter luma sample MV accuracy affine inter modes.

[0144] Bidirectional prediction with CU-level weights (BCW)

[0145] In HEVC, the bidirectional prediction signal P bi-pred is generated by averaging two prediction signals P0 and P1 obtained from two different reference pictures and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging to allow weighted averaging of the two prediction signals.

[0146] P bi-pred =((8-w)*P0+w*P1+4)>>3 (3)

[0147] Weighted average bidirectional prediction allows five weights, w∈{-2,3,4,5,10}. For each bidirectionally predicted CU, the weight w is determined in one of two ways: 1) For non-merged CUs, the weight index is signaled after the motion vector difference; 2) For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applicable to CUs with 256 or more luma samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency pictures, all 5 weights are used. For non-low-latency pictures, only 3 weights (w∈{3,4,5}) are used. At the encoder, a fast search algorithm is applied to find the weight index without significantly increasing the complexity of the encoder. These algorithms are summarized below. The detailed information is disclosed in the VTM software and the document JVET-L0646 (Yu-Chi Su et al., "CE4 correlation: Generalized bidirectional prediction improvement combining JVET-L0197 and JVET-L0296", Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC1 / SC 29, 12th Meeting: Macau, China, October 3-12, 2018, Archives: JVET-L0646).

[0148] When combined with AMVR, unequal weights are only checked conditionally for 1-pixel and 4-pixel motion vector precision if the current picture is a low-latency picture.

[0149] When combined with affine, affine ME is performed on unequal weights if and only if the affine mode is selected as the current best mode.

[0150] Unequal weights are only conditionally checked when the two reference pictures in bidirectional prediction are the same.

[0151] Unequal weights are not searched when certain conditions are met, which depend on the POC distance between the current picture and its reference pictures, the coding QP, and the temporal level.

[0152] The BCW weight index is encoded using a context coded bin and a bypass coded bin. The first context coded bin indicates whether equal weights are used; if unequal weights are used, bypass coding is used to signal an additional box to indicate which unequal weights are used.

[0153] Weighted prediction (WP) is a coding tool supported by the H.264 / AVC and HEVC standards for efficiently encoding video content with fading. Support for WP has also been added to the VVC standard. WP allows weight parameters (weights and offsets) to be sent for each reference picture in each reference picture list L0 and L1. Then, during motion compensation, the weights and offsets of the corresponding reference pictures are applied. WP and BCW are designed for different types of video content. To avoid interaction between WP and BCW, which would complicate the VVC decoder design, if a CU uses WP, the BCW weight index is not notified and the weight w is inferred to be 4 (i.e., equal weights are applied). For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. This can be applied to both normal merge mode and inherited affine merge mode. For constructed affine merge mode, affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index of a CU using constructed affine merge mode is simply set equal to the BCW index of the first control point MV.

[0154] In VVC, CIIP and BCW cannot be jointly applied to a CU. When a CU is encoded using CIIP mode, the BCW index of the current CU is set to 2, that is, w = 4 for equal weights. Equal weights imply the default value of the BCW index.

[0155] Combined Inter and Intra Prediction (CIIP)

[0156] In VVC, when a CU is encoded in merge mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and if both the CU width and the CU height are less than 128 luma samples, an additional flag is sent to indicate whether the combined inter / intra prediction (CIIP) mode is applied to the current CU. As its name suggests, CIIP prediction combines the inter prediction signal with the intra prediction signal. CIIP mode P inter The inter prediction signal in is derived using the same inter prediction process applied to the normal merge mode; and the intra prediction signal P intta is derived after the conventional intra prediction process with planar mode. Then, the intra and inter prediction signals are combined using weighted averaging, where the weight value wt is calculated based on the top and left neighboring blocks of the current CU 1210 (e.g. Fig.12 The encoding mode of ) is calculated as follows:

[0157] If the top neighbor is available and intra-coded, set isIntraTop to 1, otherwise set isIntraTop to 0;

[0158] If the left neighbor is available and intra-coded, set isIntraLeft to 1, otherwise set isIntraLeft to 0;

[0159] If (isIntraLeft+isIntraTop) is equal to 2, then wt is set to 3;

[0160] Otherwise, if (isIntraLeft+isIntraTop) is equal to 1, wt is set to 2;

[0161] Otherwise, set wt to 1.

[0162] The CIIP forecast is formed as follows:

[0163] P CIIP =((4-wt)*P inter +wt*P intra +2)>>2 (4)

[0164] Geometric Partitioning Mode (GPM)

[0165] In VVC, the Geometric Partitioning Mode (GPM) can support inter-frame prediction, such as JVET-W2002 (Adrian Browne, et al., Algorithm description for Versatile Video Coding and Test Model 14 (VTM 14), ITU-T / ISO / IEC Joint Video Exploration Team (JVET), 23rd Meeting, by teleconference, 7–16 July 2021, document: document JVET-M2002). The geometric partitioning mode uses a CU-level flag as a merge mode. Other merge modes include regular merge mode, MMVD mode, CIIP mode, and sub-block merge mode. For each possible CU size, the geometric partitioning mode supports a total of 64 partitions, w×h=2m×2n where m,n∈{3…6}, excluding 8x64 and 64x8. The GPM mode can be applied to skip or merge CUs whose size is within the above limits and has at least two regular merge modes.

[0166] When this mode is used, the CU is divided into two parts by a straight line that is geometrically positioned at a certain angle. In VVC, there are a total of 20 angles and 4 offset distances used for GPM, which has been reduced from 24 angles in the early draft. The location of the partition line is mathematically derived based on the angle and offset parameters of the specific partition. In VVC, there are a total of 64 partitions, such as Fig.13 As shown in Figure 1, where partitions are grouped by angle and dotted lines indicate redundant partitions. Each part of the geometric partition in the CU uses its own motion for inter-frame prediction; each partition only allows unidirectional prediction, that is, each part has a motion vector and a reference index. Fig.13 , each line corresponds to the boundary of a partition. Partitions are grouped according to their angles. For example, partition group 1310 consists of three vertical GPM partitions (i.e., 90°). Partition group 1320 consists of four inclined GPM partitions at a small angle to the vertical direction. In addition, partition group 1330 consists of three vertical GPM partitions (i.e., 270°), similar to group 1310, but in the opposite direction. Unidirectional prediction motion constraints are applied to ensure that each CU only requires two motion compensated predictions, the same as traditional bidirectional prediction. The unidirectional prediction motion of each partition is derived using the process described later.

[0167] If geometry partitioning mode is used for the current CU, a geometry partition index indicating the selected partitioning mode (angle and offset) of the geometry partition and two merge indices (one for each partition) are further sent. The number of maximum GPM candidate sizes is explicitly represented in the Sequence Parameter Set (SPS), as well as the syntax binarization that specifies the GPM merge indices. After predicting each part of the geometry partition, the sample values ​​along the edges of the geometry partition are adjusted by a blending process with adaptive weights using the process described later. This is the prediction signal for the entire CU and, as with other prediction modes, the transform and quantization process will be applied to the entire CU. Finally, using the process described later, the motion field of the CU predicted using the geometry partitioning mode is stored.

[0168] One-way prediction candidate list construction

[0169] The unidirectional prediction candidate list is derived directly from the merge candidate list constructed according to the extended merge prediction process. Denote n as the index of the unidirectional prediction motion in the geometric unidirectional prediction candidate list. The LX motion vector (X=0 or 1, i.e. LX=L0 or L1) of the nth extended merge candidate, where X is equal to the parity of n, is used as the nth unidirectional prediction motion vector geometric partition mode. These motion vectors are in Fig.14 If there is no corresponding LX motion vector of the extended merge candidate, the L(1-X) motion vector of the same candidate replaces the unidirectional prediction motion vector of the geometric partition mode.

[0170] Blending along geometric partition edges

[0171] After predicting each part of the geometric partition using its own motion, blending is applied to the two prediction signals to derive samples around the geometric partition edges. The blending weights for each position of the CU are derived based on the distance between the respective position and the partition edge.

[0172] Two integer mixing matrices (W0 and W1) are used for the GPM mixing process. The weights in the GPM mixing matrix contain a range of [0,8] and are derived based on the displacement from the sample position to the GPM partition boundary 1540, such as Fig.15 shown.

[0173] Specifically, the weights are given by a discrete ramp function with a displacement and two thresholds, as Fig.16 As shown in Figure 2, the two endpoints of the slope (i.e., -τ and τ) correspond to Fig.15 in lines 1542 and 1544.

[0174] Here, the threshold τ defines the width of the GPM mixing area and is selected as a fixed value in VVC. That is, as described in JVET-Z0137 (Han Gao, et.al., "Non-EE2: Adaptive Blending for GPM", Joint Video Experts Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29, 26th Meeting, bytle conference, 20–29 April 2022, JVET-Z0137), the mixing strength or mixing area width θ is fixed for all different contents.

[0175] The weight values ​​in the mixed mask can be given by the gradient function:

[0176]

[0177] With θ=2 pixels fixed in the current ECM (VVC) design, this gradient function can be quantized as:

[0178] ω m,n =Clip3(0,8,(d(m,n)+32+4)>>3) (6)

[0179] The distance from a position (x,y) to the edge of the partition is derived as:

[0180]

[0181] where i,j are the angle and offset indices of the geometry partition, which depend on the geometry partition index sent. The signs of ρx,j and ρy,j depend on the angle index i.

[0182] Fig.17 An example of GPM mixing according to ECM 4.0 (Muhammed Coban, et.al., "Algorithm description of Enhanced Compression Model 4 (ECM 4)", Joint Video Experts Team (JVET) of ITU-TSG 16WP 3 and ISO / IEC JTC 1 / SC 29, 26th Meeting, by teleconference, 20–29 April 2022, JVET-Y2025) is shown. Fig.17 In , the size of the mixed region on each side of the partition boundary is denoted by θ. The weight of each part of the geometric partition is derived as follows:

[0183] wIdxL(x,y)=partIdx? 32+d(x,y):32-d(x,y) (11)

[0184]

[0185] w1(x,y)=1-w0(x,y) (13)

[0186] partIdx depends on the angle index i. An example of weight w0 is Fig.15 As shown, the angle 1510 and offset 1520 indicates the GPM index i, point 1530 corresponds to the center of the block. Row 1540 corresponds to the GPM partition boundary.

[0187] Motion field storage in geometric partitioning mode

[0188] Mv1 from the first part of the geometry partition, Mv2 from the second part of the geometry partition, and the combined MV of Mv1 and Mv2 are stored in the motion field of the CU encoded by the geometry partition mode.

[0189] The type of stored motion vector for each individual position in the motion field is determined as:

[0190] sType=abs(motionIdx)<32?2:(motionIdx≤0?(1-partIdx):partIdx) (14)

[0191] where motionIdx is equal to d(4x+2,4y+2), which is recalculated from equation (7). partIdx depends on the angle index i.

[0192] If sType is equal to 0 or 1, then Mv0 or Mv1 is stored in the corresponding motion field, otherwise if sType is equal to 2, the combined MV from Mv0 and Mv2 is stored. The combined Mv is generated using the following process:

[0193] 1) If Mv1 and Mv2 are from different reference picture lists (one from L0 and the other from L1), then Mv1 and Mv2 are simply combined to form a bi-directional prediction motion vector.

[0194] 2) Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored.

[0195] Symmetric MVD (SMVD) encoding

[0196] In VVC, in addition to normal uni-prediction and bi-prediction mode MVD signaling, a symmetric MVD mode for bi-prediction MVD signaling is applied. In symmetric MVD mode, the motion information including reference picture indexes of both list 0 and list 1 and MVD of list 1 is not signaled but derived.

[0197] The decoding process of the symmetric MVD mode is as follows:

[0198] At the slice level, the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 are derived as follows:

[0199] If mvd_l1_zero_flag is 1, BiDirPredFlag is set equal to 0.

[0200] Otherwise, if the nearest reference picture in list-0 and the nearest reference picture in list-1 form a forward and backward reference picture pair or a backward and forward reference picture pair, BiDirPredFlag is set to 1, and the list-0 and list-1 reference pictures are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0.

[0201] At the CU level, if the CU is bidirectionally predictive coded and BiDirPredFlag is equal to 1, the symmetric mode flag indicating whether the symmetric mode is used is explicitly signaled.

[0202] When the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag and MVD0 are explicitly signaled. The reference index of list-0 and list-1 are set equal to the reference picture pair respectively. MVD1 is set equal to (-MVD0). The final motion vector is shown in the following equation.

[0203]

[0204] In the encoder, symmetric MVD motion estimation starts with initial MV evaluation. A set of initial MV candidates includes MVs obtained from unidirectional prediction search, MVs obtained from bidirectional prediction search, and MVs from the AMVP list. The one with the lowest rate-distortion cost is selected as the initial MV for symmetric MVD motion search.

[0205] Fig.18 An example of a symmetric MVD pattern is shown, where frames 1800, 1810, and 1820 correspond to the current picture, list 0 reference picture, and list 1 reference picture, respectively. Based on the initial MV evaluation, block 1802 corresponds to the current block, and blocks 1814 and 1822 correspond to reference blocks. The final MV is determined by searching the initial MV, and the lowest cost position is selected as the final reference block (1812 and 1819). The MVDs are numbered 1818 and 1826.

[0206] Multi-Hypothesis Prediction (MHP)

[0207] In the multi-hypothesis inter-frame prediction mode (JVET-M0425), in addition to the traditional bidirectional prediction signal, one or more additional motion compensation prediction signals are sent. The final overall prediction signal is obtained by superposition of sample weights. bi and the first additional inter-frame prediction signal / hypothesis h3, the resulting prediction signal p3 is as follows:

[0208] p3=(1-α)p bi +αh3 (17)

[0209] According to the following mapping (Table 3), the weight factor α is composed of the new syntax element

[0210] add_hyp_weight_idx specifies:

[0211] Table 3. Mapping α to add_hyp_weight_idx

[0212] add_hyp_weight_idx α 0 1 / 4 1 -1 / 8

[0213] Similar to the above, more than one additional prediction signal may be used. The resulting overall prediction signal is iteratively accumulated with each additional prediction signal.

[0214] p n+1 =(1-α n+1 ) n +α n+1 h n+1 (18)

[0215] As the last p n (i.e., the p with the largest index n n ) to obtain the resulting overall prediction signal. For example, at most two additional prediction signals can be used (ie, n is limited to 2).

[0216] The motion parameters for each additional prediction hypothesis can be signaled explicitly by specifying the reference index, motion vector predictor index and motion vector difference, or implicitly by specifying the merge index. A separate multi-hypothesis merge flag distinguishes these two signaling modes.

[0217] For inter-AMVP mode, MHP is applied only when non-equal weights in BCW are selected in bi-prediction mode. Details of MHP for VVC can be found in JVET-W2025 (Muhammed Coban et al., "Algorithmic Description of Enhanced Compression Model 2 (ECM 2)", Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC29, 23rd Meeting, Teleconference, July 7-16, 2021, Document: JVET-W2025).

[0218] GPM Extensions

[0219] Several variations of the GPM model (JVET-W0097 (Zhipin Deng, et.al., “AEE2-related: Combination of EE2-3.3, EE2-3.4 and EE2-3.5”, Joint Video Experts Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29, 23rd Meeting, by teleconference, 7–16 July 2021, Document: JVET-W0097) and JVET-Y0065 (Yoshitaka Kidani, et.al., “EE2-3.1: GPM with inter and intra prediction (JVET-X0166)”, Joint Video Experts Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29, 25th Meeting, byteleleconference, 12–21 January 2021) 2022, Document: JVET-Y0065)) have proposed to improve the encoding and decoding efficiency of the GPM mode in VVC. These methods were included in the exploration experiment (EE2) for further evaluation. The main technical aspects are described as follows:

[0220] EE2-3.3 Regarding GPM with MMVD (GPM-MMVD): 1) Additional MVDs are added to the existing GPM merge candidates; 2) The signaling method of MVD is the same as MMVD in VVC, that is, one distance index plus one direction index; 3) Two flags are issued to control whether MMVD is applied to each GPM partition separately.

[0221] EE2-3.4-3.5 Regarding GPM with template matching (GPM-TM): 1) Extend template matching to GPM mode by refining the GPM MV based on the upper left neighboring samples of the current CU; 2) Select template samples according to the GPM split direction; 3) Issue a flag to jointly control whether template matching is applied to the MVs of two GPM partitions.

[0222] JVET-W0097 proposes a combination of EE2-3.3, EE2-3.4 and EE2-3.5 to further improve the encoding and decoding efficiency of the GPM mode. Specifically, in the proposed combination, the existing designs in EE2-3.3, EE2-3.4 and EE2-3.5 remain unchanged, while the following modifications are further made to coordinate the two encoding and decoding tools:

[0223] GPM-MMVD and GPM-TM are specific to one GPM CU. This is done by first sending the GPM-MMVD syntax. When both GPM-MMVD control flags are equal to false (i.e., GPM-MMVD is disabled for both GPM partitions), the GPM-TM flag is sent to indicate whether template matching is applied to both GPM partitions. Otherwise (at least one GPM-MMVD flag is equal to true), the value of the GPM-TM flag is inferred to be false.

[0224] The GPM merge candidate list generation method in EE2-3.3 and EE2-3.4-3.5 is directly combined with the MV pruning scheme in EE2-3.4-3.5 (where the MV pruning threshold is adjusted according to the current CU size) to replace the default MV pruning scheme applied in EE2-3.3; in addition, as in EE2-3.4-3.5, multiple zero MVs are added until the GPM candidate list is completely filled.

[0225] In JVET-Y0065, in GPM with inter-frame and intra-frame prediction (or named as GPM intra-frame), the final prediction sample is generated by weighting the inter-frame prediction sample and the intra-frame prediction sample of each GPM partition area. The inter-frame prediction sample is derived by the same scheme as the GPM in the current ECM, and the intra-frame prediction sample is derived by the intraprediction mode (IPM) candidate list and the index signal sent from the encoder. The IPM candidate list size is pre-defined as 3. As shown in Figure 19, the available IPM candidates are the parallel angle mode (parallel mode) relative to the GPM block boundary, the vertical angle mode (vertical mode) relative to the GPM block boundary, and the planar mode (Planar mode). In addition, as Fig.19D As shown, GPM with intra and intra prediction is restricted in the proposed method to reduce the signaling overhead of IPM and avoid the increase in the size of intra prediction circuit on the hardware decoder. In addition, direct motion vector and IPM storage are introduced in the GPM hybrid region to further improve the codec performance.

[0226] Space GPM

[0227] Similar to inter-frame GPM, spatial GPM (SGPM) consists of a partition mode and two associated intra-frame prediction modes. If these modes are sent directly in the bitstream, such as Fig. 20A As shown in , it will generate a lot of overhead bits. In order to express the necessary partition and prediction information more efficiently in the bitstream, the candidate list is applied and only the candidate index is sent in the bitstream. Fig. 20BAs shown, each candidate in the list can derive a combination of a partition mode and two intra prediction modes.

[0228] A template is used to generate the candidate list. The shape of the template is as follows Fig.21 As shown. For each possible combination of one partition mode and two intra prediction modes, predictions are generated for the template, and the partition weights are extended to the template, as shown in Figure 26. These combinations are ranked in ascending order of their SATD based on the predictions and reconstructions of the template. The length of the candidate list is set to 16, and these candidates are considered to be the most likely SGPM combinations for the current block. Both the encoder and the decoder build the same candidate list based on the template.

[0229] To reduce the complexity of building the candidate list, the number of possible partition modes and the number of possible intra prediction modes are pruned. In the following tests, 26 of the 64 partition modes are used, and only MPM of the 67 intra prediction modes is used.

[0230] Template matching for MV refinement

[0231] Template matching (TM) is a decoder-side MV derivation method used to refine the motion information of the current CU by finding the closest match between templates (i.e., the top 2214 and / or left 2216 neighboring blocks of the current CU 2212). Fig. 22 As shown, the blocks in the current picture 2210 and the reference picture 2220 (ie, blocks 2224 and 2226 of the same size in the template). Fig. 22 In the example, a better MV is searched around the initial MV 2230 of the current picture. The CU 2212 of the current picture 2210 is within a [–8, +8] pixel search range 2222 around the position 2228 in the reference picture 2220 pointed to by the initial MV 2230. The template matching method in JVET-J0021 is used with the following modifications: the search step size is determined according to the AMVR mode, and the TM can be cascaded with the bidirectional matching process in the merge mode.

[0232] In AMVP mode, MVP candidates are determined based on the template matching error to select the MVP candidate with the smallest difference between the current block template and the reference block template. TM is then performed only for this specific MVP candidate for MV refinement. TM refines this MVP candidate by using an iterative diamond search starting with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode) within the [–8, +8] pixel search range. The AMVP candidate can be further refined by using a cross search with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode), followed by half-pixel and quarter-pixel searches in turn according to the AMVR mode specified in Table 4. This search process ensures that the MVP candidate still maintains the same MV accuracy as indicated by the AMVR mode after the TM process. During the search process, if the difference between the previous minimum cost and the current minimum cost in the iteration is less than or equal to the threshold of the block area, the search process terminates.

[0233] Table 4. Search model of AMVR and hybrid mode with AMVR

[0234]

[0235] In merge mode, a similar search method is applied to the blend candidates indicated by the blend index. As shown in Table 4, TM can be performed all the way to 1 / 8 pixel MVD accuracy, or skip accuracy beyond half-pixel MVD accuracy, depending on whether an alternative interpolation filter is used based on the merged motion information (for AMVR in half-pixel mode). In addition, when TM mode is enabled, template matching can be used as an independent process between block-based and sub-block-based bilateral matching (BM) methods or an additional MV refinement process, depending on whether BM can be enabled based on its enabled condition check.

[0236] In VVC, some inter-frame tools are proposed as shortcut modes to reduce the syntax overhead of motion candidates. Initially, for each direction (list 0 or list 1), the bidirectional prediction motion candidate requires the following syntax to obtain its motion information, as shown in Table 5 of the VVC standard:

[0237] - Reference Index

[0238] -MVD

[0239] -MVP Index

[0240] Table 5. Syntax table of motion information

[0241]

[0242]

[0243] When SMVD is enabled, as described in the SMVD encoding section, the reference indexes of List 0 and List 1 are respectively set equal to the reference picture pair, and MVD1 is set equal to (-MVD0) without additional signaling. In order to improve the performance of such shortcut tools, the present invention proposes to perform a derivation process using templates on both the encoder and decoder sides. The key idea behind the present invention is to expand the selection of possible MVDs and use template matching to help select the final MVD candidate in the MVD candidate set, thereby avoiding or reducing signaling overhead. Specifically, the matching cost based on the template of the current block and the corresponding template of the reference block can be evaluated and used to sort the MVD candidates. The MVD that achieves the minimum cost can be selected as the final MVD. Fig. 22 The templates shown include the top template and the left template. However, the template usually covers some of the neighboring samples of the current block that have been encoded. Similarly, the corresponding template of the reference block covers the neighboring samples of the reference block.

[0244] For each MVD candidate, the matching cost between the template of the current block and the reference block determined based on the MVD candidate and MVP is calculated. In other words, the reference block is located based on the MV candidate and the current block position, where the MV candidate is determined based on the MVD candidate and MVP. Similarly, the MV candidate is also used to locate the corresponding template of the reference block.

[0245] Below we take the shortcut tool SMVD as an example. However, the use of SMVD should not be interpreted as a limitation of the present invention. The present invention can be applied to unidirectional prediction or bidirectional prediction. For example, the present invention can be used for any unidirectional prediction or bidirectional prediction inter-frame tool mentioned in the standard or the above-mentioned inter-frame tool. When applied to MHP, CIIP and / or GPM, at least one motion for a predefined direction (list 0 or list 1) and / or at least one motion for one or more predefined prediction assumptions is refined. When applied to affine or sub-block inter-frame prediction mode, the present invention is used to refine at least one motion for a predefined direction (list 0 or list 1) and / or at least one motion for one or more predefined sub-blocks. When applied to BCW, the present invention is used to refine at least one motion for a predefined direction (list 0 or list 1). When applied to MMVD, the present invention is used to refine at least one motion (referring to MVD distance (or offset) and / or MVD direction) in a predefined direction (list 0 or list 1). When using the present invention to refine the motion of List0 and List1, the refinement order is predefined to refine List0 first and then List1 with or without refining List0, or to refine List1 first and then List0 with or without refining List1.

[0246] In one embodiment, a candidate set is predefined according to the MVD0 explicitly signaled or parsed at the encoder or decoder, respectively. Then, according to the derivation process, one candidate is selected from the candidate set as the MVD1 of the current block.

[0247] In a sub-embodiment, the derivation process refers to template matching. A candidate with the smallest template matching error is selected from the candidate set as the MVD1 of the current block.

[0248] In another sub-embodiment, the candidate set depends on signaled / parsed information from List 0. For example, the candidate set depends on MVD0, where MVD0 is also referred to herein as the initial MVD.

[0249] In another sub-embodiment, the candidate set includes -2MVD0, -(1 / 2)*MVD0, 0, (1 / 2)*MVD0, 2*MVD0, or any subset of the mentioned candidates.

[0250] In another sub-embodiment, the candidate set includes -2MVD0, -MVD0, -(1 / 2)*MVD0, 0, (1 / 2)*MVD0, MVD0, 2*MVD0, or any subset of the mentioned candidates.

[0251] In another sub-embodiment, the candidate set includes -4MVD0, -2MVD0, -MVD0, -(1 / 2)*MVD0, -(1 / 4)*MVD0, 0, (1 / 4)*MVD0, (1 / 2)*MVD0, MVD0, 2MVD0, 4MVD0, or any subset of the above candidates.

[0252] In another sub-embodiment, the candidate set includes -kMVD0, -(k / 2)*MVD0, -(k / 4)*MVD0, ..., -MVD0, ..., -(1 / k)*MVD0, 0,

[0253] (1 / k)*MVD0, ...MVD0, 2MVD0, 4MVD0, ..., kMVD0 or any subset of the above candidates. k is a positive integer. For example, k=16, and the candidate set includes -16MVD0, -8MVD0, -4MVD0, -2MVD0, -MVD0, -(1 / 2)*MVD0, -(1 / 4)*MVD0, -(1 / 8)*MVD0, -(1 / 16)*MVD0, 0, (1 / 16)*MVD0, (1 / 8)*MVD0, (1 / 4)*MVD0, (1 / 2)*MVD0, MVD0, 2MVD0, 4MVD0, 8MVD0, 16MVD0 or any subset of the above candidates.

[0254] In another sub-embodiment, the candidate set includes -MVD0+b and -MVD0-b, where b can be any value within the predefined search range, or any subset of the proposed candidates. For example, as shown in Table 6, b can depend on the candidate index in

[0255] Table 6.b depends on the candidate index

[0256]

[0257] In another embodiment, the candidate set (including the symbol information) is predefined. Then, according to the derivation process, a candidate is selected from the candidate set to determine the symbol used in the derivation of MVD1.

[0258] In one sub-embodiment, the candidate set includes a positive sign and a negative sign. If a positive sign is selected, MVD1 is set to k*MVD0; otherwise, MVD1 is set to –k*MVD0. For example, k is predefined as 1. For another example, k is determined by explicit signaling of block-level, SPS-level, PPS-level, APS-level, PH-level, and / or SH-level syntax. For another example, k is determined by an implicit derivation process.

[0259] In another sub-embodiment, the derivation process refers to template matching. The candidate (from the candidate set) with the smallest template matching error is selected.

[0260] In another embodiment, the candidate set is predefined. Then, according to the derivation process, a candidate is selected from the candidate set to determine the MVD0 and MVD1 of the current block.

[0261] In another sub-embodiment, the candidate set includes -k*delta, -(k / 2)*delta, -(k / )4*delta, ..., -delta, ..., -(1 / k)*delta, 0, (1 / k)*delta, ...delta, 2*delta, 4*delta, ..., k*delta, or any subset of the above candidates. k is a positive integer. For example, k=16: the candidate set includes -16*delta, -8*delta, -4*delta, -2*delta, -delta, -(1 / 2)*delta, -(1 / 4)*delta, -(1 / 8)*delta, -(1 / 16)*delta, 0,

[0262] (1 / 16)*delta, (1 / 8)*delta, (1 / 4)*delta, (1 / 2)*delta, delta, 2*delta, 4*delta, 8*delta, 16*delta or any subset of the above candidates. The increment delta can be any value within the predefined search range.

[0263] In another sub-embodiment, the candidate set including a and -a (or any subset of the above candidates) may depend on the candidate index specified in Table 7.

[0264] Table 7.a Dependencies on candidate indexes

[0265]

[0266] In another sub-embodiment, the candidate set varies according to the AMVR of the current block.

[0267] In another sub-embodiment, MVD0 and MVD1 of the current block are set to (MVD0 + the selected candidate) and (-MVD0 - the selected candidate).

[0268] In another sub-embodiment, the derivation process refers to template matching. A candidate (from the candidate set) with the smallest template matching error is selected.

[0269] The template matching error can be the distortion calculated by SATD (Sum of Absolute Transform Difference), SAD (Sum of Absolute Difference), MSE (Mean Square Error), SSE (Sum of Squared Error) or any distortion measurement equation / metric. For example, the template matching in the present invention is as follows.

[0270] Step 1: Use the template (or the neighboring area of ​​the current block, which is encoded or decoded before the current block) to measure the cost of each candidate.

[0271] Step 2: For each candidate, calculate the template cost (template matching error) by the distortion between the “prediction” and reconstruction of the template.

[0272] A “prediction” is generated by applying a shortcut mode with motion (e.g. SMVD) or MVD (using candidates) to the template.

[0273] The method proposed in the present invention can be enabled and / or disabled according to implicit rules (such as block width, height or area) or according to explicit rules (such as syntax on block, tile, slice, picture, SPS or PPS level). For example, when the block area is greater than a threshold, the proposed method is applied. For another example, when the longer block edge is greater than or equal to a threshold (such as 2) multiplied by the shorter block edge, the proposed method is applied.

[0274] The term "block" in the present invention may refer to a TU / TB, a CU / CB, a PU / PB, a predefined area, or a CTU / CTB.

[0275] The AMVP in the present invention is similar to the "AMVP" in JVET-T2002 (Jianle Chen et al., "Algorithmic Description of General Video Coding and Test Model 11 (VTM 11)", Joint Video Experts Group (JVET) T SG16WP 3 of ITU and ISO / IEC JTC 1 / SC 29, 20th Meeting, by Teleconference, October 7-16, 2020, Document: JVET-T2002). AMVP motion comes from motion candidates with syntax "mixed flag" equal to false, for example, general_merge_flag in VVC is equal to false.

[0276] Any combination of the methods proposed in the present invention can be applied. For example, although the embodiment in which the candidate set depends on MVD0 and the embodiment in which the candidate with the minimum template matching error is selected from the candidate set as the MVD1 of the current block are described respectively, the combination of these two embodiments obviously falls into a misunderstanding. Within the scope of the present invention. For example, one embodiment is to predefine a candidate set and select a candidate from the candidate set to derive the MVD of List 0 and / or List 1 of the current block according to the derivation process, while the other embodiment is applied to MMVD. The combination of the two embodiments will be to refine the MMVD offset and / or direction (used to derive the MVD of List 0 and / or List 1 of the current MMVD coding block) by testing more candidate refinement positions and refinement directions, which is not directly indicated by MMVD signaling, but predefined in the candidate set. When defining the candidate set, the MMVD signaling related to the MMVD offset and / or direction can be considered or not considered to derive the candidates in the candidate set.

[0277] Any of the aforementioned proposed adaptive predictor hybrid methods for coding tools using hybrid predictors can be implemented in an encoder and / or a decoder. For example, a hybrid predictor corresponds to a hybrid of two intra predictors or an intra and an inter predictor, which can be implemented in an inter / intra / prediction module of an encoder and / or an inter / intra / prediction module of a decoder. For example, on the encoder side, the required processing can be as follows Figure 1A The inter-frame prediction unit 112 and / or the intra-frame prediction unit 110 shown in the figure are implemented as part of the above. However, the encoder can also use additional processing units to implement the required processing. For the decoder side, the required processing can be implemented as follows Figure 1BThe MC unit 152 and / or part of the intra prediction 150 shown. However, the decoder can also use additional processing units to implement the required processing. Alternatively, any of the proposed methods can be implemented as a circuit coupled to the inter / intra / prediction module of the encoder and / or the inter / intra / prediction module of the decoder to provide the information required by the inter / intra / prediction module. Intra / prediction module. The inter prediction 112 and intra prediction 110 on the encoder side and the MC152 and intra prediction 150 on the decoder side are shown as separate processing units, which can correspond to executable software or firmware code stored on a medium (such as a hard disk or flash memory) for a CPU (central processing unit) or a programmable device (such as a DSP (digital signal processor) signal processor) or an FPGA (field programmable gate array).

[0278] Fig.23A flowchart of an exemplary video encoding system for selecting an MVD from a set of MVD candidates using template matching according to an embodiment of the present invention is shown. The steps shown in the flowchart can be implemented as executable program codes on one or more processors (e.g., one or more CPUs) on the encoder side. The steps shown in the flowchart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to the method, data associated with a current block on the encoder side or encoded data associated with a current block to be decoded on the decoder side is received in step 2310, wherein the current block is encoded and decoded using unidirectional prediction or bidirectional prediction. At least one of the first MVP (motion vector prediction, Motion Vector Predictor) and the second MVP of the current block is determined in step 2320. In step 2330, at least one of the first MVD (MV difference) associated with the first MVP and the second MVD associated with the second MVP from at least one predefined MVD candidate set is determined based on the matching cost. Step 2330 includes two paths 2332 and 2334 for unidirectional prediction and bidirectional prediction, respectively. In step 2332, each matching cost is determined between one or more neighboring samples of the current block and one or more prediction samples, wherein the one or more prediction samples come from one or more corresponding neighboring samples of each reference block pointed to by a unidirectional prediction candidate MV, wherein the unidirectional prediction candidate MV is determined based on one MVD candidate in the at least one predefined MVD candidate set and one of a first MVP and a second MVP. In step 2334, each matching cost is determined between one or more neighboring samples of the current block and one or more prediction samples, wherein the one or more prediction samples come from one or more corresponding neighboring samples of each reference block pointed to by a bidirectional prediction candidate MV, wherein the bidirectional prediction candidate MV is determined based on: a first MVP, a second MVP, and a candidate in the at least one predefined MVD candidate set. In step 2340, the current block is encoded or decoded by using motion information including at least one of a first final MV associated with the first MVP and the first MVD and a second final MV associated with the second MVP and the second MVD.

[0279] The flow chart shown is intended to illustrate an example of video encoding according to the present invention. Without departing from the spirit of the present invention, those skilled in the art may modify each step, rearrange the steps, split the steps, or combine the steps to implement the present invention. In the present disclosure, specific syntax and semantics have been used to illustrate examples to implement embodiments of the present invention. Without departing from the spirit of the present invention, those skilled in the art may implement the present invention by replacing syntax and semantics with equivalent syntax and semantics.

[0280] The above description is provided to enable those of ordinary skill in the art to practice the present invention provided in the context of a specific application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described, but to conform to the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are illustrated to provide a thorough understanding of the present invention. However, those skilled in the art will appreciate that the present invention may be implemented.

[0281] The embodiments of the present invention as described above can be implemented in various hardware, software codes or a combination of the two. For example, one embodiment of the present invention can be one or more circuits integrated into a video compression chip or a program code integrated into the video compression software to perform the processing described herein. The embodiments of the present invention can also be a program code to be executed on a digital signal processor (DSP) to perform the processing described herein. The present invention can also be related to many functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array (FPGA). These processors can be configured to perform specific tasks according to the present invention by executing machine-readable software codes or firmware codes that define the specific methods embodied by the present invention. The software code or firmware code can be developed in different programming languages ​​and different formats or styles. The software code can also be compiled for different target platforms. However, the different code formats, styles and languages ​​of the software code and other ways of configuring the code to perform tasks according to the present invention will not depart from the spirit and scope of the present invention.

[0282] The invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples should be considered in all respects as illustrative and not restrictive. Therefore, the scope of the invention is indicated by the appended claims rather than by the foregoing description. All changes that fall within the meaning and scope of the equivalents of the claims should be included within their scope.

Claims

1. A video encoding and decoding method, the method comprising: Receiving input data associated with a current block, wherein the input data comprises pixel data of the current block to be encoded at an encoder side or encoded data associated with the current block to be decoded at a decoder side, and wherein the current block is encoded and decoded using unidirectional prediction or bidirectional prediction; determining at least one of a first motion vector predictor (MVP) and a second MVP of the current block; Determining at least one of a first MVD associated with the first MVP and a second MVD associated with the second MVP from at least one predefined set of MVD candidates based on a matching cost includes at least one of: In response to encoding and decoding the current block using the unidirectional prediction, determining each matching cost between one or more neighboring samples of the current block and one or more prediction samples, wherein the one or more prediction samples are from one or more prediction samples of one or more corresponding neighboring samples of each reference block pointed to by a unidirectional prediction candidate MV, wherein the unidirectional prediction candidate MV is based on a candidate in the at least one predefined MVD candidate set and one of the first MVP and the second MVP; In response to encoding and decoding the current block using the bidirectional prediction, determining each matching cost between one or more neighboring samples of the current block and one or more prediction samples, wherein the one or more prediction samples are from one or more corresponding neighboring samples of each reference block pointed to by a bidirectional prediction candidate MV, wherein the bidirectional prediction candidate MV is based on at least a first MVP, a second MVP, and a candidate in the at least one predefined MVD candidate set; as well as The current block is encoded or decoded by using motion information including at least one of a first final MV associated with the first MVP and the first MVD and a second final MV associated with a second MVP and the second MVD.

2. The method according to claim 1, characterized in that: In response to encoding and decoding the current block using unidirectional prediction, a unidirectional prediction candidate MV achieving a minimum matching cost is selected to derive at least one of the first final MV and the second final MV.

3. The method according to claim 1, characterized in that In response to encoding and decoding the current block using bi-directional prediction, a bi-directional prediction candidate MV that achieves a minimum matching cost is selected to derive at least one of a first final MV and a second final MV.

4. The method according to claim 1, characterized in that In response to encoding and decoding the current block using the unidirectional prediction, the at least one predefined MVD candidate set corresponds to only one predefined MVD candidate set for deriving the unidirectional prediction candidate MV in list 0 or list 1.

5. The method according to claim 1, characterized in that In response to encoding and decoding the current block using the bidirectional prediction, the at least one predefined MVD candidate set corresponds to only one predefined MVD candidate set used to derive the bidirectional prediction candidate MV.

6. The method according to claim 1, characterized in that In response to encoding and decoding the current block using the bidirectional prediction, the at least one predefined MVD candidate set corresponds to two separate predefined MVD candidate sets used for deriving list 0 MV in bidirectional prediction candidate MVs and list 1 MV in bidirectional prediction candidate MVs, respectively.

7. The method according to claim 1, characterized in that In response to encoding or decoding the current block using the unidirectional prediction or the bidirectional prediction, one or more candidates in the at least one predefined MVD candidate set are derived from an initial MVD.

8. The method according to claim 7, characterized in that Signal or resolve the initial MVD of List0 or List1.

9. The method of claim 7, wherein the at least one predefined MVD candidate set comprises one or more candidates determined based on one or more signs of the initial MVD, one or more values ​​of the initial MVD, or both.

10. The method of claim 9, wherein the one or more signs of the initial MVD correspond to plus and minus signs.

11. The method of claim 9, wherein the one or more values ​​of the initial MVD correspond to k*(initial MVD) or 0, and wherein k corresponds to N or 1 / N, and N is a positive integer.

12. The method of claim 9, wherein the one or more values ​​of the initial MVD correspond to (initial MVD) ± b, and wherein b corresponds to an integer or a fraction.

13. The method of claim 7, wherein the at least one set of predefined MVD candidates includes one or more candidates determined based on one or more symbols of an initial MVD, and wherein the symbol of a target MVD candidate is determined from the at least one predefined set of MVD candidates based on a matching cost. The method according to claim 13 , wherein the value of the target MVD candidate is predefined.

15. The method of claim 13, wherein the value of the target MVD candidate is signaled or parsed.

16. The method of claim 15, wherein one or more syntax elements related to the value of the target MVD candidate are transmitted or parsed at a block level, an SPS level, a PPS level, an APS level, a PH level, a SH level, or a combination thereof.

17. The method of claim 1, wherein the matching cost corresponds to distortion between the one or more neighboring samples of the current block and one or more corresponding neighboring samples of each reference block, and wherein the distortion is measured using one or more metrics including: SATD, SAD, MSE, or SSE.

18. An apparatus for video encoding and decoding, the apparatus comprising one or more electronic devices or processors arranged to: Receive input data associated with the current block, where The input data includes pixel data of the current block to be encoded at the encoder side or encoded data associated with the current block to be decoded at the decoder side, and wherein encoding and decoding the current block uses unidirectional prediction or bidirectional prediction; Determine at least one of a first MVP and a second MVP of the current block; Determining at least one of a first MVD associated with the first MVP and a second MVD associated with the second MVP from at least one predefined MVD candidate set based on the matching cost includes at least one of the following: In response to encoding and decoding the current block using unidirectional prediction, determining each matching cost between one or more neighboring samples of the current block and one or more prediction samples, wherein the one or more prediction samples are from one or more corresponding neighboring samples of each reference block pointed to by a unidirectional prediction candidate MV, wherein the unidirectional prediction candidate MV is based on a candidate in the at least one predefined MVD candidate set and one of a first MVP and a second MVP; In response to encoding the current block using bidirectional prediction, determining each matching cost between one or more neighboring samples of the current block and one or more prediction samples, wherein the one or more prediction samples are from one or more corresponding neighboring samples of each reference block pointed to by a bidirectional prediction candidate MV, wherein the bidirectional prediction candidate MV is based on at least a first MVP, a second MVP, and a candidate in the at least one predefined MVD candidate set; as well as The current block is encoded or decoded by using motion information including at least one of a first final MV associated with the first MVP and the first MVD and a second final MV associated with the second MVP and the second MVD.

Images