Method and apparatus for improving video coding using MVD merge mode with template matching

Abstract

A method and apparatus for video coding and decoding using MMVD mode are disclosed. According to the method, a first extended merge MV of the current block is decided, where the first extended merge MV is derived by adding a first selected offset of a first set of offsets to a base MV, and whether the first extended merge MV is applied to a first reference picture in L0 or a second reference picture in L1 is decided implicitly by the decoder side, where the first extended merge MV is derived by adding a second selected offset of a second set of offsets to the base MV. Or the first extension merge MV is applied to the first reference picture in L0 and the second extension merge MV is applied to the second reference picture in L1. The current block is encoded or decoded by using motion information including the first extension merge MV. According to another method, separate MVDs are used for reference pictures in different reference lists.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure is part of a non-provisional application that claims priority to U.S. Provisional Patent Application No. 63 / 336,389, filed on April 29, 2022. The contents of the above-listed applications are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to a video codec system using a Merge Mode Motion Vector Difference (MMVD) codec tool, and more particularly to adding flexibility to the MMVD design to improve codec performance. Background Art

[0004] Versatile video coding (VVC) is the latest international video coding standard developed by the Joint Video Experts Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). The standard was published as an ISO standard in February 2021: ISO / IEC23090-3:2021, Information technology - Codec representation of immersive media - Part 3: Versatile video coding. VVC is based on its previous generation High Efficiency Video Coding (HEVC) by adding more codec tools to improve codec efficiency and handle various types of video sources including three-dimensional (3D) video signals.

[0005] Figure 1AAn example adaptive inter / intra video codec system incorporating loop processing is shown. For intra prediction, prediction data is derived based on previously encoded video data in the current picture. For inter prediction 112, motion estimation (ME) is performed at the encoder end and motion compensation (MC) is performed based on the results of ME to provide prediction data derived from other pictures and motion data. Switch 114 selects intra prediction 110 or inter prediction 112, and the selected prediction data is provided to adder 116 to form a prediction error, also known as a residual. The prediction error is then processed by a transform (T) 118 followed by a quantization (Q) 120. The transformed and quantized residual is then encoded by an entropy encoder 122 to be included in a video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged with auxiliary information (such as auxiliary information such as motion and coding modes associated with intra prediction and inter prediction) and other information (parameters associated with loop filters applied to underlying image areas, etc.). As Figure 1A As shown, auxiliary information associated with intra prediction 110, inter prediction 112 and loop filter 130 is provided to entropy encoder 122. When inter prediction mode is used, one or more reference pictures must also be reconstructed at the encoder end. Therefore, the transformed and quantized residual is processed by inverse quantization (Inverse Quantization, referred to as IQ) 124 and inverse transformation (Inverse Transformation, referred to as IT) 126 to restore the residual. The residual is then added back to the prediction data 136 in reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in the reference picture buffer 134 and used for prediction of other frames.

[0006] like Figure 1A As shown, the input video data undergoes a series of processes in the encoding system. Due to the series of processes, the reconstructed video data from REC128 may be subject to various damages. Therefore, before the reconstructed video data is stored in the reference picture buffer 134, a loop filter 130 is generally applied to the reconstructed video data to improve the video quality. For example, a deblocking filter (DF for short), a sample adaptive offset (SAO for short), and an adaptive loop filter (ALF for short) may be used. The loop filter information may need to be incorporated into the bitstream so that the decoder can correctly recover the required information. Therefore, the loop filter information is also provided to the entropy encoder 122 to be incorporated into the bitstream. In Figure 1A In , a loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in a reference picture buffer 134 . Figure 1A The system in is intended to illustrate an example structure of a typical video encoder. It may correspond to a High Efficiency Video Coding (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0007] like Figure 1B The decoder shown may use similar or partially identical functional blocks as the encoder, except for transform 118 and quantization 120, since the decoder only needs inverse quantization 124 and inverse transform 126. The decoder uses an entropy decoder 140 instead of an entropy encoder 122 to decode the video bitstream into quantized transform coefficients and required codec information (e.g., ILPF information, intra-frame prediction information, and inter-frame prediction information). The intra-frame prediction 150 at the decoder end does not need to perform a pattern search. Instead, the decoder only needs to generate an intra-frame prediction based on the intra-frame prediction information received from the entropy decoder 140. In addition, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the intra-frame prediction information received from the entropy decoder 140 without motion estimation.

[0008] According to VVC, the input picture is divided into non-overlapping square block areas called coding tree units (CTUs), similar to HEVC. Each CTU can be divided into one or more smaller-sized coding units (CUs). The generated CU partitions can be square or rectangular. In addition, VVC divides CTUs into prediction units (PUs) as a unit to apply prediction processing such as inter-frame prediction, intra-frame prediction, etc.

[0009] The VVC standard incorporates various new codec tools to further improve the codec efficiency beyond the HEVC standard. Among the various new codec tools, some codec tools related to the present invention are summarized as follows. For example, the merge mode with MVD (MMVD) technology reuses the same merge candidates as in VVC, and the selected candidates can be further expanded by the motion vector expression method. Technology to reduce the complexity of MMVD needs to be developed. Summary of the invention

[0010] A method and apparatus for video encoding and decoding using a Merge with Motion Vector Difference (MMVD) mode are disclosed. According to the method, input data associated with a current block encoded and decoded in a bidirectional prediction mode is received, wherein the input data includes pixel data of the current block to be encoded on the encoder side or encoded data associated with the current block to be decoded on the decoder side. A first extended merge motion vector (MV) of the current block is determined, wherein the first extended merge MV is derived by adding a first selected offset in a first set of offsets to a base MV. Whether the first extended merge MV is applied to a first reference picture in L0 (reference list 0) or a second reference picture in L1 (reference list 1) is implicitly determined by the decoder side, or the first extended merge MV is applied to the first reference picture in L0, and the second extended merge MV is applied to the second reference picture in L1. The current block is encoded or decoded by using motion information including the first extended merge MV.

[0011] In one embodiment, whether the first extended merge MV is applied to the first reference picture in L0 or L1 is determined based on a matching cost measured between one or more first neighboring regions of the current block and one or more second neighboring regions of the first reference block in L0 or L1. The one or more first neighboring regions of the current block include a first top neighboring region and a first left neighboring region of the current block, and the one or more second neighboring regions of the first reference block include a second top neighboring region and a second left neighboring region of the first reference block. If the first extended merge MV is applied to the first reference picture in L0(L1), the matching cost is calculated only for the first reference picture in L0(L1), and the first reference picture in L1(L0) is ignored.

[0012] In one embodiment, one or more syntaxes related to the motion vector difference (MVD) between the first extended merged MV and the base MV are sent at the encoder side or parsed at the decoder side. When the first extended merged MV is applied to the first reference picture of one of L0 and L1, the second reference picture in the other of L0 and L1 uses the scaled MVD or the cropped and scaled MVD sent at the encoder side or parsed at the decoder side.

[0013] In one embodiment, the second extended merge MV is derived by adding a second offset selected from the second set of offsets to the base MV. In one embodiment, according to the matching cost associated with the first extended merge MV candidate set and the second extended merge MV candidate set, M first extended merge MV candidates corresponding to a portion of the first extended merge MV candidate set are selected, and N second extended merge MV candidates corresponding to a portion of the second extended merge MV candidate set are selected, and where M and N are positive integers. MxN joint extended merge MV candidates can be generated from M first extended merge MV candidates and N second extended merge MV candidates. Then the MxN joint extended merge MV candidates are reordered according to the matching cost. The first extended merge MV and the second extended merge MV can be selected from the K best joint extended merge MV candidates among the MxN joint extended merge MV candidates according to the matching cost, and K is less than MxN. In one embodiment, M and N correspond to a predetermined number, a number that varies adaptively based on a matching cost distribution, a number that varies adaptively based on a bi-prediction with CU-level Weights (BCW) index with CU-level weights, or an explicitly sent value.

[0014] According to another method, the extended merged motion vector (MV) of the current block is determined by adding the selected offset in the first set of offsets to the base MV, and the selected offset is indicated by MMVD (Merged MV Difference), and MMVD is sent on the encoder side or parsed on the decoder side. The extended merged MV is always applied to the reference frame that is associated with a higher weight of BCW (bidirectional prediction with CU-level weight). BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1A An example adaptive inter / intra video codec system incorporating loop processing is shown.

[0016] Figure 1B Shown with Figure 1A The decoder corresponding to the encoder in .

[0017] Figure 2 An example of Current Picture Referencing (CPR) compensation is shown, where a block is predicted by a corresponding block in the same picture.

[0018] Figure 3An example of a Merge Mode Motion Vector Difference (MMVD) search process is shown, in which a current block in a current frame is processed by bidirectional prediction using an L0 reference frame and an L1 reference frame.

[0019] Figure 4 The offset distances of the L0 reference block and the L1 reference block in the horizontal and vertical directions according to MMVD are shown.

[0020] Figure 5 An example of deriving merge mode candidates from spatial and temporal neighboring blocks is shown.

[0021] Figure 6 An example of a template for a current block and a corresponding reference block to measure a matching cost associated with a merge candidate is shown.

[0022] Figure 7 An example of a template of a block for performing sub-block motion using motion information of a sub-block of a current block and a reference sample of the template is shown.

[0023] Figure 8 A flowchart of an exemplary video coding and decoding system using a flexible MMVD design to improve coding and decoding performance according to an embodiment of the present invention is shown.

[0024] Fig. 9 A flow chart of another exemplary video coding system using separate MVDs for reference pictures in different reference lists according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] It is readily understood that the components of the present invention as generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of embodiments of the systems and methods of the present invention, as shown in the figures, is not intended to limit the scope of the claimed invention, but is merely representative of selected embodiments of the present invention. References in this specification to "an embodiment," "some embodiments," or similar language mean that specific features, structures, or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in an embodiment" or "in some embodiments" appearing in various places throughout this specification do not necessarily all refer to the same embodiment.

[0026] In addition, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. However, those of ordinary skill in the art will recognize that the present invention may be implemented without one or more specific details or using other methods, components, etc. In other cases, well-known structures or operations are not shown or described in detail to avoid obscuring various aspects of the present invention. The illustrated embodiments of the present invention will be best understood by reference to the accompanying drawings, in which the same parts are represented by the same numbers throughout. The following description is by way of example only, and simply describes some selected embodiments of the apparatus and method consistent with the present invention as claimed herein.

[0027] Current Image Reference

[0028] Motion compensation is one of the key techniques in hybrid video codecs, which explores pixel correlations between adjacent images. It is usually assumed that in a video sequence, patterns corresponding to objects or backgrounds in a frame are shifted to form corresponding objects in subsequent frames or to be correlated with other patterns within the current frame. By estimating this displacement (e.g. using block matching techniques), the pattern can be largely reproduced without re-encoding the pattern. Similarly, block matching and replication have also been attempted to allow reference blocks to be selected from the same picture as the current block. When this concept is applied to video acquired by a camera, it is found to be inefficient. This is partly because text patterns in spatially adjacent areas may be similar to the current encoded block, but usually have some gradual changes in space. For a block, it is difficult to find an exact match in the same picture in the video acquired by the camera. Therefore, the improvement in codec performance is limited.

[0029] However, the situation of spatial correlation between pixels within the same picture is different for screen content. For typical videos with text and graphics, there are usually repeated patterns in the same picture. Therefore, it has been observed that intra-frame (picture) block compensation is very effective. A new prediction mode, intra block copy (IBC) mode or current picture referencing (CPR) is introduced in screen content coding to take advantage of this feature. In CPR mode, the prediction unit (PU) is predicted based on the previously reconstructed block in the same picture. In addition, a displacement vector (called block vector or BV) is used to indicate the relative displacement from the position of the current block to the position of the reference block. Then transform, quantization and entropy coding are used to encode and decode the prediction error. Figure 2An example of CPR compensation is shown in FIG. 2 , where block 212 is a corresponding block of block 210, and block 222 is a corresponding block of block 220. In this technique, the reference sample corresponds to the reconstructed sample currently decoded. The image before the loop filter operation includes a deblocking filter and a sample adaptive offset (SAO) filter in HEVC.

[0030] The first version of CPR was proposed in JCTVC-M0350 (Budagavi et al., AHG8: Video coding using Intra motion compensation, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO / IEC JTC 1 / SC 29 / WG11, 13th Meeting: Incheon, KR, 18–26 April 2013, Document: JCTVC-M0350) for HEVC Range Extension (RExt) development. In this version, CPR compensation was limited to a small local region, with only 1-D block vectors, and only applicable to block sizes of 2Nx2N. Later, during the HEVC SCC (Screen Content Codec) standardization process, a more advanced CPR design was developed.

[0031] When CPR is used, only a portion of the current picture can be used as a reference picture. Some bitstream consistency constraints are used to adjust the effective MV values ​​that refer to the current picture. First, one of the following two must be true:

[0032] BV_x+offsetX+nPbSw+xPbs–xCbs<=0 (1)

[0033] BV_y+offsetY+nPbSh+yPbs–yCbs<=0 (2)

[0034] Second, the following WPP conditions must be true:

[0035] (xPbs+BV_x+offsetX+nPbSw-1) / CtbSizeY–xCbs / CtbSizeY<=

[0036] yCbs / CtbSizeY-(yPbs+BV_y+offsetY+nPbSh-1) / CtbSizeY (3)

[0038] In equations (1) to (3), (BV_x, BV_y) is the luma block vector of the current PU (motion vector of the CPR); nPbSw and nPbSh are the width and height of the current PU; (xPbS, yPbs) is the position of the upper left pixel of the current PU relative to the current picture; (xCbs, yCbs) is the position of the upper left pixel of the current CU relative to the current picture; CtbSizeY is the size of the CTU. OffsetX and offsetY are two offsets adjusted in two dimensions to take into account the chroma sample interpolation of the CPR mode.

[0039] offsetX = BVC_x & 0x7? 2:0 (4)

[0040] offsetY=BVC_y&0x7?2:0 (5)

[0041] (BVC_x, BVC_y) is the chroma block vector of 1 / 8 pixel resolution in HEVC.

[0042] Third, the reference blocks of the CPR must be within the same tile / slice boundary.

[0043] Merge with MVD Mode (MMVD) technology

[0044] The MMVD technology is proposed in JVECT-J0024. MMVD is used for skip or merge mode through the proposed motion vector expression method. MMVD reuses the same merge candidates as in VVC. Among the merge candidates, candidates can be selected and further expanded through the proposed motion vector expression method. MMVD provides a new motion vector expression with simplified signaling. The expression method includes prediction direction information, starting point (also referred to as base point in the present invention), motion amplitude (also referred to as distance in the present invention) and motion direction. Figure 3 An example of an MMVD search process is shown, where a current block 312 in a current frame 310 is processed by bidirectional prediction using an L0 reference frame 320 and an L1 reference frame 330. Pixel position 350 is projected to pixel position 352 in the L0 reference frame 320 and pixel position 354 in the L1 reference frame 330. According to the MMVD search process, an updated position will be searched by adding an offset in a selected direction. For example, the updated position corresponds to a position along line 342 or 344 in the horizontal direction, with a distance of s, 2s, or 3s.

[0045] The proposed technique uses the merge candidate list as is. However, the MMVD extension only considers candidates of the default merge type (i.e., MRG_TYPE_DEFAULT_N). The prediction direction information indicates the prediction direction among L0, L1, and L0 and L1 predictions. In the B segment, the proposed method can generate bidirectional prediction candidates from merge candidates with unidirectional prediction by using a mirroring technique. For example, if the merge candidate is a unidirectional prediction with L1, the reference index of L0 is determined by searching the reference picture in list 0, which is mirrored with the reference picture in list 1. If there is no corresponding picture, the reference picture closest to the current picture is used. The L0'MV is obtained by scaling the MV of L1, and the scaling factor is calculated by the POC distance.

[0046] In MMVD, after a merge candidate is selected, it is further expanded or refined by the MVD information sent. Further information includes a merge candidate flag, an index specifying the magnitude of motion, and an index indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is used as the MV basis. The MMVD candidate flag is sent to specify which one to use between the first and second merge candidates. The initial MV (i.e., merge candidate) selected from the merge candidate list is also referred to as the basis in this disclosure. After searching the position set, the selected MV candidate is referred to as an extended MV candidate in this disclosure.

[0047] If the prediction direction of the MMVD candidate is the same as one of the original merge candidates, an index with a value of 0 is sent as the MMVD prediction direction. Otherwise, an index with a value of 1 is sent. After the first bin is sent, the remaining prediction directions are sent according to the predetermined priority of the MMVD prediction directions. The priorities are L0 / L1 prediction, L0 prediction, and L1 prediction. If the prediction direction of the merge candidate is L1, "0" is sent to indicate that the MMVD' prediction direction is L1. "10" is sent to indicate that the MMVD' prediction direction is L0 and L1. "11" is sent to indicate that the MMVD' prediction direction is L0. If the L0 and L1 prediction lists are the same, the prediction direction information of MMVD will not be sent.

[0048] The basic candidate index as shown in Table 1 defines the starting point. The basic candidate index represents the best candidate among the candidates in the list, as shown below.

[0049] Table 1. Basic candidate IDX

[0050]

[0051] like Figure 4 As shown, the distance index specifies the motion magnitude information and indicates the predetermined offsets of the L0 reference block 410 and the L1 reference block 420 from the starting points (412 and 422). Figure 4In , the offset is added to the horizontal component or vertical component of the starting MV, where small circles of different styles correspond to different offsets from the center. The relationship between the distance index and the predetermined offset is shown in Table 2.

[0052] Table 2. Distance IDX

[0053]

[0054] The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions as shown below. The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions as shown in Table 3. It is worth noting that the meaning of the MVD symbol can change according to the information of the starting MV. When the starting MV is a non-predicted MV or a bidirectionally predicted MV, both lists point to the same side of the current picture (that is, the POCs of both references are greater than the POC of the current picture, or are less than the POC of the current picture), and the symbol in Table 3 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectionally predicted MV, the two MVs point to different sides of the current picture (that is, the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), and the POC difference in list 0 is greater than the POC difference in list 1, the symbol in Table 3 specifies the sign of the MV offset added to the list 0 MV component of the starting MV, and the sign of the list 1 MV has the opposite value. Otherwise, if the difference of POC in list 1 is greater than that in list 0, the sign in Table 3 specifies the sign of the MV offset of the MV component of list 1 added to the starting MV, and the sign of the MV of list 0 has the opposite value.

[0055] Table 3. Direction IDX

[0056] Distance from IDX 00 01 10 11 x-axis + – N / A N / A y-axis N / A N / A + –

[0057] To reduce encoder complexity, block restrictions are applied. If the width or height of the CU is less than 4, MMVD is not performed.

[0058] Multi-Hypothesis (MH) Technology

[0059] Multi-hypothesis prediction is proposed to improve existing prediction modes in inter-pictures, including advanced motion vector prediction (AMVP) mode, skip and merge modes, and unidirectional prediction for intra mode. The general concept is to combine existing prediction modes with additional merge index prediction. Merge index prediction is performed in the same way as regular merge mode, where the merge index is sent to obtain motion information for motion compensated prediction. The final prediction is a weighted average of the merge index prediction and the prediction generated by the existing prediction mode, where different weights are applied depending on the combination. For detailed information, please refer to JVET-K1030 (Chih-Wei Hsu, et al., Description of CoreExperiment 10:Combined and multi-hypothesis prediction,Joint CollaborativeTeam on Video Coding(JCT-VC) of ITU-T SG16 WP3 and ISO / IEC JTC 1 / SC 29 / WG11,11th Meeting:Ljubljana,SI,10–18July 2018,Document:JVET-K1030),or JVET-L0100(Man-Shu Chiang,et al.,CE10.1.1:Multi-hypothesis prediction for improving AMVP mode,skip or merge mode,and intra mode,Joint Collaborative Team on VideoCoding(JCT-VC)of ITU-T SG16 WP3 and ISO / IEC JTC 1 / SC 29 / WG11,12th Meeting: Macao, CN, 3–12 Oct. 2018, Document: JVET-L0100).

[0060] Pairwise average merge candidates

[0061] The pairwise average candidate is generated by averaging the predetermined candidate pairs in the current merge candidate list, and the predetermined pairs are defined as {(0,1),(0,2),(1,2),(0,3),(1,3),(2,3)}, where the number represents the merge index of the merge candidate list. The average motion vector is calculated separately for each reference list. If two motion vectors are available in one list, they are averaged even if they point to different reference pictures; if only one motion vector is available, it is used directly; if no motion vector is available, this list is considered invalid.

[0062] Merge Mode

[0063] In order to improve the coding efficiency of motion vector (MV) coding in HEVC, HEVC has skip and merge modes. The skip and merge modes obtain motion information from spatially adjacent blocks (spatial candidates) or temporally co-located blocks (temporal candidates). When the PU is in skip or merge mode, the motion information is not encoded and decoded, but only the index of the selected candidate is encoded and decoded. For skip mode, the residual signal is forced to zero and is not encoded and decoded. In HEVC, if a particular block is encoded as skip or merge, a candidate index is sent to indicate which candidate in the candidate set is used for merging. Each merged PU reuses the MV, prediction direction, and reference picture index of the selected candidate.

[0064] For the merge mode in HM-4.0 in HEVC, such as Figure 5 As shown, up to four spatial MV candidates are derived from A0, A1, B0 and B1, and one temporal MV candidate is derived from TBR or TCTR (TBR is used first, if TBR is not available, TCTR is used). Note that if any of the four spatial MV candidates is not available, position B2 is used to derive another MV candidate as a substitute. After the derivation process of four spatial MV candidates and one temporal MV candidate, redundancy removal (pruning) is used to remove redundant MV candidates. If the number of available MV candidates is less than 5 after redundancy removal (pruning), three types of additional candidates are derived and added to the candidate set (candidate list). The encoder makes a decision based on rate-distortion optimization (RDO), and a final candidate is selected from the candidate set in skip or merge mode, and the index is transmitted to the decoder.

[0065] Hereinafter, we refer to the skip and merge modes as “merge mode.” In other words, when “merge mode” is mentioned in the following description, “merge mode” may refer to both the skip mode and the merge mode.

[0066] Adaptive Reordering of MergeCandidates with template Matching (ARMC-TM)

[0067] The merge candidates are adaptively reordered based on the cost evaluated using template matching (TM). The reordering method can be applied to regular merge mode, template matching (TM) merge mode, and affine merge mode (excluding SbTMVP candidates). For TM merge mode, merge candidates are reordered before refinement.

[0068] After the merge candidate list is constructed, the merge candidates are divided into multiple subgroups. For the normal merge mode and the TM merge mode, the subgroup size is set to 5. For the affine merge mode, the subgroup size is set to 3. The merge candidates in each subgroup are reordered in ascending order according to the cost value based on template matching. For ARMC-TM, the candidates in the subgroup are skipped if the subgroup meets the following two conditions: (1) the subgroup is the last subgroup and (2) the subgroup is not the first subgroup. In short, the merge candidates in the last subgroup but not in the first subgroup will not be reordered.

[0069] The template matching cost of the merged candidate is measured as the sum of absolute differences (SAD) between the samples of the template of the current block and its corresponding reference samples. The template includes the set of reconstructed samples adjacent to the current block. The reference samples of the template are located by the motion information of the merged candidate.

[0070] When the merge candidate uses bidirectional prediction, the reference sample of the merge candidate template is also obtained by Figure 6 The bidirectional prediction shown is generated. Figure 6 , block 612 corresponds to a current block in current picture 610, and blocks 622 and 632 correspond to reference blocks in reference pictures 620 and 630 in list 0 and list 1, respectively. Templates 614 and 616 are used for current block 612, templates 624 and 626 are used for reference block 622, and templates 634 and 636 are used for reference block 632. Motion vectors 640, 642, and 644 are merge candidates in list 0, and motion vectors 650, 652, and 654 are merge candidates in list 1.

[0071] For a sub-block-based merge candidate with a sub-block size equal to Wsub×Hsub, the upper template includes multiple sub-templates of size Wsub×1, and the left template includes multiple sub-templates of size 1×Hsub. Figure 7As shown in FIG. 1 , the motion information of the sub-blocks in the first row and the first column of the current block is used to derive the reference samples of each sub-template. Figure 7 , block 712 corresponds to the current block in the current picture 710, and block 722 corresponds to the co-located block in the reference picture 720. Each small square in the current block and the co-located block corresponds to a sub-block. The dot-filled area on the left and top of the current block corresponds to the template of the current block. The boundary sub-blocks are labeled from A to G. The arrows associated with each sub-block correspond to the motion vector of the sub-block. The reference sub-blocks (labeled Aref to Gref) are located according to the motion vectors associated with the boundary sub-blocks.

[0072] Improving MMVD with Template Matching

[0073] In some designs of MMVD using template matching (TM), for each base MV, it selects K MVD candidates from a total of S*D combinations of S steps and D directions. The selection is based on the TM cost. If bidirectional prediction is used, the transmitted MVD is implicitly applied to the reference frame with a larger temporal distance. For the other reference frame, the applied MVD is the transmitted MVD, but is reduced according to the temporal distance difference. In such a design, the MVD selection for the bidirectional case lacks freedom. The method proposed below will improve MMVD in this regard.

[0074] In one approach, if bi-prediction is used in MMVD, the transmitted MVD is implicitly applied to the higher weighted reference frame of bi-prediction with CU-level weight (BCW).

[0075] In another method, if bidirectional prediction is used in MMVD, the reference frame to which the transmitted MVD is applied is determined by the TM cost. Specifically, two TM costs are derived by applying the transmitted MVD to one reference frame each time, and the final transmitted MVD is applied to the reference frame with the lower TM cost. In one embodiment, when two TM costs are derived by applying the signaled MVD to one reference frame each time, the other frame applies the scaled transmitted MVD. In another embodiment, the other frame applies the clipped scaled transmitted MVD. In another embodiment, the other frame is not considered in the TM cost calculation.

[0076] In another method, if bidirectional prediction is used in MMVD, the two reference frames can have independent MVDs. Specifically, each reference frame is reordered based on TM, and M candidates of one reference frame and N candidates of another reference frame are selected to form M*N bidirectional prediction candidates. Another TM-based reordering is performed on these M*N candidates to select K candidates for further transmission. Note that this method can be implemented without changing the codeword. The values ​​of M and N can be predetermined fixed numbers, numbers that are adaptively changed based on TM cost distribution, numbers that are adaptively changed based on BCW index, or values ​​that are explicitly sent.

[0077] In another method, if the basis is bidirectional prediction, multiple unidirectional prediction candidates are generated by using only one of the reference frames. The TM costs of all bidirectional prediction and unidirectional prediction candidates are calculated and compared, wherein the bidirectional prediction candidates can be generated by using the original MMVD design (i.e., S*D candidates) or the aforementioned proposed design (i.e., M*N candidates), and the unidirectional prediction candidates can be generated by considering all S*D candidates or only a subset of S*D candidates. According to one embodiment of the present invention, after the TM cost comparison, K candidates are selected from all candidates for further transmission. Note that this method can be implemented without changing the codeword.

[0078] In another method, if the base MV is unidirectional prediction, multiple bidirectional prediction candidates are generated by using the same reference frame with two different MVDs. The TM costs of all unidirectional prediction and bidirectional prediction candidates are calculated and compared, where the unidirectional prediction candidates can be generated by considering all possible S*D candidates or only a subset of S*D candidates, and the bidirectional prediction candidates can be generated by combining any two different unidirectional prediction candidates in the S*D candidates, combining any two different unidirectional prediction candidates in a subset of S*D candidates, or combining two unidirectional prediction candidates, one of which is from a subset of S*D candidates and the other is from another subset of S*D candidates. After TM cost comparison, K candidates are selected from all candidates for further transmission. Note that this method can be implemented without changing the codeword. In addition, this method can be combined with the previously proposed method of generating unidirectional prediction candidates for a bidirectional prediction basis.

[0079] Any of the above MMVD methods can be implemented in an encoder and / or decoder. For example, any of the proposed methods can be implemented in an inter-frame coding module (e.g., Figure 1A inter-frame prediction 112 in the decoder), the motion compensation module of the decoder (e.g., Figure 1BMC 152 in the example and the merge candidate derivation module. Alternatively, any of the proposed methods may be implemented as a circuit coupled to the inter-frame coding module of the encoder and / or the motion compensation module of the decoder and the merge candidate derivation module. MC 112 and MC 152 are shown as independent processing units supporting the MMVD method, which may correspond to executable software or firmware codes stored on a medium, such as a hard disk or flash memory, for a central processing unit (CPU) or a programmable device (such as a digital signal processor (DSP) or a field programmable gate array (FPGA).

[0080] Figure 8 A flowchart of an exemplary video codec system for improving codec performance using a flexible MMVD design according to an embodiment of the present invention is shown. The steps shown in the flowchart can be implemented as executable program codes on one or more processors (e.g., one or more CPUs) on the encoder side. The steps shown in the flowchart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to the method, in step 810, input data associated with a current block encoded and decoded in a bidirectional prediction mode is received, wherein the input data includes pixel data of the current block to be encoded on the encoder side or prediction residual data associated with the current block to be decoded on the decoder side. In step 820, a first extended merged motion vector (MV for short) of the current block is determined, wherein the first extended merged MV is derived by adding a first selected offset in a first set of offsets to a base MV, and wherein whether the first extended merged MV is applied to a first reference picture in L0 (reference list 0) or a second reference picture in L1 (reference list 1) is implicitly determined by the decoder side, or the first extended merged MV is applied to the first reference picture in L0, and the second extended merged MV is applied to the second reference picture in L1. In step 830, the current block is encoded or decoded by using motion information including the first extended merged MV.

[0081] Fig. 9A flowchart of another exemplary video coding system using separate MVDs for reference pictures in different reference lists according to an embodiment of the present invention is shown. According to the method, in step 910, input data associated with a current block encoded and decoded in a bidirectional prediction mode is received, wherein the input data includes pixel data of the current block to be encoded on the encoder side or prediction residual data associated with the current block to be decoded on the decoder side. In step 920, an extended merged motion vector (MV) of the current block is determined, wherein the extended merged MV is derived by adding an offset selected from a first set of offsets to a base MV, and the selected offset is indicated by a merged MV difference (MMVD), and wherein the MMVD is sent on the encoder side or parsed on the decoder side. In step 930, the extended merged MV is applied to a reference frame associated with a higher weight of a bidirectional prediction (BCW) with a CU-level weight. In step 940, the current block is encoded or decoded by using motion information including the first extended merged MV.

[0082] The flowchart shown is intended to illustrate an example of video encoding and decoding according to the present invention. Without departing from the spirit of the present invention, a person skilled in the art may modify each step, rearrange the steps, split the steps, or combine the steps to implement the present invention. In the present disclosure, specific syntax and semantics are used to illustrate examples to implement embodiments of the present invention. A person skilled in the art may implement the present invention by replacing the above syntax and semantics with equivalent syntax and semantics without departing from the spirit of the present invention.

[0083] The above description is presented to enable one of ordinary skill in the art to implement the present invention provided in the context of a specific application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described, but to conform to the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are described in order to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be implemented.

[0084] The embodiments of the present invention as described above can be implemented in various hardware, software codes or a combination of the two. For example, one embodiment of the present invention can be a program code integrated into one or more circuits in a video compression chip or integrated into video compression software to perform the processing described herein. An embodiment of the present invention can also be a program code to be executed on a digital signal processor (Digital Signal Processor, referred to as DSP) to perform the processing described herein. The present invention can also be related to many functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array (field programmable gate array, referred to as FPGA). These processors can be configured to perform specific tasks according to the present invention by executing machine-readable software code or firmware code that defines the specific method embodied in the present invention. The software code or firmware code can be developed in different programming languages ​​and different formats or styles. The software code can also be compiled for different target platforms. However, the different code formats, styles and languages ​​of the software code and other ways of configuring the code to perform tasks according to the present invention will not depart from the spirit and scope of the present invention.

[0085] The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered in all respects as illustrative only and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the foregoing description. All changes within the equivalent meaning and scope of the claims should be included within their scope.

Claims

1. A video encoding and decoding method using a merge mode with motion vector difference, the method comprising: Receiving input data associated with a current block encoded and decoded in a bidirectional prediction mode, wherein the input data includes pixel data of the current block to be encoded at an encoder side or encoded data associated with the current block to be decoded at a decoder side; determining a first extended merged motion vector for the current block, wherein the first extended merged motion vector is derived by adding a first selected offset from a first set of offsets to a base motion vector, and wherein whether the first extended merged motion vector is applied to a first reference picture in a reference list L0 or a second reference picture in a reference list L1 is implicitly determined by the decoder side, or the first extended merged motion vector is applied to the first reference picture in the reference list L0, and the second extended merged motion vector is applied to the second reference picture in the reference list L1; and The current block is encoded or decoded by using the motion information including the first extended merged motion vector.

2. The video encoding and decoding method using a merge mode with motion vector difference according to claim 1, characterized in that: Whether the first extended merged motion vector is applied to the first reference picture in the reference list L0 or the reference list L1 is determined based on the matching cost measured between one or more first neighboring regions of the current block and one or more second neighboring regions of the first reference block in the reference list L0 or the reference list L1.

3. The video encoding and decoding method using a merge mode with motion vector difference according to claim 2, characterized in that: The one or more first neighboring regions of the current block include a first top neighboring region and a first left neighboring region of the current block, and the one or more second neighboring regions of the first reference block include a second top neighboring region and a second left neighboring region of the first reference block.

4. The video encoding and decoding method using a merge mode with motion vector difference according to claim 2, characterized in that: If the first extended merged motion vector is applied to the first reference picture in the reference list L0 (L1), the matching cost is calculated only for the first reference picture in the reference list L0 (L1), and the matching cost is ignored for the first reference picture in the reference list L1 (L0).

5. The video encoding and decoding method using a merge mode with motion vector difference according to claim 1, characterized in that: One or more syntaxes related to the motion vector difference between the first extended merged motion vector and the base motion vector are sent at the encoder side or parsed at the decoder side.

6. The video encoding and decoding method using a merge mode with motion vector difference according to claim 5, characterized in that: When the first extended merged motion vector is applied to the first reference picture in one of the reference lists L0 and L1, the second reference picture in the other of the reference lists L0 and L1 uses the scaled motion vector difference sent at the encoder side or parsed at the decoder side.

7. The video encoding and decoding method using a merge mode with motion vector difference according to claim 5, characterized in that: When the first extended merged motion vector is applied to the first reference picture of a reference list in one of the reference lists L0 and L1, the first reference picture in the other reference list of the reference lists L0 and L1 uses the clipped and scaled motion vector difference sent at the encoder side or parsed at the decoder side.

8. The video encoding and decoding method using a merge mode with motion vector difference according to claim 1, characterized in that: The second extended merged motion vector is derived by adding a second selected offset from a second set of offsets to the base motion vector.

9. The video encoding and decoding method using a merge mode with motion vector difference according to claim 8, characterized in that: Based on multiple matching costs associated with a first extended merge motion vector candidate set and a second extended merge motion vector candidate set, M first extended merge motion vector candidates corresponding to a portion of the first extended merge motion vector candidate set are selected, and N second extended merge motion vector candidates corresponding to a portion of the second extended merge motion vector candidate set are selected.

10. The video encoding and decoding method using a merge mode with motion vector difference according to claim 9, characterized in that: MxN joint extended merged motion vector candidates are generated from the M first extended merged motion vector candidates and the N second extended merged motion vector candidates, and wherein the MxN joint extended merged motion vector candidates are reordered according to the plurality of matching costs.

11. The video encoding and decoding method using a merge mode with motion vector difference according to claim 10, characterized in that: The first extended merged motion vector and the second extended merged motion vector are selected from K best joint extended merged motion vector candidates among MxN joint extended merged motion vector candidates according to the multiple matching costs, and K is less than MxN.

12. The video encoding and decoding method using a merge mode with motion vector difference according to claim 11, characterized in that: M and N correspond to predetermined numbers, numbers that vary adaptively based on a matching cost distribution, numbers that vary adaptively based on a bidirectional prediction index having a codec-unit level weight, or explicitly transmitted values.

13. A video codec apparatus using merge mode with motion vector difference, the apparatus comprising one or more electronic devices or processors arranged to: receiving input data associated with a current block encoded and decoded in a bidirectional prediction mode, wherein: The input data includes pixel data of a current block to be encoded at the encoder side or encoded data associated with the current block to be decoded at the decoder side; determining a first extended merged motion vector for the current block, wherein the first extended merged motion vector is derived by adding a first selected offset from a first set of offsets to a base motion vector, and wherein whether the first extended merged motion vector is applied to a first reference picture in a reference list L0 or a second reference picture in a reference list L1 is implicitly determined by the decoder side, or the first extended merged motion vector is applied to the first reference picture in the reference list L0, and the second extended merged motion vector is applied to the second reference picture in the reference list L1; as well as The current block is encoded or decoded by using the motion information including the first extended merged motion vector.

14. A coding method using a merge mode with motion vector difference, the method comprising: Receiving input data associated with a current block encoded and decoded in a bidirectional prediction mode, wherein the input data includes pixel data of the current block to be encoded at an encoder side or encoded data associated with the current block to be decoded at a decoder side; determining an extended merged motion vector for the current block, wherein the extended merged motion vector is derived by adding an offset selected from a first set of offsets to a base motion vector, and the selected offset is indicated by a merged motion vector difference (MMVD), and wherein the merged motion vector difference is sent at the encoder side or parsed at the decoder side; and applying the extended merge MV to reference frames associated with a higher weight of bidirectional prediction with codec unit level weight; and The current block is encoded or decoded by using the motion information including the extended merged motion vector.

15. A coding and decoding apparatus using merge mode with motion vector difference, the apparatus comprising one or more electronic devices or processors arranged to: receiving input data associated with a current block encoded and decoded in a bidirectional prediction mode, wherein: The input data includes pixel data of a current block to be encoded at the encoder side or encoded data associated with the current block to be decoded at the decoder side; determining an extended merged motion vector for the current block, wherein the extended merged motion vector is derived by adding an offset selected from a first set of offsets to a base motion vector, and the selected offset is indicated by a merged motion vector difference (MMVD), and wherein the merged motion vector difference is sent at the encoder side or parsed at the decoder side; as well as Applying the extended merge MV to reference frames associated with a higher weight of bidirectional prediction with codec unit level weights; as well as The current block is encoded or decoded by using the motion information including the extended merged motion vector.

Images