Method and apparatus for reducing video coding complexity using mvd merging mode

Abstract

Disclosed are a method and apparatus for video coding and decoding using an MMVD mode. According to the method, a base merge MV is determined from a merge list. A set of extended merge candidates is determined from a set of steps and a set of directions by adding step and direction pairs to the base MV. At least one combination of the set of steps and the set of directions is excluded from the partial set of extended merge candidates. The partial set of extended merge candidates is reordered according to a template matching cost measured between a template of the current block and a template of a corresponding reference block in the partial set of extended merge candidates. And encoding or decoding the current block by using motion information including the reordered partial set of the extended merge candidates.

Description

[0001] Cross-references

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 336,388 filed on April 29, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to a video coding system using a Merge Mode Motion Vector Difference (MMVD) coding tool. In particular, the present invention relates to reducing the complexity associated with MMVD. Background Art

[0004] Versatile Video Coding (VVC) is the latest international video coding standard jointly developed by the Joint Video Experts Group (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). The standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Information technology - Coded representation of immersive media - Part 3: Versatile video coding, published in February 2021. VVC is based on its predecessor HEVC (High Efficiency Video Coding), by adding more codec tools to improve codec efficiency, and can also handle various types of video sources, including 3D video signals.

[0005] Figure 1A An exemplary adaptive inter / intra video coding system including a loop process is described. For intra prediction, prediction data is derived from previously encoded video data in the current picture. For inter prediction 112, motion estimation (ME) is performed on the encoder side and motion compensation (MC) is performed based on the results of ME to provide prediction data derived from other pictures and motion data. Switch 114 selects intra prediction 110 or inter prediction 112 and the selected prediction data is provided to adder 116 to form a prediction error, also known as a residual. The prediction error is then processed by transform (T) 118 and subsequent quantization (Q) 120. The transformed and quantized residual is then encoded by entropy encoder 122 to be included in a video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged with auxiliary information (such as motion and coding modes associated with intra prediction and inter prediction) and other information (such as parameters associated with loop filters applied to underlying image areas). Auxiliary information associated with intra prediction 110, inter prediction 112, and in-loop filter 130 is provided to entropy encoder 122, such as Figure 1AAs shown. When using inter-frame prediction mode, one or more reference pictures must also be reconstructed at the encoder end. Therefore, the transformed and quantized residual is processed by inverse quantization (IQ) 124 and inverse transform (IT) 126 to recover the residual. The residual is then added back to the prediction data 136 at reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in a reference picture buffer 134 and used to predict other frames.

[0006] like Figure 1A As shown, the input video data undergoes a series of processes in the encoding system. Due to the series of processes, the reconstructed video data from REC128 may be subject to various impairments. Therefore, the loop filter 130 is often applied to the reconstructed video data before it is stored in the reference picture buffer 134 to improve the video quality. For example, a deblocking filter (DF), sample adaptive offset (SAO), and adaptive loop filter (ALF) may be used. It may be necessary to merge the loop filter information into the bitstream so that the decoder can correctly recover the required information. Therefore, the loop filter information is also provided to the entropy encoder 122 to be merged into the bitstream. Figure 1A In , a loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in a reference picture buffer 134 . Figure 1A The system in is intended to illustrate an exemplary structure of a typical video encoder. It may correspond to the High Efficiency Video Coding (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0007] like Figure 1B As shown, the decoder can use similar or identical functional blocks as the encoder, except for transform 118 and quantization 120, because the decoder only needs inverse quantization 124 and inverse transform 126. Instead of entropy encoder 122, the decoder uses entropy decoder 140 to decode the video bitstream into quantized transform coefficients and required encoding information (such as ILPF information, intra-frame prediction information, and inter-frame prediction information). Intra-frame prediction 150 on the decoder side does not need to perform mode search. Instead, the decoder only needs to generate intra-frame prediction based on the intra-frame prediction information received from entropy decoder 140. In addition, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the inter-frame prediction information received from entropy decoder 140 without motion estimation.

[0008] According to VVC, similar to HEVC, the input picture is divided into non-overlapping square block areas called CTUs (Coding Tree Units). Each CTU can be divided into one or more coding units (CUs) of smaller size. The generated CU partitions can be square or rectangular. In addition, VVC divides CTUs into prediction units (PUs) as units for applying prediction processes such as inter-frame prediction, intra-frame prediction, etc.

[0009] The VVC standard combines various new codec tools to further improve the coding efficiency of the HEVC standard. Among the various new codec tools, some codec tools related to the present invention are reviewed as follows. For example, the merged MVD mode (MMVD) technology reuses the same merge candidates as in VVC, and the selected candidates can be further expanded by the motion vector expression method. It is necessary to develop technologies that reduce the complexity of MMVD. Summary of the invention

[0010] A method and apparatus for video encoding and decoding using an MMVD mode are disclosed. According to the method, input data associated with a current block is received, wherein the input data includes pixel data of the current block to be encoded at the encoder side or encoded data associated with the current block to be decoded at the decoder side. At least one basic merge motion vector (MV) is determined from a merge list of the current block. A set of extended merge candidates is determined for the at least one basic merge MV according to a step size set and a direction set, wherein the set of extended merge candidates is determined by adding an offset to the at least one basic merge MV, and wherein the offset corresponds to a pair of combinations from the step size set and the direction set, wherein at least one combination of the step size set and the direction set is excluded from a partial set of extended merge candidates. Candidates in the partial set of extended merge candidates are reordered according to template matching costs associated with the candidates, and wherein each template matching cost is measured between a first sample in one or more first adjacent regions of the current block and a second sample in one or more second adjacent regions of a reference block, wherein the reference block is located according to each candidate in the partial set of extended merge candidates. The current block is encoded or decoded by using motion information of the reordered partial set of extended merge candidates.

[0011] In one embodiment, a partial set of extended merge candidates is generated by limiting member step sizes in the step size set to be smaller than a step size threshold, so that at least one member step size candidate in the step size set is excluded from the partial set of extended merge candidates.

[0012] In one embodiment, the partial set of extended merge candidates is generated by restricting the extended merge candidate set within a bounding box to exclude at least one extended merge candidate in the extended merge candidate set.

[0013] In one embodiment, a first syntax is signaled or parsed in a coding unit (CU) level to indicate whether a partial set of extended merge candidates is used for a current block. In one embodiment, when the first syntax indicates that the partial set of extended merge candidates is used for the current block, a second syntax is signaled or parsed in a CU level to indicate a target candidate selected from the partial set of extended merge candidates. In another embodiment, when the partial set of extended merge candidates is not used for the current block, a second syntax is signaled or parsed in a CU level to indicate a target candidate selected from a remaining candidate set of the current block, and wherein the remaining candidate set corresponds to an extended merge candidate that belongs to the extended merge candidate set but is not in the partial set of extended merge candidates.

[0014] In one embodiment, the one or more first neighboring regions of the current block include a first top neighboring region and a first left neighboring region of the current block, and the one or more second neighboring regions of the reference block include a second top neighboring region, and a second left neighboring region of the reference block.

[0015] In one embodiment, a remaining candidate set of the current block is generated for the current block, and the remaining candidate set corresponds to an extended merge candidate that belongs to the extended merge candidate set but is not in the partial set of extended merge candidates. In addition, the motion information also includes the remaining candidate set that is not reordered using the template matching cost associated with the remaining candidate set. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A An exemplary adaptive inter / intra video coding system incorporating loop processing is shown.

[0017] Figure 1B Shows Figure 1A The corresponding decoder of the encoder in .

[0018] Figure 2 An example of Current Picture Referencing (CPR) compensation is shown, where a block is predicted by a corresponding block in the same picture.

[0019] Figure 3 An example of an MMVD search process is shown, in which a current block in a current frame is processed by bidirectional prediction using an L0 reference frame and an L1 reference frame.

[0020] Figure 4 The offset distances of the L0 reference block and the L1 reference block in the horizontal and vertical directions according to MMVD are shown.

[0021] Figure 5 An example of deriving merge mode candidates from spatial and temporal neighboring blocks is shown.

[0022] Figure 6 Examples of templates used for a current block and corresponding reference blocks to measure matching costs associated with merge candidates are shown.

[0023] Figure 7 An example of a template and a reference sample of a block having sub-block motion using motion information of a sub-block of a current block is shown.

[0024] Figure 8 A flow chart of an exemplary video encoding system utilizing reduced complexity MMVD according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] It will be readily understood that the components of the present invention as generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of the embodiments of the systems and methods of the present invention as shown in the figures is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. References throughout this specification to "one embodiment" or similar language mean that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment" or "in an embodiment" that appear throughout this specification do not necessarily all refer to the same embodiment.

[0026] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. However, those skilled in the relevant art will recognize that the present invention may be practiced without one or more specific details, or using other methods, components, etc. In other cases, well-known structures or detailed descriptions of operations are not shown or are not shown to avoid obscuring aspects of the present invention. The illustrated embodiments of the present invention will be best understood with reference to the accompanying drawings, in which like parts are represented by like numbers throughout. The following description is intended to be exemplary only and simply illustrates certain selected embodiments of the apparatus and methods consistent with the present invention as claimed herein.

[0027] Current Picture Referencing

[0028] Motion compensation is one of the key techniques in hybrid video coding, which explores pixel correlations between adjacent images. It is usually assumed that in a video sequence, patterns corresponding to objects or backgrounds in a frame are shifted to form corresponding objects in subsequent frames or correlated with other patterns within the current frame. By estimating this displacement (for example, using block matching techniques), the pattern can be largely reproduced without re-encoding the pattern. Similarly, block matching and replication have also been attempted to allow reference blocks to be selected from the same picture as the current block. When this concept was applied to video captured by a camera, it was found to be inefficient. This is partly because text patterns in spatially adjacent areas may be similar to the current coded block, but usually have some gradual changes in space. For a block, it is difficult to find an exact match in the same picture in the video captured by the camera. Therefore, the improvement in coding performance is limited.

[0029] However, the situation of spatial correlation between pixels within the same picture is different for screen content. For typical videos with text and graphics, there are usually repeated patterns in the same picture. Therefore, it has been observed that intra-frame (picture) block compensation is very effective. Screen content coding introduces a new prediction mode, namely intrablock copy (IBC) mode or current picture referencing (CPR) to exploit this property. In CPR mode, the prediction unit (PU) is predicted based on the previously reconstructed block within the same picture. In addition, a displacement vector (called block vector or BV) is used to indicate the relative displacement from the position of the current block to the position of the reference block. The prediction error is then encoded and decoded using transform, quantization and entropy coding. Figure 2 An example of CPR compensation is shown in FIG. 2 , where block 212 is a corresponding block of block 210, and block 222 is a corresponding block of block 220. In this technique, the reference sample corresponds to the reconstructed sample currently decoded. The image before the loop filter operation includes a deblocking filter and a sample adaptive offset (SAO) filter in HEVC.

[0030] The first version of CPR was proposed in JCTVC-M0350 (Budagavi et al., AHG8: Video Coding with Intra Motion Compensation, Joint Collaboration Group on Video Coding (JCT-VC) / SC29 / WG11 of ITU-T SG16 WP3 and ISO / IEC JTC 1, 13th Meeting: Incheon, South Korea, April 18-26, 2013, Document: JCTVC-M0350) HEVC Range Extensions (RExt) development. In this version, CPR compensation was limited to small local regions, with only 1-D block vectors, and only applicable to block sizes of 2Nx2N. Later, a more advanced CPR design was developed during the HEVC Screen Content Coding (SCC) standardization process.

[0031] When using CPR, only a portion of the current picture can be used as a reference picture. Some bitstream consistency constraints are imposed to regulate the effective MV values ​​that refer to the current picture. First, one of the following two must be true:

[0032] BV_x+offsetX+nPbSw+xPbs–xCbs<=0(1)

[0033] BV_y+offsetY+nPbSh+yPbs–yCbs<=0(2)

[0034] Second, the following WPP conditions must be true:

[0035] (xPbs+BV_x+offsetX+nPbSw-1) / CtbSizeY–xCbs / CtbSizeY<=yCbs / CtbSizeY-(yPbs+BV_y+offsetY+nPbSh-1) / CtbSizeY (3)

[0037] In equations (1) to (3), (BV_x, BV_y) is the luma block vector of the current PU (motion vector of the CPR); nPbSw and nPbSh are the width and height of the current PU; (xPbS, yPbs) is the position of the upper left pixel of the current PU relative to the current picture; (xCbs, yCbs) is the position of the upper left pixel of the current CU relative to the current picture; CtbSizeY is the size of the CTU. OffsetX and offsetY are two offsets adjusted in two dimensions to take into account the chroma sample interpolation of the CPR mode.

[0038] offsetX=BVC_x&0x7?2:0(4)

[0039] offsetY=BVC_y&0x7?2:0(5)

[0040] (BVC_x, BVC_y) is the chroma block vector with 1 / 8 pixel resolution in HEVC.

[0041] Third, the reference blocks of CPR must be within the same tile / slice boundary.

[0042] Merged with MVD Mode (MMVD) technology

[0043] The MMVD technology is proposed in JVECT-J0024. MMVD is used for skip or merge mode through the proposed motion vector expression method. MMVD reuses the same merge candidate members as in VVC. Among the merge candidate members, one candidate member can be selected and further expanded through the proposed motion vector expression method. MMVD provides a new motion vector expression with simplified signaling. The expression method includes prediction direction information, starting point (also called base point in the present invention), motion amplitude (also called distance in the present invention) and motion direction. Figure 3 An example process of MMVD search is illustrated, where a current block 312 in a current frame 310 is processed by bidirectional prediction using an L0 reference frame 320 and an L1 reference frame 330. Pixel position 350 is projected to pixel position 352 in the L0 reference frame 320 and pixel position 354 in the L1 reference frame 330. According to the MMVD search process, an updated position is searched by adding an offset in a selected direction. For example, the updated position corresponds to a position along line 342 or 344 in the horizontal direction, with a distance of s, 2s, or 3s.

[0044] The proposed technique uses the merge candidate list as is. However, the MMVD extension only considers candidates of the default merge type (i.e., MRG_TYPE_DEFAULT_N). The prediction direction information indicates the prediction direction among L0, L1, and L0 and L1 predictions. In B slices, the proposed method can generate bidirectional prediction candidates from merge candidates with unidirectional prediction by using a mirroring technique. For example, if the merge candidate is a unidirectional prediction with L1, the reference index of L0 is determined by searching the reference pictures in list 0, which is mirrored with the reference pictures in list 1. If there is no corresponding picture, the nearest reference picture of the current picture is used. L0'

[0045] The MV is obtained by scaling the L1 MV, and the scaling factor is calculated by the POC distance.

[0046] In MMVD, after a merge candidate is selected, it is further extended or refined by signaled MVD information. Further information includes a merge candidate flag, an index specifying the magnitude of motion, and an index indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected as the MV basis. The MMVD candidate flag is signaled to specify which one to use between the first and second merge candidates. The initial MV (i.e., merge candidate) selected from the merge candidate list is also referred to as the basis in this disclosure. After searching the position set, the selected MV candidate is referred to as an extended MV candidate in this disclosure.

[0047] If the prediction direction of the MMVD candidate is the same as one of the original merge candidates, an index with a value of 0 is signaled as the MMVD prediction direction. Otherwise, an index with a value of 1 is signaled. After sending the first bit, the remaining prediction directions are signaled according to a predefined priority order of the MMVD prediction directions. The priority order is L0 / L1 prediction, L0 prediction, and L1 prediction. If the prediction direction of the merge candidate is L1, signaling '0' indicates that the MMVD prediction direction is L1. Signaling '10' indicates that the MMVD prediction direction is L0 and L1. Signaling '11' indicates that the MMVD prediction direction is L0. If the L0 and L1 prediction lists are the same, the prediction direction information of the MMVD is not signaled.

[0048] The base candidate index defines the starting point as shown in Table 1. The base candidate index represents the best candidate among the candidates in the list, as shown below.

[0049] Table 1. Basic candidate IDX

[0050]

[0051] The distance index specifies the motion magnitude information and indicates how Figure 4 The L0 reference block 410 and the L1 reference block 420 are shown as predefined offsets from the starting points (412 and 422). Figure 4 In , an offset is added to the horizontal component or vertical component of the starting MV, where small circles of different styles correspond to different offsets from the center. The relationship between the distance index and the predefined offset is shown in Table 2.

[0052] Table 2. Distance IDX

[0053]

[0054] The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions as shown below. The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions as shown in Table 3. It is worth noting that the meaning of the MVD symbol can change according to the information of the starting MV. When the starting MV is a non-predicted MV or a bidirectionally predicted MV, both lists point to the same side of the current picture (that is, the POCs of both references are greater than the POC of the current picture, or are less than the POC of the current picture), and the symbol in Table 3 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectionally predicted MV, the two MVs point to different sides of the current picture (that is, the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), and the POC difference in list 0 is greater than the POC in list 1, the symbol in Table 3 specifies the sign of the MV offset added to the list 0 MV component of the starting MV, and the sign of the list 1 MV has the opposite value. Otherwise, if the difference of the POC in list 1 is greater than that in list 0, the sign in Table 3 specifies the sign of the MV offset of the MV component of list 1 added to the starting MV, and the sign of the MV of list 0 has the opposite value.

[0055] Table 3. Direction IDX

[0056] Directions to IDX 00 01 10 11 x-axis + – N / A N / A y-axis N / A N / A + –

[0057] To reduce encoder complexity, block restrictions are applied. If the width or height of the CU is less than 4, MMVD is not performed.

[0058] Multi-Hypothesis Prediction (MH) Technology

[0059] Multi-hypothesis prediction is proposed to improve existing inter-prediction modes, including Advanced Motion Vector Prediction (AMVP) mode, Skip and Merge modes, and unidirectional prediction for intra mode. The general concept is to combine existing prediction modes with an additional merge index prediction. Merge index prediction is performed in the same way as regular merge mode, where a merge index is signaled to obtain motion information for motion compensated prediction. The final prediction is a weighted average of the merge index prediction and the prediction generated by the existing prediction modes, where different weights are applied depending on the combination. Detailed information may be found in JVET-K1030 (Chih-Wei Hsu et al., Description of Core Experiment 10: Combination and Multi-hypothesis Prediction, Joint Collaborative Group on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO / IEC JTC 1 / SC 29 / WG11, 11th Meeting: Ljubljana, SI, 10-18 July 2018, Document: JVET-K1030), or JVET-L0100 (Man-Shu Jiang et al., CE10.1.1: Multi-hypothesis prediction for improved AMVP mode, skip or merge mode and intra mode, Joint Collaborative Group on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO / IEC JTC 1 / SC 29 / WG11, 12th Meeting: Macau, China, 3-12 October 2018, Document: JVET-L0100).

[0060] Pairwise Averaged Merge Candidates

[0061] The pairwise average candidates are generated by averaging the predefined candidate pairs in the current merge candidate list, where the predefined pairs are defined as {(0,1),(0,2),(1,2),(0,3),(1,3),(2,3)}, where the number represents the merge index of the merge candidate list. The average motion vector is calculated separately for each reference list. If both motion vectors are available in one list, they are averaged even if they point to different reference pictures; if there is only one motion vector, it is used directly; if no motion vector is available, this list is considered invalid.

[0062] Merge Mode

[0063] In order to improve the coding efficiency of motion vector (MV) coding in HEVC, HEVC has Skip and Merge modes. Skip and Merge modes obtain motion information from spatially adjacent blocks (spatial candidates) or temporally co-located blocks (temporal candidates). When the PU is in Skip or Merge mode, motion information is not encoded, but only the index of the selected candidate is encoded. For Skip mode, the residual signal is forced to zero and is not encoded. In HEVC, if a particular block is encoded as skip or merge, the candidate index is signaled to indicate which candidate in the candidate set is used for merging. Each merged PU reuses the MV, prediction direction, and reference picture index of the selected candidate.

[0064] For the merge mode in HM-4.0 in HEVC, such as Figure 5 As shown, up to four spatial MV candidates are derived from A0, A1, B0 and B1, and from T BR or T CTR Derive a temporal MV candidate (first use T BR , if T BR If not available, use T CTR instead). Note that if any of the four spatial MV candidates is not available, position B2 is used to derive another MV candidate as a substitute. After the derivation process of four spatial MV candidates and one temporal MV candidate, pruning is applied to remove redundant MV candidates. If the number of available MV candidates is less than 5 after redundancy removal, three types of additional candidates are derived and added to the candidate set (candidate list). The encoder selects a final candidate from the candidate set in skip or merge mode based on the rate-distortion optimization (RDO) decision and transmits the index to the decoder.

[0065] Hereinafter, we refer to the skip and merge modes as “merge mode.” In other words, when “merge mode” is mentioned in the following description, “merge mode” may refer to both the skip mode and the merge mode.

[0066] Adaptive Re-ranking of Merge Candidates Using Template Matching (ARMC-TM)

[0067] The merge candidates are adaptively reordered according to the cost evaluated using template matching (TM). The reordering method can be applied to regular merge mode, template matching (TM) merge mode, and affine merge mode (excluding SbTMVP candidates). For TM merge mode, merge candidates are reordered before the refinement process.

[0068] After building the merge candidate list, the merge candidates are divided into multiple subsets. For the normal merge mode and TM merge mode, the subset size is set to 5. For the affine merge mode, the subset size is set to 3. The merge candidates in each subset are reordered in ascending order according to the cost value based on template matching. For ARMC-TM, if the subset meets the following 2 conditions, the candidates in the subset are skipped: (1) the subset is the last subset; (2) the subset is not the first subset. For simplicity, the merge candidates in the last subset (but not the first subset) are not reordered.

[0069] The template matching cost of the merged candidate is measured as the sum of absolute differences (SAD) between the samples of the template of the current block and its corresponding reference samples. The template includes a set of reconstructed samples adjacent to the current block. The reference samples of the template are located by merging the motion information of the candidate.

[0070] When the merge candidate exploits bidirectional prediction, such as Figure 6 As shown, the reference samples of the templates of the merge candidates are also generated by bidirectional prediction. Figure 6 , block 612 corresponds to a current block in current picture 610. As shown, blocks 622 and 632 correspond to reference blocks in reference pictures 620 and 630 in list 0 and list 1, respectively. Templates 614 and 616 are used for current block 612, templates 624 and 626 are used for reference block 622, and templates 634 and 636 are used for reference block 632. Motion vectors 640, 642, and 644 are list 0 merge candidates, and motion vectors 650, 652, 654 are merge candidates in list 1.

[0071] For a sub-block-based merge candidate with a sub-block size equal to Wsub×Hsub, the above template includes multiple sub-templates of size Wsub×1, and the left template includes multiple sub-templates of size 1×Hsub. Figure 7 As shown, the motion information of the sub-block in the first row and first column of the current block is used to derive the reference sample of each sub-template. Figure 7 , block 712 corresponds to a current block in current picture 710, and block 722 corresponds to a collocated block in reference picture 720. Each small square in the current block and the collocated block corresponds to a subblock. The dot-filled areas on the left and top of the current block correspond to the template of the current block. The boundary subblocks are labeled from A to G. The arrows associated with each subblock correspond to the motion vector of the subblock. The reference subblocks (labeled Aref to Gref) are located according to the motion vectors associated with the boundary subblocks.

[0072] Reducing the complexity of MMVD by template matching

[0073] The above-mentioned MMVD technology has been proven to improve coding performance. However, the MMVD technology also brings higher complexity. It is expected to develop a technology that reduces complexity while maintaining performance. In some designs of MMVD with template matching (TM), for each basic MV, it selects K MVD candidates from the total number S*D combination of S step size and D direction. The selection is based on TM cost, where the matching cost is evaluated for each MMVD candidate in the MMVD set (i.e., each candidate member in the MMVD set). The number of MMVD candidates may be large. Therefore, TM matching cost evaluation may lead to high computational complexity. In addition, reference samples of large areas must be obtained on the decoder side. Therefore, a method for solving such problems is disclosed.

[0074] In one approach, TM is applied only to candidates with smaller step sizes. That is, for the first S0 small step sizes, TM-based reordering is performed, and the best K0 candidates are retained for further selection. For the remaining S1=S-S0 step sizes, they are combined with a subset of the D direction to form another K1 candidate, where TM is not performed. In one embodiment, at the CU-level MMVD syntax, a flag is first signaled to indicate whether it is a small step size. If it is a small step size, the subsequent syntax indicates how to select from the K0 candidates; otherwise, the following syntax indicates how to select from the K1 candidates.

[0075] In another approach, TM is applied only within the bounding box. That is, for the MVD candidates within the bounding box, TM-based reordering is performed, and the best K0 candidate is retained for further selection. For the remaining MVD candidates, their step sizes and directions are signaled separately to form another K1 candidate, where TM is not performed. In one embodiment, at the CU-level MMVD syntax, a flag is first signaled to indicate whether TM is used to reorder the K0 candidates. If TM is used, the subsequent syntax indicates how to select from the K0 candidates; otherwise, the subsequent syntax indicates how to select from the K1 candidates.

[0076] Any of the above MMVD methods can be implemented in an encoder and / or decoder. For example, any of the proposed methods can be implemented in an inter-frame coding module (e.g., Figure 1A inter-frame prediction 112 in, motion compensation module (eg, Figure 1BMC 152 in the MMVD method) and the merge candidate derivation module. Alternatively, any of the proposed methods can be implemented as a circuit coupled to the inter-frame coding module and / or motion compensation module of the encoder and the merge candidate derivation module of the decoder. While the inter-frame prediction 112 and MC 152 are shown as independent processing units supporting the MMVD method, they can correspond to executable software or firmware code stored on a medium, such as a hard disk or flash memory, for a CPU (central processing unit) or a programmable device (such as a digital signal processor (DSP) or a field programmable gate array (FPGA).

[0077] Figure 8 A flowchart of another exemplary video coding system for modifying the search position using MMVD according to an embodiment of the present invention is shown. The steps shown in the flowchart can be implemented as executable program codes on one or more processors (e.g., one or more CPUs) on the encoder side. The steps shown in the flowchart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to the method, input data associated with the current block is received in step 810, wherein the input data includes pixel data of the current block to be encoded on the encoder side or prediction residual data associated with the current block to be decoded on the decoder side. In step 820, at least one basic merge MV is determined from the merge list of the current block. In step 830, a set of extended merge candidate sets is determined for the at least one basic merge MV according to the step set and the direction set. The extended merge candidate set is determined by adding an offset to the at least one basic merge MV, and the offset corresponds to a pair of combinations from the step set and the direction set, wherein at least one combination of the step set and the direction set is excluded from the partial set of extended merge candidates. In step 840, candidates in the partial set of extended merge candidates are reordered according to template matching costs associated with the candidates, and each template matching cost is measured between a first sample in one or more first neighboring regions of the current block and a second sample in one or more second neighboring regions of a reference block, wherein the reference block is located according to each candidate in the partial set of extended merge candidates. In step 850, the current block is encoded or decoded by using motion information of the reordered partial set of extended merge candidates.

[0078] The flow chart shown is intended to illustrate an example of video encoding according to the present invention. Without departing from the spirit of the present invention, those skilled in the art may modify each step, rearrange the steps, split the steps, or combine the steps to implement the present invention. In the present disclosure, specific syntax and semantics have been used to illustrate examples to implement embodiments of the present invention. Without departing from the spirit of the present invention, those skilled in the art may implement the present invention by replacing syntax and semantics with equivalent syntax and semantics.

[0079] The above description is provided to enable those of ordinary skill in the art to practice the present invention provided in the context of a specific application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described, but to conform to the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are illustrated to provide a thorough understanding of the present invention. However, those skilled in the art will appreciate that the present invention may be implemented.

[0080] The embodiments of the present invention as described above can be implemented in various hardware, software codes or a combination of the two. For example, one embodiment of the present invention can be one or more circuits integrated into a video compression chip or a program code integrated into the video compression software to perform the processing described herein. The embodiments of the present invention can also be a program code to be executed on a digital signal processor (DSP) to perform the processing described herein. The present invention can also be related to many functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array (FPGA). These processors can be configured to perform specific tasks according to the present invention by executing machine-readable software codes or firmware codes that define the specific methods embodied by the present invention. The software code or firmware code can be developed in different programming languages ​​and different formats or styles. The software code can also be compiled for different target platforms. However, the different code formats, styles and languages ​​of the software code and other ways of configuring the code to perform tasks according to the present invention will not depart from the spirit and scope of the present invention.

[0081] The invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples should be considered in all respects as illustrative and not restrictive. Therefore, the scope of the invention is indicated by the appended claims rather than by the foregoing description. All changes that fall within the meaning and scope of the equivalents of the claims should be included within their scope.

Claims

1. A video encoding and decoding method using a merged motion vector difference (MMVD) mode, the method comprising: Receiving input data related to a current block, wherein the input data includes pixel data of the current block to be encoded at an encoder side or encoded data related to the current block to be decoded at a decoder side; Determining at least one basic merge motion vector (MV) from a merge list of the current block; Determine an extended merge candidate set of the at least one basic merge MV according to a step size set and a direction set, wherein the extended merge candidate set is determined by adding an offset to the at least one basic merge MV, and wherein the offset corresponds to a group of combinations from the step size set and the direction set, wherein at least one combination of the step size set and the direction set is excluded from a partial set of the extended merge candidate set; reordering the candidates in the partial set of extended merge candidates according to template matching costs associated with the candidates, and wherein each template matching cost is measured between a first sample in one or more first neighboring regions of the current block and a second sample in one or more second neighboring regions of a reference block, wherein the reference block is positioned according to each candidate in the partial set of extended merge candidates; and The current block is encoded or decoded by using motion information including the reordered partial set of extended merging candidates.

2. The method according to claim 1, characterized in that The partial set of extended merge candidates is generated according to a pair of restricted combinations from a restricted step size set and the direction set, and wherein the restricted step size set is generated by excluding at least one member step size from the step size set.

3. The method according to claim 1, characterized in that The partial set of extended merge candidates is generated by limiting the extended merge candidate set within a bounding box to exclude at least one extended merge candidate in the extended merge candidate set.

4. The method according to claim 1, characterized in that: A first syntax is signaled or parsed in a coding unit (CU) level to indicate whether the partial set of extended merge candidates is for the current block.

5. The method according to claim 4, characterized in that When the first syntax indicates that the partial set of extended merge candidates is for the current block, a second syntax is signaled or parsed in the CU level to indicate a target candidate selected from the partial set of extended merge candidates for the current block.

6. The method according to claim 4, characterized in that When the partial set of extended merge candidates is not used for the current block, a second syntax is signaled or parsed in the CU level to indicate a target candidate selected from a remaining candidate set of the current block, and wherein the remaining candidate set corresponds to an extended merge candidate that belongs to the extended merge candidate set but is not in the partial set of extended merge candidates.

7. The method according to claim 1, characterized in that The one or more first neighboring regions of the current block include a first top neighboring region and a first left neighboring region of the current block, and the one or more second neighboring regions of the reference block include a second top neighboring region and a second left neighboring region of the reference block.

8. The method according to claim 1, characterized in that A remaining candidate set of the current block is generated for the current block, wherein the remaining candidate set corresponds to extended merge candidates that belong to the extended merge candidate set but are not in the partial extended merge candidates.

9. The method according to claim 8, characterized in that The motion information further includes the remaining candidate set, and the remaining candidate set is not reordered using template matching costs associated with the remaining candidate set.

10. An apparatus for performing video encoding and decoding using MMVD mode, the apparatus comprising one or more electronic devices or processors arranged to: Receive input data related to the current block, where The input data includes pixel data to be encoded related to the current block at the encoder end or prediction residual data to be decoded related to the current block at the decoder end; Determine at least one basic merged MV from the merge list of the current block; Determine an extended merge candidate set of the at least one basic merge MV according to a step size set and a direction set, wherein the extended merge candidate set is determined by adding an offset to the at least one basic merge MV, and wherein the offset corresponds to a pair of combinations from the step size set and the direction set, wherein at least one combination of the step size set and the direction set is excluded from a partial set of extended merge candidates; reordering the candidates in the extended partial set of merge candidates according to template matching costs associated with the candidates in the extended partial set of merge candidates, and wherein each template matching cost is measured between a first sample in one or more first neighboring regions of the current block and a second sample in one or more second neighboring regions of a reference block located according to each candidate member in the extended partial set of merge candidates; as well as The current block is encoded or decoded by using the motion information including the partial set of reordered extended merging candidates.

11. The device according to claim 10, characterized in that The partial set of extended merge candidates is generated according to a pair of restricted combinations from a restricted step size set and the direction set, and wherein the restricted step size set is generated by excluding at least one member step size from the step size set.

12. The device according to claim 10, characterized in that The partial set of extended merge candidates is generated by limiting the extended merge candidate set within a bounding box to exclude at least one extended merge candidate in the extended merge candidate set.

13. The device according to claim 10, characterized in that A first syntax is signaled or parsed in the CU level to indicate whether the partial set of extended merge candidates is for the current block.

14. The device according to claim 13, characterized in that When the first syntax indicates that the partial set of extended merge candidates is for the current block, a second syntax is signaled or parsed in the CU level to indicate a target candidate selected from the partial set of extended merge candidates for the current block.

15. The device according to claim 13, characterized in that When the partial set of extended merge candidates is not used for the current block, a second syntax is signaled or parsed in the CU level to indicate a target candidate selected from a remaining candidate set of the current block, and wherein the remaining candidate set corresponds to an extended merge candidate that belongs to the extended merge candidate set but is not in the partial set of extended merge candidates.

16. The device according to claim 10, characterized in that The one or more first neighboring regions of the current block include a first top neighboring region and a first left neighboring region of the current block, and the one or more second neighboring regions of the reference block include a second top neighboring region and a second left neighboring region of the reference block.

17. The device according to claim 10, characterized in that A remaining candidate set of the current block is generated for the current block, and wherein the remaining candidate set corresponds to an extended merge candidate that belongs to an extended merge candidate set but is not in the partial set of extended merge candidates.

18. The device according to claim 17, characterized in that The motion information further includes the remaining candidate set, and the remaining candidate set is not reordered using template matching costs associated with the remaining candidate set.

Images