Sub-block decoder side motion vector refinement

Abstract

A method for decoding a bitstream including a digital representation of a video using sub-block decoder-side motion vector refinement includes decoding motion information for a current video block from the bitstream, estimating a matching cost for the current video block using one or more templates based on a partial set of pixel positions of the current block, wherein each of the one or more templates includes a video block having a plurality of samples, and refining the motion information for the current block using the template with the minimum matching cost. The method also includes estimating a matching cost for the current video block to be performed by partitioning the current video block into sub-blocks and estimating a matching cost for each sub-block using a corresponding partial set of pixel positions of the sub-blocks.

Description

[0001] This divisional application is a divisional application filed on June 10, 2019, with application number 201910498694.9 and the invention title “Sub-block decoder side motion vector refinement”

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application is filed under the applicable Patent Act and / or Paris Convention to timely claim the benefit of prior U.S. Provisional Patent Application No. 62 / 682,150, filed on June 7, 2018, the entire disclosure of which is incorporated by reference into this disclosure as a part of this application. Technical Field

[0004] This document deals with video coding technology. Background Art

[0005] Despite advances in video compression technology, digital video still accounts for the largest amount of bandwidth usage on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth required for digital video usage is expected to continue to grow. Summary of the Invention

[0006] Disclosed are techniques related to decoder-side motion vector derivation (DMVD) in video coding. The techniques can be applied to existing video coding standards, such as HEVC, or the finalized standard (Versatile Video Coding (VVC)). The techniques can also be applied to future video coding standards or video codecs.

[0007] In one exemplary aspect, a method for decoding a bitstream comprising a digital representation of a video is disclosed. The method includes decoding motion information for a current video block from the bitstream, estimating a matching cost for the current video block using one or more templates based on a partial set of pixel positions of the current block, wherein each of the one or more templates comprises a video block having a plurality of samples, and refining the motion information for the current block using the template having the minimum matching cost. The method also includes estimating a matching cost for the current video block to be performed by partitioning the current video block into sub-blocks and estimating a matching cost for each sub-block using a corresponding partial set of pixel positions of the sub-block.

[0008] In another example aspect, an apparatus is disclosed comprising a processor configured to implement each of the above methods.

[0009] In yet another example aspect, the methods may be embodied in the form of computer-executable instructions and stored on a computer-readable program medium.

[0010] These and other aspects are described further throughout this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 An example of the derivation process for Merge candidate list construction is shown.

[0012] Figure 2 Example locations of spatial merge candidates are shown.

[0013] Figure 3 An example of candidate pairs for redundancy check considering spatial Merge candidates is shown.

[0014] Figure 4A and Figure 4B Example locations of the second PU for Nx2N and 2NxN partitions are shown.

[0015] Figure 5 is an example illustration of motion vector scaling for temporal merge candidates.

[0016] Figure 6 An example of candidate positions of the temporal merge candidates C0 and C1 is shown.

[0017] Figure 7 An example of a combined bi-predictive Merge candidate is shown.

[0018] Figure 8 An example derivation process of motion vector prediction candidates is shown.

[0019] Figure 9 An example illustration of motion vector scaling for spatial motion vector candidates is shown.

[0020] Figure 10 An example of bilateral matching is shown.

[0021] Figure 11 An example of template matching is shown.

[0022] Figure 12 An example of one-sided ME in FRUC is shown.

[0023] Figure 13 An example of DMVR based on bilateral template matching is shown.

[0024] Figure 14 An example of a simplified template in template matching is shown.

[0025] Figure 15 An example of ATMVP motion prediction for a CU is shown.

[0026] Figure 16 An example of one CU having four sub-blocks (AD) and their neighboring blocks (ad) is shown.

[0027] Figure 17 This is a description of the sub-block to which OBMC is applied.

[0028] Figure 18 An example of neighboring samples used to derive IC parameters is shown.

[0029] Figure 19 is a flow chart of an example method for video decoding.

[0030] Figure 20 is a block diagram of a video decoding device.

[0031] Figure 21 An example implementation of a video encoder is shown. DETAILED DESCRIPTION

[0032] This document provides various techniques that can be used by decoders of video bitstreams to improve the quality of decompressed or decoded digital video. In addition, video encoders can also implement these techniques during the encoding process to reconstruct decoded frames for further encoding.

[0033] For ease of understanding, section headings are used in this document and do not limit the embodiments and techniques to the corresponding sections. Thus, embodiments from one section can be combined with embodiments from other sections. In addition, although some embodiments describe video encoding steps in detail, it should be understood that decoding by the decoder will undo the corresponding steps of encoding. In addition, the term video processing includes video encoding or compression, video decoding or decompression, and video transcoding, in which video pixels are represented from one compression format to another compression format or at different compression bit rates.

[0034] 1. Technical Framework

[0035] Video coding standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T produced H.261 and H.263, ISO / IEC produced MPEG-1 and MPEG-4 Visual, and the two organizations jointly produced the H.262 / MPEG-2 Video, H.264 / MPEG-4 Advanced Video Coding (AVC), and H.265 / HEVC standards. Starting with H.262, video coding standards have been based on a hybrid video coding structure that uses temporal prediction plus transform coding. To explore future video coding technologies beyond HEVC, the Joint Video Exploration Team (JVET) was jointly established by VCEG and MPEG in 2015. Since then, many new methods have been adopted by JVET and incorporated into reference software called the Joint Exploration Model (JEM). In April 2018, the Joint Video Expert Team (JVET) between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) was created for the VVC standard, with the goal of reducing bitrate by 50% compared to HEVC.

[0036] 2. Inter-frame prediction in HEVC / H.265

[0037] Each inter-predicted prediction unit (PU) has motion parameters for one or two reference picture lists. The motion parameters include a motion vector and a reference picture index. The use of one of the two reference picture lists can also be signaled using inter_pred_idc. The motion vector can be explicitly encoded as a delta relative to the predictor.

[0038] When a Coding Unit (CU) is encoded in skip mode, one PU is associated with the CU and there are no significant residual coefficients, no coded motion vector increments, or reference picture indices. Merge mode is specified so that the motion parameters of the current PU are obtained from neighboring PUs, including spatial and temporal candidates. Merge mode can be applied to any inter-predicted PU, not only to skip mode. An alternative to Merge mode is explicit transmission of motion parameters, where for each PU, the motion vector (more precisely, the motion vector difference compared to the motion vector predictor), the corresponding reference picture index for each reference picture list, and the reference picture list used are explicitly signaled. In this document, this mode is referred to as Advanced Motion Vector Prediction (AMVP).

[0039] When signaling indicates that one of two reference picture lists will be used, a PU is generated from a block of samples. This is called "uni-prediction." Uni-prediction can be used for both P and B slices.

[0040] When signaling indicates that two reference picture lists are to be used, a PU is generated from two sample blocks. This is called "bi-prediction." Bi-prediction is applicable only to B slices.

[0041] The following text provides details on the inter prediction modes specified in HEVC. The description starts with the Merge mode.

[0042] 2.1.1 Merge Mode

[0043] 2.1.1.1 Merge Mode Candidate Inference

[0044] When using Merge mode to predict a PU, the index pointing to the entry in the Merge candidate list is parsed from the bitstream and used to retrieve the motion information. The construction of this list is specified in the HEVC standard and can be summarized according to the following steps in sequence:

[0045] Step 1: Initial candidate derivation

[0046] Step 1.1: Spatial Candidate Derivation

[0047] Step 1.2: Redundancy check of spatial candidates

[0048] Step 1.3: Temporal Candidate Derivation

[0049] Step 2: Insert additional candidates

[0050] Step 2.1: Create bidirectional prediction candidates

[0051] Step 2.2: Insert zero motion candidates

[0052] These steps are also Figure 1 Schematically depicted in . For spatial Merge candidate derivation, a maximum of four Merge candidates are selected from candidates located at five different positions. For temporal Merge candidate derivation, a maximum of one Merge candidate is selected from two candidates. Since a constant number of candidates is assumed at the decoder for each PU, additional candidates are generated when the number of candidates obtained from step 1 does not reach the maximum number of Merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, truncated unary binarization (TU) is used to encode the index of the best Merge candidate. If the size of the CU is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

[0053] Hereinafter, operations related to the above steps are described in detail.

[0054] 2.1.1.2 Spatial Candidate Derivation

[0055] In the derivation of spatial Merge candidates, Figure 2 Up to four Merge candidates are selected from the candidates at the depicted positions. The order of derivation is A1, B1, B0, A0 and B2. Position B2 is considered only when any PU at position A1, B1, B0, A0 is unavailable (for example, because it belongs to another slice or block) or is intra-coded. After adding the candidate at position A1, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, resulting in improved coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only the Figure 3 The pairs connected by arrows in , and candidates are added to the list only if the corresponding candidates used for redundancy checking have different motion information. Another source of repeated motion information is a "second PU" associated with a partition other than 2N×2N. As an example, Figure 4A and Figure 4BThe second PU is depicted for the N×2N and 2N×N cases, respectively. When the current PU is partitioned as N×2N, the candidate at position A1 is not considered for list construction. In fact, adding this candidate would result in two prediction units with the same motion information, which is redundant for a coding unit with only one PU. Similarly, position B1 is not considered when the current PU is partitioned as 2N×N.

[0056] 2.1.1.3 Time Candidate Derivation

[0057] In this step, only one candidate is added to the list. Specifically, in the derivation of this temporal merge candidate, the scaled motion vector is derived based on the co-located PU that belongs to the picture with the smallest POC difference with the current picture within a given reference picture list. The reference picture list to be used for derivation of the co-located PU is explicitly signaled in the slice header. Figure 5 The dotted line in the figure shows the scaled motion vector obtained for the temporal merge candidate, which is scaled from the motion vector of the co-located PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero. The actual implementation of the scaling process is described in the HEVC specification. For B slices, two motion vectors are obtained, one for reference picture list 0 and the other for reference picture list 1, and these two motion vectors are combined to obtain a bidirectional prediction merge candidate.

[0058] Among the co-located PU(Y) belonging to the reference frame, the position of the temporal candidate is selected between candidates C0 and C1, as Figure 6 If the PU at position C0 is not available, is intra-coded, or is outside the current CTU row, position C1 is used. Otherwise, position C0 is used for the derivation of temporal merge candidates.

[0059] 2.1.1.4 Inserting Additional Candidates

[0060] In addition to spatial and temporal Merge candidates, there are two additional types of Merge candidates: combined bi-predictive Merge candidates and zero Merge candidates. Combined bi-predictive Merge candidates are generated by utilizing spatial and temporal Merge candidates. Combined bi-predictive Merge candidates are only used for B slices. Combined bi-predictive candidates are generated by combining the first reference picture list motion parameters of an initial candidate with the second reference picture list motion parameters of another candidate. If the two tuples provide different motion hypotheses, they will form a new bi-predictive candidate. As an example, Figure 7 Depicted is the case when two candidates with mvL0 and refIdxL0 or mvL1 and refIdxL1 in the original list (left) are used to create combined bi-predictive Merge candidates that are added to the final list (right). Many rules are defined in the HEVC standard regarding the combinations that are considered to generate these additional Merge candidates.

[0061] Zero-motion candidates are inserted to fill the remaining entries in the Merge candidate list, thus reaching the MaxNumMergeCand capacity. These candidates have zero spatial displacement and a reference picture index that starts at zero and increases each time a new zero-motion candidate is added to the list. The number of reference frames used by these candidates is 1 and 2, for unidirectional and bidirectional prediction, respectively. Finally, no redundancy check is performed on these candidates.

[0062] 2.1.1.5 Motion Estimation Regions for Parallel Processing

[0063] To speed up the encoding process, motion estimation can be performed in parallel, thereby deriving motion vectors for all prediction units within a given region at the same time. Deriving merge candidates from spatial neighbors may interfere with parallel processing because a prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is complete. To alleviate the trade-off between coding efficiency and processing latency, HEVC defines a motion estimation region (MER), the size of which is signaled in the picture parameter set using the HEVC syntax element "log2_parallel_merge_level_minus2". When an MER is defined, merge candidates that fall into the same region are marked as unavailable and are therefore not considered in list construction.

[0064] 2.1.2 AMVP

[0065] AMVP exploits the spatiotemporal correlation of motion vectors with neighboring PUs for explicit transmission of motion parameters. For each reference picture list, a motion vector candidate list is constructed by first checking the availability of temporally adjacent PU positions above and to the left, removing redundant candidates and adding zero vectors to make the candidate list a constant length. The encoder can then select the best predictor from the candidate list and send the corresponding index indicating the selected candidate. Similar to Merge index signaling, the index of the best motion vector candidate is encoded using truncated unary. The maximum value to be encoded in this case is 2 (see Figure 8 ). In the following section, details on the derivation process of motion vector prediction candidates are provided.

[0066] 2.1.2.1 Derivation of AMVP Candidates

[0067] Figure 8 The derivation process of motion vector prediction candidates is summarized.

[0068] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For spatial motion vector candidate derivation, the final Figure 2 The motion vectors of each PU at the five different positions shown derive two motion vector candidates.

[0069] For temporal motion vector candidate derivation, a motion vector candidate is selected from two candidates derived based on two different co-located positions. After generating the first list of spatiotemporal candidates, duplicate motion vector candidates are removed from the list. If the number of potential candidates is greater than 2, motion vector candidates whose reference picture index within the associated reference picture list is greater than 1 are removed from the list. If the number of spatiotemporal motion vector candidates is less than 2, an additional zero motion vector candidate is added to the list.

[0070] 2.1.2.2 Spatial Motion Vector Candidates

[0071] In the derivation of spatial motion vector candidates, a maximum of two candidates are considered among five potential candidates, which are located at Figure 2 The derivation order of the left side of the current PU is defined as A0, A1, and scaled A0, scaled A1. The derivation order of the upper side of the current PU is defined as B0, B1, B2, scaled B0, scaled B1, scaled B2. Therefore, for each side, there are four cases that can be used as motion vector candidates, two of which do not require spatial scaling and two use spatial scaling. The four different cases are summarized as follows:

[0072] No spatial scaling

[0073] -(1) Same reference picture list, and same reference picture index (same POC) -(2) Different reference picture list, but same reference picture (same POC)

[0074] Spatial scaling

[0075] -(3) Same reference picture list, but different reference pictures (different POC)

[0076] -(4) Different reference picture lists, and different reference pictures (different POCs)

[0077] First, check for no spatial scaling, then check for spatial scaling. Spatial scaling is considered when the POC differs between the reference picture of the neighboring PU and the reference picture of the current PU, regardless of the reference picture list. If all PUs of the left candidate are unavailable or intra-coded, scaling of the above motion vectors is allowed to facilitate the parallel derivation of the left and above MV candidates. Otherwise, spatial scaling of the above motion vectors is not allowed.

[0078] During spatial scaling, the motion vectors of neighboring PUs are scaled in a similar way to temporal scaling, e.g. Figure 9 The main difference is that the reference picture list and the index of the current PU are given as input; the actual scaling process is the same as the temporal scaling process.

[0079] 2.1.2.3 Temporal Motion Vector Candidates

[0080] Except for the reference picture index derivation, all the processes for deriving temporal merge candidates are the same as those for deriving spatial motion vector candidates (see Figure 6 ). The reference picture index is signaled to the decoder.

[0081] 2.2 New inter-frame prediction method in JEM

[0082] 2.2.1 Motion Vector Derivation by Pattern Matching

[0083] The Pattern Matched Motion Vector Derivation (PMMVD) mode is a special Merge mode based on the Frame-Rate Up Conversion (FRUC) technique. In this mode, the motion information of the block is derived at the decoder side instead of being signaled.

[0084] When the CU's Merge flag is true, the FRUC flag is signaled to the CU. When the FRUC flag is false, the Merge index is signaled and the normal Merge mode is used. When the FRUC flag is true, an additional FRUC mode flag is signaled to indicate which method (bilateral matching or template matching) will be used to derive the motion information for the block.

[0085] On the encoder side, the decision on whether to use the FRUC merge mode for a CU is based on the RD cost selection made for normal merge candidates. That is, both matching modes (bilateral matching and template matching) for the CU are checked using RD cost selection. The matching mode that results in the lowest cost is further compared with the other CU modes. If the FRUC matching mode is the most efficient mode, the FRUC flag is set to true for the CU, and the relevant matching mode is used.

[0086] The motion inference process in FRUC Merge mode has two steps: first, a CU-level motion search is performed, followed by sub-CU-level motion refinement. At the CU level, an initial motion vector for the entire CU is derived based on bilateral matching or template matching. First, a list of MV candidates is generated, and the candidate that results in the minimum matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bilateral matching or template matching is performed near the starting point, and the resulting MV with the minimum matching cost is used as the MV for the entire CU. Subsequently, the motion information is further refined at the sub-CU level, using the derived CU motion vector as the starting point.

[0087] For example, the following derivation process is performed for the W×HCU motion information derivation. In the first stage, the MV of the entire W×HCU is derived. In the second stage, the CU is further divided into M×M sub-CUs. The value of M is calculated as shown in (1), and D is a predefined partition depth, which is set to 3 by default in JEM. The MV of each sub-CU is then derived.

[0088]

[0089] like Figure 10 As shown in Figure 1, bilateral matching is used to derive the motion information of the current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. Under the assumption of continuous motion trajectory, the motion vectors MV0 and MV1 pointing to the two reference blocks should be proportional to the temporal distance between the current picture and the two reference pictures (i.e., TD0 and TD1). As a special case, when the current picture is temporally between the two reference pictures and the temporal distance from the current picture to the two reference pictures is the same, bilateral matching becomes mirror-based bidirectional MV.

[0090] like Figure 11As shown, template matching is used to derive the motion information of the current CU by finding the closest match between the template in the current picture (the top and / or left adjacent blocks of the current CU) and the block in the reference picture (the same size as the template). In addition to the above-mentioned FRUC Merge mode, template matching is also applied to AMVP mode. In JEM, as done in HEVC, there are two candidates for AMVP. New candidates are derived through the template matching method. If the candidate newly derived by template matching is different from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and then the list size is set to 2 (meaning removing the second existing AMVP candidate). When applied to AMVP mode, only CU-level search is applied.

[0091] 2.2.2 CU-level MV candidate set

[0092] The MV candidates set at the CU level include:

[0093] (i) If the current CU is in AMVP mode, it is the original AMVP candidate

[0094] (ii) All Merge candidates,

[0095] (iii) Several MVs in the interpolated MV field introduced in Section 2.2.4.

[0096] (iv) Top and left neighboring motion vectors

[0097] When bilateral matching is used, each valid MV of the Merge candidate is used as input to generate an MV pair assuming bilateral matching. For example, in reference list A, one valid MV of the Merge candidate is (MVa, refa). Then, the reference picture refb of its paired bilateral MV is found in the other reference list B, so that refa and refb are temporally located on different sides of the current picture. If such refb is not available in reference list B, refb is determined to be a reference different from refa, and its temporal distance to the current picture is the minimum in list B. After determining refb, MVb is derived by scaling MVa based on the temporal distance between refa and refb in the current picture.

[0098] The four MVs from the interpolated MV field are also added to the CU-level candidate list. More specifically, the interpolated MVs at positions (0, 0), (W / 2, 0), (0, H / 2), and (W / 2, H / 2) of the current CU are added.

[0099] When FRUC is applied in AMVP mode, the original AMVP candidates are also added to the CU-level MV candidate set.

[0100] At the CU level, for an AMVP CU, up to 15 MVs are added to the candidate list, and for a Merge CU, up to 13 MVs are added to the candidate list.

[0101] 2.2.3 Sub-CU Level MV Candidate Set

[0102] The MV candidates set at the sub-CU level include:

[0103] (i) MV determined from CU-level search,

[0104] (ii) neighboring MVs at the top, left, upper-left, and upper-right corners,

[0105] (iii) a scaled version of the collocated MV from the reference picture,

[0106] (iv) Up to 4 ATMVP candidates,

[0107] (v) Up to 4 STMVP candidates

[0108] The scaled MV from the reference picture is derived as follows: All reference pictures in the two lists are traversed. The MV at the collocated position of the sub-CU in the reference picture is scaled to the reference of the starting CU level MV.

[0109] ATMVP and STMVP candidates are limited to the top four.

[0110] At the sub-CU level, up to 17 MVs are added to the candidate list.

[0111] 2.2.4 Generation of interpolated MV fields

[0112] Before encoding a frame, an interpolated motion field is generated for the entire picture based on unilateral ME. The motion field can then be used later as a CU-level or sub-CU-level MV candidate.

[0113] First, the motion fields of each reference picture in both reference lists are traversed at the 4×4 block level. For each 4×4 block, if the motion associated with the block passes through a 4×4 block in the current picture (e.g. Figure 12 (as shown) and the block is not assigned any interpolated motion, the motion of the reference block is scaled to the current picture according to the temporal distances TD0 and TD1 (in the same way as the MV scaling of TMVP in HEVC), and the scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to the 4×4 block, the motion of the block is marked as unavailable in the interpolated motion field.

[0114] 2.2.5 Interpolation and Matching Cost

[0115] When the motion vector points to a fractional sample location, motion compensated interpolation is required. To reduce complexity, bilinear interpolation is used for bilateral matching and template matching instead of conventional 8-tap HEVC interpolation.

[0116] The calculation of the matching cost is slightly different at different steps. When selecting a candidate from the candidate set at the CU level, the matching cost is the absolute sum difference (SAD) of the bilateral match or template match. After determining the starting MV, the matching cost of the bilateral match of the sub-CU level search is calculated as follows:

[0117]

[0118] where w is a weighting factor set to 4 based on experience, MV and MV s Indicates the current MV and the starting MV respectively. SAD is still used as the matching cost of template matching in sub-CU level search.

[0119] In FRUC mode, MV is derived by using only luma samples. The derived motion will be used for both luma and chroma for MC inter-frame prediction. After determining the MV, the final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma.

[0120] 2.2.6 MV Refinement

[0121] MV refinement is a pattern-based MV search with bilateral matching cost or template matching cost as the criterion. In JEM, two search modes are supported - Unrestricted Center-Biased Diamond Search (UCBDS) and Adaptive Cross Search, which perform MV refinement at the CU level and sub-CU level, respectively. For both CU and sub-CU level MV refinement, the MV is directly searched with quarter luma sample MV precision, followed by eighth luma sample MV refinement. The search range for MV refinement for both CU and sub-CU steps is set to equal 8 luma samples.

[0122] 2.2.7 Selection of prediction direction in template matching FRUC Merge mode

[0123] In bilateral matching Merge mode, bidirectional prediction is always applied because the motion information of the CU is derived based on the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. There is no such restriction for template matching Merge mode. In template matching Merge mode, the encoder can choose from unidirectional prediction in list 0, unidirectional prediction in list 1, or bidirectional prediction for the CU. The selection is based on the template matching cost as follows:

[0124] If costBi<=factor*min(cost0,cost1)

[0125] Bidirectional prediction is used;

[0126] Otherwise, if cost0 <= cost1

[0127] Then use the one-way prediction in list 0;

[0128] otherwise,

[0129] Use the one-way prediction in Listing 1;

[0130] Where cost0 is the SAD of template matching for list 0, cost1 is the SAD of template matching for list 1, and costBi is the SAD of template matching for bidirectional prediction. The value of factor is equal to 1.25, which means that the selection process is biased towards bidirectional prediction.

[0131] Inter prediction direction selection is only applied to the CU-level template matching process.

[0132] 2.2.8 Decoder-side Motion Vector Refinement

[0133] In bidirectional prediction, for the prediction of a block region, two prediction blocks formed using motion vectors (MVs) from list0 and list1 are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of the bidirectional prediction are further refined through a bilateral template matching process. Bilateral template matching is applied in the decoder to perform a distortion-based search between the bilateral template and the reconstructed samples in the reference picture to obtain the refined MV without transmitting additional motion information.

[0134] In DMVR, the bilateral template is generated as a weighted combination (i.e., average) of two prediction blocks from the initial MV0 of list 0 and MV1 of list 1, respectively, as Figure 13 As shown. The template matching operation consists of calculating a cost metric between the generated template and the sample area (around the initial prediction block) in the reference picture. For each of the two reference pictures, the MV that produces the minimum template cost is considered as the updated MV of the list to replace the original MV. In JEM, nine MV candidates are searched for each list. The nine MV candidates include the original MV and 8 surrounding MVs that have a luma sample offset from the original MV in the horizontal or vertical direction or in both directions. Finally, the two new MVs, i.e. Figure 13MV0' and MV1' shown in are used to generate the final bidirectional prediction result. The sum of absolute differences (SAD) is used as the cost metric. Note that when calculating the cost of a prediction block generated by a surrounding MV, the rounded MV (to integer pixels) is actually used instead of the true MV to obtain the prediction block.

[0135] DMVR is applied to the Merge mode of bi-prediction, where one MV comes from the past reference picture and the other MV comes from the future reference picture, without transmitting additional syntax elements. In JEM, DMVR is not applied when LIC, affine motion, FRUC or sub-CU Merge candidate is enabled for the CU.

[0136] 2.2.9 Example of Problem

[0137] DMVD methods such as DMVR and FRUC perform motion estimation to derive or refine motion information, which is very complex for the decoder. During motion estimation, they share a common problem: the difference (absolute difference, squared difference, etc.) between the template and the candidate block is calculated for all pixels in the block and summed, which is then used to select the best matching block. This is not necessary, as the difference of a few pixels may be sufficient to select the best candidate block or MV. Also, typically only the luma component is used in the derivation or refinement of the motion vector, without considering the chroma components.

[0138] DMVR has another complexity issue: it performs motion compensation twice, once to generate the template and once to generate the final prediction block. As a result, for each reference picture list (i.e., prediction direction), it performs both horizontal and vertical interpolation twice, in case the initial MV and the refined MV have only fractional components. This greatly increases the complexity of the worst case. At the same time, DMVR only works in Merge mode, not in AMVP mode. In MV refinement, it uses the signaled MV (the MV derived from the Merge candidate) as the starting MV and verifies the MVs surrounding it. However, the MV accuracy of the signaled MV is not taken into account. In AMVR, a low-precision MV may be selected. For example, assuming the highest allowed MV accuracy is 1 / 4 pixel, in AMVR, a 4-pixel or 1-pixel MV can be used. In this case, DMVR can be used to refine the MV accuracy. Unlike FRUC, which can be applied at the sub-block level, DMVR is performed at the block level, except in the ATMVP and STMVP cases, which may result in a loss in coding performance.

[0139] For FURC, when performing bilateral matching, the MV difference between the starting MV and the candidate MV is considered to suppress unreliable motion vectors, as shown in Equation 2. The MV difference is multiplied by a fixed weighting factor, which may not be reasonable. For larger blocks, SAD plays a dominant role and the MV difference is negligible, while for smaller blocks, the MV difference may be too large.

[0140] 2.2.10 Example Embodiments

[0141] We propose several aspects to reduce the complexity of DMVD methods and improve coding performance. The disclosed approach can be applied to existing DMVD methods, but can also be applied to future methods of motion / mode derivation at the decoder side.

[0142] First, only a subset of pixels are used in decoder-side motion estimation, i.e., during motion information derivation or refinement, the cost between the template and the candidate block is calculated (e.g., taking into account the difference in distortion and MV, distortion, or cost). Second, with DMVR, the number of interpolation steps is reduced. Third, some embodiments of the disclosed technology are used to apply DMVR to AMVP mode. Fourth, the weighting factor for MV differences can be different for different block sizes.

[0143] The following examples provide some ways in which the disclosed technology can be implemented in the video decoding process.

[0144] Let prec be the motion vector precision. When prec is equal to N, it means the motion vector has 1 / 2^N pixel precision. N can be a positive integer, zero, or a negative integer.

[0145] 1. Calculate the cost (eg, difference) between the template and the candidate block only for partially selected rows in the motion information derivation or refinement process.

[0146] a. In one example, the selected rows are defined as the i-th row of all N rows, where N>1 and 1<=i<=N. For example, N is equal to 2 and i is equal to 1.

[0147] b. In one example, for each group of N rows, some rows within the group are used as selected rows. For example, the first and second rows of every 4 rows are used.

[0148] c. In one example, the cost is calculated for any selected rows of the block, for example, the first and last row, or the first two rows and the last two rows.

[0149] d. When selecting partial rows, the same rules may be applied to all block sizes. Alternatively, different rules may be applied to different block sizes and / or block shapes (e.g., square or rectangular or the ratio between block width and block height).

[0150] i. In one example, during cost calculation, more rows are skipped for larger block sizes, and vice versa. For example, when the block size is less than 16×16 (i.e., width*height<16*16), the difference is calculated for the first row of every 2 rows, but for other block sizes, the difference is calculated for the first row of every 4 rows.

[0151] ii. In one example, during cost calculation, more rows are skipped for block shapes with larger heights, and vice versa. For example, when the block height is less than 16, the cost is calculated for the first row of every 2 rows, but for other block sizes, the cost is calculated for the first row of every 4 rows.

[0152] iii. In one example, this simplification is only applied to one or a few minimum block sizes (ie, minimum width*height) to suppress the worst-case complexity. For example, the simplification is only applied to blocks with an area smaller than 8×8.

[0153] iv. In one example, this simplification is applied only to one or a few largest block sizes. For example, the simplification is applied only to blocks larger than 32×32 in area.

[0154] v. In one example, this simplification is applied only to one or a few block shapes with the largest block height or width.

[0155] vi. In one example, this simplification is applied only to some selected block shapes.

[0156] 2. For each row of a block or each selected row of a block, calculate the cost of all columns or only some of the columns.

[0157] a. In one example, calculate the cost of M consecutive columns (which can start at any valid column Y) for each T column, where T>0, 1<=M<=T, 1<=Y<=T-M+1. For example, T=8, M=4 and Y=1.

[0158] b. In one example, the cost of the M selected columns is calculated for every T columns.

[0159] c. In one example, the cost of M arbitrarily selected columns of a row (eg, the first K columns and the last L columns) is calculated.

[0160] d. When selecting a partial column, the same rule can be applied to all block sizes. Alternatively, different rules can be applied to different block sizes and / or block shapes (e.g., square or rectangular or the ratio between block width and block height).

[0161] i. In one example, during cost calculation, more columns are skipped for larger block sizes, and vice versa. For example, when the block size is less than 16×16, the difference is calculated for the first 4 columns of every 8 columns, but for other block sizes, the difference is calculated for the first 4 columns of every 16 columns. When the number of columns is less than 8 or 16, only the first 4 columns are used to calculate the difference.

[0162] ii. In one example, during cost calculation, more columns are skipped for block shapes with larger widths, and vice versa. For example, when the width of the block is less than 16, the cost of the first 4 columns of every 8 columns is calculated, but for other block sizes, the cost of the first 4 columns of every 16 columns is calculated.

[0163] iii. In one example, this simplification is applied only to one or a few minimum block sizes to suppress the worst-case complexity.

[0164] iv. In one example, this simplification is applied only to one or a few maximum block sizes.

[0165] v. In one example, this simplification is applied only to one or a few block shapes with the largest block width.

[0166] vi. In one example, this simplification is applied only to some selected block shapes.

[0167] 3. In DMVR, when generating a template, motion compensation is performed using an MV with integer horizontal or vertical components or an integer MV, rather than using a true MV as in JEM.

[0168] a. In one example, for both prediction directions, MVs (both horizontal and vertical components) are rounded to integer precision.

[0169] b. In one example, the MV of one prediction direction is rounded to integer precision, while the MVs of other prediction directions are not changed.

[0170] c. In one example, for each prediction direction, only one MV component (horizontal component or vertical component) is rounded to integer precision.

[0171] d. In one example, the MV of one prediction direction is rounded to integer precision, and only one MV component of the other prediction direction is rounded to integer precision.

[0172] e. In one example, the MV of one prediction direction is not changed, and only one MV component of the other prediction directions is rounded to integer precision.

[0173] f. Express fmv as the fraction mv and imv as the rounded integer precision mv. Express sign(x) as the sign of x, and

[0174] i.imv=(fmv+(1《K(prec-1)))>>prec

[0175] ii. Alternatively, imv=fmv>>prec

[0176] iii. Alternatively, imv = (fmv + sign (fmv) * (1 < (prec - 1))) > > prec

[0177] g. This simplification can be applied to all block sizes or only to one or a few block sizes and / or certain block shapes.

[0178] i. In one example, it is applied to one or a few minimum block sizes, such as 4x4 in JEM or BMS (benchmark sets), or 4x8 and 8x4 in HEVC.

[0179] ii. In one example, it is applied to one or several maximum block sizes.

[0180] iii. In one example, this is applied to certain selected block sizes.

[0181] 4. Alternatively, in DMVR, when generating the template, a shorter tap interpolation filter (such as a bilinear filter) is used in motion compensation.

[0182] 5. We propose that DMVR be performed at the sub-block level. A block can be divided into sub-blocks in different ways.

[0183] a. In one example, all blocks are divided into fixed M×N sub-block sizes, e.g., 4×4, 4×8, 8×4, 8×8, 8×16, 16×8, 16×16, etc. When the block width / height is an integer multiple of the sub-block width / height, it is divided into sub-blocks; otherwise, it is not divided into sub-blocks.

[0184] b. In one example, a block is divided into K sub-blocks of equal size, where K >= 2. For example, an M×N block is divided into 4 (M / 2)×(N / 2) sub-blocks, or 2 (M / 2)×N sub-blocks, or 2 M×(N / 2) blocks.

[0185] c. In one example, the partitioning method depends on the block size or block shape or other coding information. For example, an 8×32 block is partitioned into 4×8 sub-blocks, and a 32×8 block is partitioned into 8×4 sub-blocks.

[0186] d. In one example, when generating the template of the sub-block, the derived motion information of the entire block can be utilized, as in the current block-level DMVR.

[0187] i. Alternatively, the template may be formed using refined motion information of neighboring sub-block(s), with or without derived motion information for the entire block.

[0188] e. In one example, the search point of a sub-block may also consider refined motion information from other sub-block(s).

[0189] 6. In one example, (in PMMVD) template matching uses a template that only includes pixels above the current block, but not pixels to the left of the current block, e.g. Figure 14 shown.

[0190] 7. In existing DMVD methods, only the luma component is considered to derive or refine the motion vector. It is proposed to also consider the chroma components. Let Ci denote the cost of the three color components of a given motion vector (where i indicates the color component index).

[0191] a. The final cost is defined as Wi*Ci, where Wi indicates the weight of the i-th color component.

[0192] b. Alternatively, the final cost is defined as (W0*C0+W1*(C1+C2)). In some examples, W0 or W1 is equal to 1.

[0193] c. In one example, when DMVR is applied to chroma components, rounding of motion vectors may be applied so that integer motion vectors may be utilized without the need to apply interpolation to the chroma components.

[0194] d. In one example, when DMVR is applied to chroma components, if interpolation is required, a shorter tap interpolation filter (such as a bilinear filter) may be applied.

[0195] 8. The above method can be applied to some color components or all color components.

[0196] a. Different rules may be applied to different color components, or luma and chroma components may utilize different rules.

[0197] b. Alternatively, how and whether to apply the above method can be further signaled in the sequence parameter set, picture parameter set, slice header, etc.

[0198] Figure 191 is a flow chart of an example method 1900 for video decoding. Method 1900 includes decoding (1902) motion information for a current video block from a bitstream, partitioning (1904) the current video block into a plurality of sub-blocks, refining (1906) the motion information for at least one of the plurality of sub-blocks, and reconstructing (1908) the current video block using the refined motion information. In some embodiments, the refining step is performed by estimating a matching cost for each sub-block using one or more templates corresponding to the sub-blocks. For example, each of the one or more templates includes a video block having a plurality of samples. Refining may include selecting a template having a minimum matching cost for the corresponding sub-block.

[0199] In some embodiments, the dividing step includes dividing the current video block into sub-blocks of only square size.

[0200] In some embodiments, the dividing step includes dividing the current video block into sub-blocks of equal size.

[0201] In some embodiments, the step of dividing the current video block into sub-blocks from at least two groups, the at least two groups being selected from the group consisting of: a first group of sub-blocks having a square dimension, a second group of sub-blocks having a vertical rectangular dimension, and a third group of sub-blocks having a horizontal rectangular dimension. In some embodiments, the division depends on a coding type of the current video block.

[0202] Section 2.2.10 provides additional example embodiments and variations that may be implemented by method 1900 .

[0203] Figure 20A block diagram of an example embodiment of a hardware device 2000 that can be used to implement various parts of the technology disclosed in the present invention is shown. The hardware device 2000 can be a laptop, a smartphone, a tablet computer, a camera, or other types of devices capable of processing video. The device 2000 includes a processor or controller 2002 that processes data, and a memory 2004 that communicates with the processor 2002, stores, and / or buffers data. For example, the processor 2002 may include a central processing unit (CPU) or a microcontroller unit (MCU). In some embodiments, the processor 2002 may include a field-programmable gate array (FPGA). In some embodiments, the device 2000 includes or communicates with a graphics processing unit (GPU), a video processing unit (VPU), and / or a wireless communication unit to implement various visual and / or communication data processing functions of the smartphone device. For example, the memory 2004 may include and store processor executable code that, when executed by the processor 2002, configures the device 2000 to perform various operations, such as receiving information, commands, and / or data, processing the information and data, and sending or providing the processed information / data to another device, such as an actuator or an external display. To support the various functions of the device 2000, the memory 2004 may store information and data, such as instructions, software, values, images, and other data processed or referenced by the processor 2002. For example, various types of random access memory (RAM) devices, read-only memory (ROM) devices, flash memory devices, and other suitable storage media may be used to implement the storage function of the memory 2004. The device 2000 may also include dedicated video processing circuitry 2006 for performing repetitive computing functions, such as transforms and decoding.

[0204] Figure 21 is a block diagram illustrating an example implementation of a video encoder. The video encoder operates on video pictures that are encoded using coding tools such as transforms, motion estimation, and residual error coding. The encoded video frames are reconstructed (motion compensated) at the encoder and used as reference pictures for inter-frame coding of other video pictures. The techniques described in this document can be used by a video encoder or a video decoder using techniques such as those described in [1]. Figure 20 The described hardware platform is implemented.

[0205] It will be appreciated from the foregoing that specific embodiments of the presently disclosed technology have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of the invention. Accordingly, the presently disclosed technology is not to be restricted, except as in the appended claims.

[0206] The disclosed and other embodiments, modules, and functional operations described in this document may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or in any combination thereof. The disclosed and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by or to control the operation of a data processing apparatus. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter that effects a machine-readable propagated signal, or any combination thereof. The term "data processing apparatus" includes all devices, equipment, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, an apparatus may include code that creates an operating environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or any combination thereof. A propagated signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver device.

[0207] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language (including compiled or interpreted languages) and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed for execution on one or more computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0208] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0209] For example, processors suitable for running a computer program include general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from read-only memory or random access memory, or both. The essential elements of a computer are a processor that executes instructions and one or more memory devices that store instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or be operatively coupled to one or more mass storage devices to receive data from or transfer data to, or both. However, a computer need not necessarily have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated into, special-purpose logic circuitry.

[0210] Although this patent document contains many details, they should not be interpreted as limitations on any invention or the scope of what is claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various functions described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable subcombination. Furthermore, although features may be described as functioning in certain combinations and even initially claimed as such, in some cases one or more features in the combination may be deleted from the claimed combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0211] Likewise, while operations are depicted in a particular order in the drawings, this should not be understood as requiring that these operations, or all illustrated operations, be performed in the particular order or sequential order shown in order to achieve desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0212] Only a few implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A method for processing video data, comprising: Derives motion information for the current video block; Dividing the current video block into one or more sub-blocks based on a size of the current video block; deriving refined motion information for each of the one or more sub-blocks using a decoder-side motion vector refinement (DMVR) tool based on the motion information derived for the current video block; as well as encoding and decoding the current video block using the refined motion information, The refined motion information derived for each sub-block includes: determining, for each of the sub-blocks, at least one region in a reference picture based on the motion information of the current video block, calculating at least one matching cost based on the at least one region; and determining the refined motion information for each of the sub-blocks based on the matching cost; wherein the sample values of the at least one region are generated by a bilinear filter; and The calculations include: The at least one matching cost is calculated using one of every N rows in each of the at least one region.

2. The method of claim 1, wherein the partitioning comprises: If the height of the current video block is an integer multiple of a predefined height, dividing the current video block into a plurality of sub-blocks according to the height, so that the height of the sub-blocks is equal to the predefined height; If the width of the current video block is an integer multiple of a predefined width, the current video block is divided into a plurality of sub-blocks according to the width, so that the width of the sub-blocks is equal to the predefined width. The method of claim 2 , wherein at least one of the predefined height and the predefined width is 16. The method of claim 1 , wherein a width of the sub-block is less than or equal to 16, and a height of the sub-block is less than or equal to 16.

5. The method of claim 1, wherein a luma component is used to derive the refined motion information using the DMVR tool. The method of claim 1 , wherein the plurality of sub-blocks have the same size. The method of claim 1 , wherein the row is a first row of every N rows. The method of claim 1 , wherein N is equal to 2.

9. The method of claim 1, dividing the current video block into one or more sub-blocks comprises partitioning the current video block into sub-blocks of only square size.

10. The method of claim 1, wherein dividing the current video block into one or more sub-blocks comprises partitioning the current video block into sub-blocks from at least two of the following groups: a first group of sub-blocks that are square in size, a second group of sub-blocks that are vertically rectangular, and a third group of sub-blocks that are horizontally rectangular.

11. The method of any one of claims 9-10, wherein the partitioning depends on a coding type of the current video block.

12. The method of claim 1, wherein the encoding and decoding comprises decoding the current video block from a bitstream.

13. The method of claim 1, wherein the encoding and decoding comprises encoding the current video block into a bitstream.

14. An apparatus for processing video data, comprising a processor and non-transitory memory having instructions therein, wherein the instructions, when executed by the processor, cause the processor to: Derives motion information for the current video block; Dividing the current video block into one or more sub-blocks based on a size of the current video block; deriving refined motion information for each of the one or more sub-blocks using a decoder-side motion vector refinement (DMVR) tool based on the motion information derived for the current video block; as well as encoding and decoding the current video block using the refined motion information, The refined motion information derived for each sub-block includes: determining, for each of the sub-blocks, at least one region in a reference picture based on the motion information of the current video block, calculating at least one matching cost based on the at least one region; and determining the refined motion information for each of the sub-blocks based on the matching cost; wherein the sample values of the at least one region are generated by a bilinear filter; and The calculations include: The at least one matching cost is calculated using one of every N rows in each of the at least one region.

15. The apparatus of claim 14, wherein the encoding and decoding comprises decoding the current video block from a bitstream.

16. The apparatus of claim 14, wherein the encoding and decoding comprises encoding the current video block into a bitstream.

17. A non-transitory computer-readable storage medium storing instructions that cause a processor to: Derives motion information for the current video block; Dividing the current video block into one or more sub-blocks based on a size of the current video block; deriving refined motion information for each of the one or more sub-blocks using a decoder-side motion vector refinement (DMVR) tool based on the motion information derived for the current video block; as well as encoding and decoding the current video block using the refined motion information, The refined motion information derived for each sub-block includes: determining, for each of the sub-blocks, at least one region in a reference picture based on the motion information of the current video block, calculating at least one matching cost based on the at least one region; and determining the refined motion information for each of the sub-blocks based on the matching cost; wherein the sample values of the at least one region are generated by a bilinear filter; and The calculations include: The at least one matching cost is calculated using one of every N rows in each of the at least one region.

18. A method for storing a video bitstream, comprising: Derives motion information for the current video block; Dividing the current video block into one or more sub-blocks based on a size of the current video block; deriving refined motion information for each of the one or more sub-blocks using a decoder-side motion vector refinement (DMVR) tool based on the motion information derived for the current video block; generating the bitstream using the refined motion information; as well as storing the bitstream in a non-transitory computer-readable recording medium, The refined motion information derived for each sub-block includes: determining, for each of the sub-blocks, at least one region in a reference picture based on the motion information of the current video block, calculating at least one matching cost based on the at least one region; and determining the refined motion information for each of the sub-blocks based on the matching cost; wherein the sample values of the at least one region are generated by a bilinear filter; and The calculations include: The at least one matching cost is calculated using one of every N rows in each of the at least one region.

19. A video processing device comprising a processor and a non-transitory memory storing instructions, wherein the instructions, when executed by the processor, cause the processor to implement the method defined in any one of claims 2 to 11.

20. A non-transitory computer-readable storage medium having computer code stored thereon, which, when executed by a processor, causes the processor to implement the method of any one of claims 2 to 13.

Images