Limitations of decoder-side motion vector derivation based on codec information

Abstract

Apparatus, systems and methods for digital video coding and decoding are described, including limitations on decoder-side motion vector derivation based on codec information. An exemplary method for video processing includes: for a conversion between a current block of video and a bitstream representation of the video, making a decision regarding selective enabling of a decoder-side motion vector derivation (DMVD) tool for the current block, the DMVD tool deriving a refinement of motion information signaled in the bitstream representation, and converting using a Merge mode and a motion vector difference indicated by a motion direction and a motion magnitude, and performing the conversion between the current block and the bitstream representation based on the decision.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] In accordance with applicable patent laws, this application is intended to timely claim the priority and benefits of International Patent Application Nos. PCT / CN2018 / 111224 filed on October 22, 2018 and PCT / CN2018 / 123407 filed on December 25, 2018, respectively. The entire disclosure of the above applications is incorporated by reference as part of the disclosure of this application. Technical Field

[0003] This patent document relates to video coding and decoding technology, equipment and systems. Background Art

[0004] Despite advances in video compression technology, digital video still accounts for the largest use of bandwidth on the Internet and other digital communications networks. As the number of networked user devices capable of receiving and displaying video increases, bandwidth requirements for digital video usage are expected to continue to grow. Summary of the invention

[0005] Devices, systems and methods related to digital video coding and decoding are described, in particular, devices, systems and methods related to limitations of decoder-side motion vector derivation (DMVD) based on codec information. The described methods can be applied to existing video coding standards (e.g., High Efficiency Video Coding (HEVC)) and future video coding standards or video codecs.

[0006] In one representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: for a conversion between a current block of a video and a bitstream representation of the video, making a decision regarding the selective enabling of a decoder-side motion vector derivation (DMVD) tool for the current block, wherein the DMVD tool derives a refinement of motion information signaled in the bitstream representation, and wherein the conversion uses a Merge mode and a motion vector difference indicated by a motion direction and a motion magnitude; and performing the conversion between the current block and the bitstream representation based on the decision.

[0007] In another representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: based on decoded motion information associated with a current block of the video, making a decision regarding selective enabling of a decoder-side motion vector derivation (DMVD) tool for the current block, wherein the DMVD tool derives a refinement of motion information signaled in a bitstream representation of the video; and performing conversion between the current block and the bitstream representation based on the decision.

[0008] In yet another representative aspect, the above method is embodied in the form of processor executable code and stored in a computer readable program medium.

[0009] In yet another representative aspect, a device configured or operable to perform the above method is disclosed. The device may include a processor programmed to implement the method.

[0010] In yet another representative aspect, a video decoder device may implement the methods described herein.

[0011] The above and other aspects and features of the disclosed technology are described in more detail in the drawings, the description and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An example of constructing a Merge candidate list is shown.

[0013] Figure 2 Examples of locations of spatial candidates are shown.

[0014] Figure 3 An example of a candidate pair that is subject to redundancy check for spatial merge candidates is shown.

[0015] Figure 4A and Figure 4B An example of the position of the second prediction unit (PU) based on the size and shape of the current block is shown.

[0016] Figure 5 An example of motion vector scaling of a temporal merge candidate is shown.

[0017] Figure 6 An example of candidate positions of time-domain Merge candidates is shown.

[0018] Figure 7 An example of generating a combined bi-prediction Merge candidate is shown.

[0019] Figure 8 An example of constructing motion vector prediction candidates is shown.

[0020] Fig. 9 An example of motion vector scaling of spatial motion vector candidates is shown.

[0021] Fig.10 An example of motion prediction using an Alternative Temporal Motion Vector Prediction (ATMVP) algorithm for a Coding Unit (CU) is shown.

[0022] Fig.11 An example of a codec unit (CU) with sub-blocks and neighboring blocks used by a spatial-temporal motion vector prediction (STMVP) algorithm is shown.

[0023] Fig. 12A and Fig. 12B An example snapshot of a sub-block when an Overlapped Block Motion Compensation (OBMC) algorithm is used is shown.

[0024] Fig.13 An example of neighboring samples used to derive parameters of a Local Illumination Compensation (LIC) algorithm is shown.

[0025] Fig.14 An example of a simplified affine motion model is shown.

[0026] Fig.15 An example of an affine motion vector field (MVF) for each sub-block is shown.

[0027] Fig.16 An example of motion vector prediction (MVP) of the AF_INTER affine motion mode is shown.

[0028] Fig.17A and Fig. 17B Example candidates for the AF_MERGE affine motion mode are shown.

[0029] Fig.18An example of bilateral matching in Pattern Matched Motion Vector Derivation (PMMVD) mode, which is a special Merge mode based on the Frame-Rate Up Conversion (FRUC) algorithm, is shown.

[0030] Fig.19 An example of template matching in the FRUC algorithm is shown.

[0031] Fig. 20 An example of unilateral motion estimation in the FRUC algorithm is shown.

[0032] Fig.21 An example of the Ultimate Motion Vector Expression (UMVE) search process for the current frame is shown.

[0033] Fig.22A and Fig. 22B An example of a UMVE search point is shown.

[0034] Fig.23 An exemplary mapping between distance indices and distance offsets is shown.

[0035] Fig.24 An example of an optical flow trajectory used by the Bi-directional Optical flow (BIO) algorithm is shown.

[0036] Fig.25A and Fig.25B An example snapshot using the Bidirectional Optical Flow (BIO) algorithm without block expansion is shown.

[0037] Fig.26 An example of a Decoder-side Motion Vector Refinement (DMVR) algorithm based on bilateral template matching is shown.

[0038] Figure 27A-27B A flow chart illustrating an example method of video processing is shown.

[0039] Fig.28 is a block diagram of an example of a hardware platform for implementing the visual media decoding or visual media encoding techniques described in this document.

[0040] Fig.29 is a block diagram of an example video processing system in which the disclosed technology may be implemented. DETAILED DESCRIPTION

[0041] Due to the growing demand for high-resolution video, video coding and decoding methods and technologies are ubiquitous in modern technology. Video codecs generally include electronic circuits or software that compress or decompress digital video and are constantly being improved to provide higher codec efficiency. Video codecs convert uncompressed video into a compressed format and vice versa. There is a complex relationship between video quality, the amount of data used to represent the video (determined by the bitrate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, and end-to-end delay (latency). The compression format usually conforms to a standard video compression specification, such as the High Efficiency Video Codec (HEVC) standard (also known as H.265 or MPEG-H Part 2), the to-be-finalized Versatile Video Coding (VVC) standard, or other current and / or future video codec standards.

[0042] Embodiments of the disclosed technology can be applied to existing video codec standards (e.g., HEVC, H.265) and future standards to improve compression performance. Section headings are used in this document to improve the readability of the description and do not in any way limit the discussion or embodiments (and / or implementation methods) to only the individual sections.

[0043] 1. Example of HEVC / H.265 inter-frame prediction

[0044] Video codec standards have improved significantly over the years and now offer, in part, high codec efficiency and support for higher resolutions. Recent standards such as HEVC and H.265 are based on a hybrid video codec structure that utilizes temporal prediction plus transform coding.

[0045] 1.1 Example of prediction mode

[0046] Each inter-frame PU (prediction unit) has motion parameters for one or two reference picture lists. In some embodiments, the motion parameters include motion vectors and reference picture indices. In other embodiments, the use of one of the two reference picture lists can also be notified using inter_pred_idc signaling. In some other embodiments, the motion vector can be explicitly encoded and decoded as a delta relative to the predicted value.

[0047] When a CU is encoded or decoded in skip mode, one PU is associated with the CU and there are no significant residual coefficients, no coded motion vector increments or reference picture indices. A Merge mode is specified, whereby the motion parameters of the current PU are obtained from neighboring PUs, including spatial and temporal candidates. The Merge mode applies not only to skip mode, but also to any inter-prediction PU. An alternative to the Merge mode is the explicit transmission of motion parameters, where the motion vectors, the corresponding reference picture indices for each reference picture list, and the use of the reference picture list are explicitly signaled per PU.

[0048] When signaling indicates that one of the two reference picture lists is to be used, a PU is generated from a block of samples. This is called "unidirectional prediction". Unidirectional prediction is available for P slices and B slices.

[0049] When signaling indicates that both reference picture lists are to be used, a PU is generated from two sample blocks. This is called "bi-prediction". Bi-prediction is only available for B slices.

[0050] 1.1.1 Example of constructing a candidate for Merge mode

[0051] When predicting a PU using Merge mode, an index pointing to an entry in the Merge candidate list is parsed from the bitstream and used to retrieve motion information. The construction of this list can be outlined in the following series of steps:

[0052] Step 1: Initial candidate derivation

[0053] Step 1.1: Spatial Candidate Derivation

[0054] Step 1.2: Redundancy check of spatial candidates

[0055] Step 1.3: Time Domain Candidate Derivation

[0056] Step 2: Additional candidate insertions

[0057] Step 2.1: Create bidirectional prediction candidates

[0058] оStep 2.2: Insert zero motion candidates

[0059] Figure 1An example of constructing a Merge candidate list based on the above series of steps is shown. For spatial Merge candidate derivation, up to four Merge candidates are selected from candidates located at five different positions. For time domain Merge candidate derivation, up to one Merge candidate is selected from two candidates. Since a constant number of candidates for each PU is assumed at the decoder, additional candidates are generated when the number of candidates does not reach the maximum number of Merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, the index of the best Merge candidate is encoded using truncated unary binarization (TU). If the size of the CU is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

[0060] 1.1.2 Constructing Spatial Merge Candidates

[0061] In the derivation of spatial merge candidates, Figure 2 Up to four Merge candidates are selected from the candidates at the positions depicted in . The order of derivation is A1, B1, B0, A0, and B2. Position B2 is considered only when any PU at position A1, B1, B0, A0 is unavailable (for example, because it belongs to another strip or slice) or is intra-coded. After adding the candidate at position A1, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving coding efficiency.

[0062] In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only pairs with Figure 3 The arrows in the table link pairs, and candidates are only added to the list if the corresponding candidates for redundancy checking do not have the same motion information. Another source of repeated motion information is a "second PU" associated with a partition other than 2N×2N. As an example, Figure 4A and Figure 4B The second PU is depicted for the N×2N and 2N×N cases, respectively. When the current PU is partitioned into N×2N, the candidate at position A1 is not considered for list construction. In some embodiments, adding this candidate may result in two prediction units having the same motion information, which is redundant for having only one PU in the codec unit. Similarly, when the current PU is partitioned into 2N×N, position B1 is not considered.

[0063] 1.1.3 Constructing Time Domain Merge Candidates

[0064] In this step, only one candidate is added to the list. Specifically, in the derivation of the temporal Merge candidate, the scaled motion vector is derived based on the co-located PU belonging to the picture with the minimum POC (Picture Order Count) difference with the current picture in a given reference picture list. The reference picture list to be used for the derivation of the co-located PU is explicitly signaled in the slice header.

[0065] Figure 5 An example of the derivation of a scaled motion vector (as dotted line) for a temporal Merge candidate is shown, which is scaled from the motion vector of the collocated PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal Merge candidate is set equal to zero. For B slices, two motion vectors (one for reference picture list 0 and the other for reference picture list 1) are obtained and combined to generate a bi-directional prediction Merge candidate.

[0066] In the collocated PU(Y) belonging to the reference frame, the position of the temporal candidate is selected between candidates C0 and C1, such as Figure 6 If the PU at position C0 is not available, is intra-coded, or is outside the current CTU, position C1 is used. Otherwise, position C0 is used in the derivation of the time-domain Merge candidate.

[0067] 1.1.4 Constructing additional types of merge candidates

[0068] In addition to the space-time Merge candidates, there are two additional types of Merge candidates: combined bidirectional prediction Merge candidates and zero Merge candidates. Combined bidirectional prediction Merge candidates are generated by utilizing space-time Merge candidates. Combined bidirectional prediction Merge candidates are only used for B slices. Combined bidirectional prediction candidates are generated by combining the first reference picture list motion parameters of an initial candidate with the second reference picture list motion parameters of another initial candidate. If the two tuples provide different motion hypotheses, they will form a new bidirectional prediction candidate.

[0069] Figure 7 An example of this process is shown where two candidates in the original list (710, on the left) (having either mvL0 and refIdxL0 or mvL1 and refIdxL1) are used to create a combined bi-predictive Merge candidate that is added to the final list (720, on the right).

[0070] Zero motion candidates are inserted to fill the remaining entries in the Merge candidate list and thus reach the MaxNumMergeCand capacity. These candidates have zero spatial displacement and a reference picture index that starts at zero and increases each time a new zero motion candidate is added to the list. The number of reference frames used by these candidates is 1 and 2 for unidirectional prediction and bidirectional prediction, respectively. In some embodiments, no redundancy check is performed on these candidates.

[0071] 1.1.5 Example of motion estimation regions for parallel processing

[0072] To speed up the encoding process, motion estimation can be performed in parallel, thereby deriving motion vectors for all prediction units within a given region at the same time. Deriving Merge candidates from spatial neighbors may interfere with parallel processing because one prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is completed. In order to ease the trade-off between coding efficiency and processing latency, a Motion Estimation Region (MER) can be defined. The size of the MER is signaled in the Picture Parameter Set (PPS) using the "log2_parallel_merge_level_minus2" syntax element. When a MER is defined, Merge candidates that fall into the same region are marked as unavailable and are therefore not considered in list construction.

[0073] 1.2 Embodiment of Advanced Motion Vector Prediction (AMVP)

[0074] AMVP exploits the spatiotemporal correlation of motion vectors with neighboring PUs, which is used for explicit transmission of motion parameters. A motion vector candidate list is constructed by first checking the availability of neighboring PU positions in the left and top temporal domains, removing redundant candidates, and adding zero vectors to make the candidate list a constant length. The encoder can then select the best prediction value from the candidate list and send a corresponding index indicating the selected candidate. Similar to Merge index signaling, the index of the best motion vector candidate is encoded using truncated unary. In this case, the maximum value to be encoded is 2 (see Figure 8 ). In the following sections, details about the derivation process of motion vector prediction candidates will be provided.

[0075] 1.2.1 Example of constructing motion vector prediction candidates

[0076] Figure 8 The derivation process of motion vector prediction candidates is outlined and can be implemented for each reference picture list with refidx as input.

[0077] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For spatial motion vector candidate derivation, the final Figure 2 Two motion vector candidates are derived from the motion vectors of each PU at the five different positions shown.

[0078] For temporal motion vector candidate derivation, one motion vector candidate is selected from two candidates derived based on two different collocated positions. After generating the first spatiotemporal candidate list, duplicate motion vector candidates in the list are removed. If the number of potential candidates is greater than two, motion vector candidates whose reference picture index within the associated reference picture list is greater than 1 are removed from the list. If the number of spatiotemporal motion vector candidates is less than two, an additional zero motion vector candidate is added to the list.

[0079] 1.2.2 Constructing Spatial Motion Vector Candidates

[0080] In the derivation of spatial motion vector candidates, the Figure 2 Consider up to two candidates out of the five potential candidates derived in the PU at the position shown, and those positions are the same as the position of the motion Merge. The derivation order on the left side of the current PU is defined as A0, A1, and scaling A0, scaling A1. The derivation order on the upper side of the current PU is defined as B0, B1, B2, scaling B0, scaling B1, scaling B2. Therefore, for each side, there are four cases that can be used as motion vector candidates, two of which do not require spatial scaling, and two use spatial scaling. The four different cases are summarized as follows:

[0081] No airspace scaling

[0082] (1) Same reference picture list and same reference picture index (same POC)

[0083] (2) Different reference picture lists, but the same reference picture (same POC)

[0084] Airspace scaling

[0085] (3) Same reference picture list, but different reference pictures (different POC)

[0086] (4) Different reference picture lists and different reference pictures (different POCs)

[0087] First check the non-spatial scaling case, then the case where spatial scaling is allowed. Spatial scaling is considered when the POC is different between the reference pictures of the neighboring PU and the reference pictures of the current PU regardless of the reference picture list. If all PUs of the left candidate are unavailable or are intra-coded, then scaling for the upper motion vector is allowed to help parallel derivation of the left and upper MV candidates. Otherwise, spatial scaling is not allowed for the upper motion vector.

[0088] like Fig. 9 As shown in the example in , for the case of spatial scaling, the motion vectors of neighboring PUs are scaled in a similar way to temporal scaling. One difference is that the reference picture list and the index of the current PU are given as input; the actual scaling process is the same as that of temporal scaling.

[0089] 1.2.3 Constructing temporal motion vector candidates

[0090] Except for the reference picture index derivation, all the processes used to derive the temporal Merge candidate are the same as those used to derive the spatial motion vector candidate (e.g. Figure 6 In some embodiments, the reference picture index is signaled to the decoder.

[0091] 2. Example of Inter Prediction Method in Joint Exploration Model (JEM)

[0092] In some embodiments, a reference software called Joint Exploration Model (JEM) is used to explore future video codec technologies. In JEM, sub-block based prediction is adopted in several codec tools, such as affine prediction, optional temporal motion vector prediction (ATMVP), spatiotemporal motion vector prediction (STMVP), bidirectional optical flow (BIO), frame rate up conversion (FRUC), locally adaptive motion vector resolution (LAMVR), overlapped block motion compensation (OBMC), local illumination compensation (LIC), and decoder side motion vector refinement (DMVR).

[0093] 2.1 Example of motion vector prediction based on sub-CU

[0094] In JEM with quadtrees plus binary trees (QTBT), each CU can have at most one motion parameter set for each prediction direction. In some embodiments, two sub-CU level motion vector prediction methods are considered in the encoder by dividing the large CU into sub-CUs and deriving the motion information of all sub-CUs of the large CU. The optional time domain motion vector prediction (ATMVP) method allows each CU to extract multiple sets of motion information from multiple blocks smaller than the current CU in the collocated reference picture. In the Spatial-Temporal Motion Vector Prediction (STMVP) method, the motion vector of the sub-CU is recursively derived by using the temporal motion vector predictor and the spatial domain neighboring motion vector. In some embodiments, and in order to maintain a more accurate motion field for sub-CU motion prediction, motion compression of the reference frame can be disabled.

[0095] 2.1.1 Example of Alternative Temporal Motion Vector Prediction (ATMVP)

[0096] In the ATMVP method, the temporal motion vector prediction (TMVP) method is modified by extracting multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU.

[0097] Fig.10 An example of the ATMVP motion prediction process for CU 1000 is shown. The ATMVP method predicts the motion vector of sub-CU 1001 within CU 1000 in two steps. The first step is to identify the corresponding block 1051 in the reference picture 1050 with the time domain vector. The reference picture 1050 is also called the motion source picture. The second step is to divide the current CU 1000 into sub-CUs 1001 and obtain the motion vector and the reference index of each sub-CU from the block corresponding to each sub-CU.

[0098] In the first step, the reference picture 1050 and the corresponding block are determined by the motion information of the spatial neighboring blocks of the current CU 1000. In order to avoid the repeated scanning process of the neighboring blocks, the first Merge candidate in the Merge candidate list of the current CU 1000 is used. The first available motion vector and its associated reference index are set to the index of the temporal vector and the motion source picture. In this way, compared with TMVP, the corresponding block can be identified more accurately, where the corresponding block (sometimes called a collocated block) is always in the lower right or center position relative to the current CU.

[0099] In the second step, the corresponding block of the sub-CU 1051 is identified by the time domain vector in the motion source picture 1050 by adding the time domain vector to the coordinates of the current CU. For each sub-CU, the motion information of its corresponding block (e.g., the minimum motion grid covering the center sample point) is used to derive the motion information of the sub-CU. After identifying the motion information of the corresponding N×N block, it is converted into the motion vector and reference index of the current sub-CU in the same way as the TMVP of HEVC, where motion scaling and other processes apply. For example, the decoder checks whether the low latency condition is met (e.g., the POC of all reference pictures of the current picture is less than the POC of the current picture), and may use the motion vector MV x (e.g., the motion vector corresponding to the reference picture list X) is used to predict the motion vector MV of each sub-CU y (eg, where X equals 0 or 1, and Y equals 1-X).

[0100] 2.1.2 Example of Spatio-Temporal Motion Vector Prediction (STMVP)

[0101] In the STMVP method, the motion vector of a sub-CU is recursively derived in raster scan order. Fig.11 A CU with 4 sub-blocks and neighboring blocks are shown. Consider an 8×8 CU 1100, which includes 4 4×4 sub-CUs: A (1101), B (1102), C (1103), and D (1104). The neighboring 4×4 blocks in the current frame are labeled a (1111), b (1112), c (1113), and d (1114).

[0102] The motion derivation of sub-CUA starts by identifying its two spatial neighbors. The first neighbor is the N×N block (block c 1113) above sub-CUA1101. If this block c 1113 is not available or is intra-coded, check the other N×N blocks above sub-CUA 1101 (from left to right, starting from block c 1113). The second neighbor is the block on the left of sub-CUA 1101 (block b 1112). If block b1112 is not available or is intra-coded, check the other blocks on the left of sub-CUA 1101 (from top to bottom, starting from block b 1112). The motion information obtained from the adjacent blocks in each list is scaled to the first reference frame of the given list. Next, the temporal motion vector predictor (TMVP) of sub-block A 1101 is derived by following the same process of TMVP derivation as specified by HEVC. The motion information of the collocated block at block D 1104 is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors are averaged for each reference list individually. The average motion vector is assigned as the motion vector of the current sub-CU.

[0103] 2.1.3 Example of Sub-CU Motion Prediction Mode Signaling

[0104] In some embodiments, sub-CU modes are enabled as additional Merge candidates, and no additional syntax elements are required to signal these modes. Two additional Merge candidates are added to the Merge candidate list of each CU to represent the ATMVP mode and the STMVP mode. In other embodiments, if the sequence parameter set indicates that ATMVP and STMVP are enabled, up to seven Merge candidates can be used. The encoding logic of the additional Merge candidates is the same as that of the Merge candidates in the HM, which means that for each CU in a P slice or a B slice, two RD checks may be required for the two additional Merge candidates. In some embodiments, for example, JEM, all binary values ​​of the Merge index are context-coded by CABAC (Context-based Adaptive Binary Arithmetic Coding). In other embodiments, for example, HEVC, only the first binary value is context-coded, and the remaining binary values ​​are context bypass coded.

[0105] 2.2 Example of Adaptive Motion Vector Difference Resolution

[0106] In some embodiments, when use_integer_mv_flag in the slice header is equal to 0, the motion vector difference (MVD) (between the motion vector of the PU and the predicted motion vector) is signaled in units of quarter luma samples. In JEM, local adaptive motion vector resolution (LAMVR) is introduced. In JEM, MVD can be encoded and decoded in units of quarter luma samples, integer luma samples, or four luma samples. The MVD resolution is controlled at the codec unit (CU) level, and the MVD resolution flag is conditionally signaled for each CU with at least one non-zero MVD component.

[0107] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter luma sample MV precision is not used, another flag is signaled to indicate whether integer luma sample MV precision or quad luma sample MV precision is used.

[0108] When the first MVD resolution flag of a CU is zero or is not coded for the CU (meaning that all MVDs in the CU are zero), a quarter luma sample MV resolution is used for the CU. When the CU uses integer luma sample MV precision or four luma sample MV precision, the MVP in the AMVP candidate list of the CU is rounded to the corresponding precision.

[0109] In the encoder, the CU-level RD check is used to determine which MVD resolution to use for the CU. That is, for each MVD resolution, three CU-level RD checks are performed. In order to speed up the encoder, the following encoding scheme is applied in JEM:

[0110] -- During RD check of a CU with normal quarter luma sample MVD resolution, store the motion information of the current CU (integer luma sample accuracy). The stored motion information (after rounding) is used as the starting point for further small-scale motion vector refinement during RD check for the same CU with integer luma sample and 4 luma sample MVD resolution, so that the time-consuming motion estimation process is not repeated three times.

[0111] --Conditionally call RD check for CU with 4 luma sample MVD resolution. For a CU, when the RD cost of integer luma sample MVD resolution is much greater than the RD cost of quarter luma sample MVD resolution, skip the RD check for CU's 4 luma sample MVD resolution.

[0112] 2.3 Example of Higher Motion Vector Storage Accuracy

[0113] In HEVC, the motion vector precision is quarter pixel (quarter luma sample and eighth chroma sample for 4:2:0 video). In JEM, the precision of internal motion vector storage and merge candidates is increased to 1 / 16 pixel. Higher motion vector precision (1 / 16 pixel) is used for motion compensated inter prediction of CUs encoded and decoded in skip / merge mode. For CUs encoded and decoded in normal AMVP mode, integer pixel or quarter pixel motion is used.

[0114] The SHVC upsampling interpolation filter with the same filter length and normalization factor as the HEVC motion compensation interpolation filter is used as the motion compensation interpolation filter for the additional fractional pixel position. The chroma component motion vector precision is 1 / 32 samples in JEM, and the additional interpolation filter for the 1 / 32 pixel fractional position is derived by using the average of two adjacent filters at 1 / 16 pixel fractional positions.

[0115] 2.4 Example of Overlapped Block Motion Compensation (OBMC)

[0116] In JEM, Overlapped Block Motion Compensation (OBMC) was previously used in H.263. In JEM, unlike H.263, OBMC can be turned on and off using CU-level syntax. When OBMC is used in JEM, OBMC is performed for all motion compensation (MC) block boundaries except the right and bottom boundaries of the CU. In addition, it is applied to both luminance and chrominance components. In JEM, an MC block corresponds to a codec block. When a CU is encoded and decoded with sub-CU modes, including Sub-CU Merge, Affine, and FRUC (Frame Rate Up Conversion) modes, each sub-block of the CU is an MC block. In order to handle CU boundaries in a unified manner, OBMC is performed for all MC block boundaries at the sub-block level, where the sub-block size is set equal to 4×4, as Figure 12A-12B shown.

[0117] Fig. 12A Sub-blocks at CU / PU boundaries are shown, and shaded sub-blocks are where OBMC is applied. Similarly, Fig. 12B A sub-PU in ATMVP mode is shown.

[0118] When OBMC is applied to the current sub-block, in addition to the current motion vector, the motion vectors of four connected neighboring sub-blocks (if available and different from the current motion vector) are also used to derive the prediction block of the current sub-block. These multiple prediction blocks based on multiple motion vectors are combined to generate the final prediction signal of the current sub-block.

[0119] The prediction block based on the motion vector of the neighboring subblock is denoted as PN, where N represents the index for the neighboring upper, lower, left and right subblocks, and the prediction block based on the motion vector of the current subblock is denoted as PC. When PN is based on motion information containing the same motion information of the neighboring subblock as the current subblock, OBMC is not performed from PN. Otherwise, each sample of PN is added to the same sample in PC, that is, four rows / columns of PN are added to PC. Weight factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} are used for PN, and weight factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are used for PC. The exception is a small MC block (i.e. when the height or width of the codec block is equal to 4 or the CU is coded and decoded in sub-CU mode), for which only two rows / columns of PN are added to PC. In this case, weight factors {1 / 4, 1 / 8} are used for PN, and weight factors {3 / 4, 7 / 8} are used for PC. For a PN generated based on the motion vectors of vertically (horizontally) adjacent sub-blocks, samples in the same row (column) of the PN are added to the PC with the same weight factor.

[0120] In JEM, for CUs with a size less than or equal to 256 luma samples, a CU level flag is signaled to indicate whether OBMC is applied for the current CU. For CUs with a size greater than 256 luma samples or not encoded in AMVP mode, OBMC is applied by default. At the encoder, when OBMC is applied to a CU, its impact is taken into account during the motion estimation stage. The prediction signal formed by OBMC using the motion information of the upper and left neighboring blocks is used to compensate the upper and left boundaries of the original signal of the current CU, and then the normal motion estimation process is applied.

[0121] 2.5 Example of Local Illumination Compensation (LIC)

[0122] LIC is based on a linear model of illumination variation, using a scaling factor a and an offset b, and is adaptively enabled or disabled for each codec unit (CU) in inter-mode coding.

[0123] When LIC is applied to a CU, a least square error method is used to derive parameters a and b by using neighboring samples of the current CU and their corresponding reference samples. Fig.13 An example of adjacent sample points used to derive IC algorithm parameters is shown. Specifically, Fig.13 As shown, the subsampling (2:1 subsampling) neighboring samples and corresponding samples (identified by the motion information of the current CU or sub-CU) in the reference picture are used. The IC parameters are derived and applied to each prediction direction separately.

[0124] When the CU is encoded and decoded in Merge mode, the LIC flag is copied from the adjacent block in a manner similar to the motion information copying in Merge mode; otherwise, the LIC flag is signaled for the CU to indicate whether LIC is applicable.

[0125] When LIC is enabled for a picture, an additional CU-level RD check is required to determine whether LIC is applied to the CU. When LIC is enabled for a CU, the Mean-Removed Sum of Absolute Difference (MR-SAD) and the Mean-Removed Sum of AbsoluteHadamard-Transformed Difference (MR-SATD) (instead of SAD and SATD) are used for integer pixel motion search and fractional pixel motion search, respectively.

[0126] In order to reduce the coding complexity, the following coding scheme is applied in JEM:

[0127] --When there is no significant illumination change between the current picture and its reference pictures, LIC is disabled for the entire picture. To identify this situation, the histogram of the current picture and each reference picture of the current picture is calculated at the encoder. If the histogram difference between the current picture and each reference picture of the current picture is less than a given threshold, LIC is disabled for the current picture; otherwise, LIC is enabled for the current picture.

[0128] 2.6 Example of Affine Motion Compensated Prediction

[0129] In HEVC, only the translational motion model is applied to motion compensation prediction (MCP). However, cameras and objects can have many kinds of motions, such as zooming in / out, rotation, perspective motion, and / or other irregular motions. On the other hand, JEM applies a simplified affine transformation motion compensation prediction. Fig.14 An example of an affine motion field of a block 1400 described by two control point motion vectors V0 and V1 is shown. The motion vector field (MVF) of the block 1400 is described by the following equation:

[0130]

[0131] like Fig.14 As shown, (v 0x ,v 0y ) is the motion vector of the upper left control point, and (v 1x ,v 1y ) is the motion vector of the upper right control point. In order to simplify motion compensation prediction, sub-block based affine transformation prediction can be applied. The sub-block size M×N is derived as follows:

[0132]

[0133] Here, MvPre is the fractional precision of the motion vector (e.g., 1 / 16 in JEM), (v 2x ,v 2y ) is the motion vector of the lower left control point calculated according to equation (1). If necessary, M and N can be adjusted downward to be divisors of w and h, respectively.

[0134] Fig.15An example of an affine MVF for each sub-block of block 1500 is shown. To derive the motion vector for each M×N sub-block, the motion vector of the center sample of each sub-block can be calculated according to equation (1) and rounded to a fractional precision of the motion vector (e.g., 1 / 16 in JEM). Then, a motion compensated interpolation filter can be applied to generate a prediction for each sub-block using the derived motion vector. After MCP, the high-precision motion vector of each sub-block is rounded and saved to the same precision as the normal motion vector.

[0135] 2.6.1 AF_INTER Mode Implementation

[0136] In JEM, there are two affine motion modes: AF_INTER mode and AF_MERGE mode. For CUs with width and height greater than 8, AF_INTER mode can be applied. The affine flag at the CU level is signaled in the bitstream to indicate whether AF_INTER mode is used. In AF_INTER mode, a pair of motion vectors {(v0, v1)|v0={v1|v2={v2|v3|v4|v5|v6|v7|v8|v9|v1|v1|v1|v1|v2 ... A , v B ,v c},v1={v D ,v E}}'s candidate list.

[0137] Fig.16 An example of motion vector prediction (MVP) of block 1600 in AF_INTER mode is shown. Fig.16As shown, v0 is selected from the motion vectors of sub-blocks A, B or C. The motion vector from the adjacent block is scaled according to the reference list. The motion vector can also be scaled according to the relationship between the picture order count (POC) of the reference of the adjacent block, the POC of the reference of the current CU and the POC of the current CU. The method of selecting v1 from adjacent sub-blocks D and E is similar. If the number of candidate lists is less than 2, the list is filled with motion vector pairs composed by copying each AMVP candidate. When the candidate list is greater than 2, the candidates can be first sorted according to the adjacent motion vectors (for example, based on the similarity of the two motion vectors in a pair of candidates). In some embodiments, only the first two candidates are retained. In some embodiments, the rate distortion (RD) cost check is used to determine which motion vector pair candidate is selected as the control point motion vector prediction (CPMVP) of the current CU. The index indicating the CPMVP position in the candidate list can be signaled in the bitstream. After determining the CPMVP of the current affine CU, affine motion estimation is applied and the control point motion vector (CPMV) is found. The difference between CPMV and CPMVP is then signaled in the bitstream.

[0138] 2.6.3 AF_MERGE Mode Example

[0139] When a CU is applied in AF_MERGE mode, it gets the first block coded in affine mode from the valid neighboring reconstructed blocks. Fig.17A An example of the selection order of candidate blocks of the current CU 1700 is shown. Fig.17A As shown, the selection order may be from the left side ( 1701 ), the top side ( 1702 ), the upper right side ( 1703 ), the lower left side ( 1704 ) to the upper left side ( 1705 ) of the current CU 1700 . Fig. 17B FIG. 1 shows another example of candidate blocks of the current CU 1700 in AF_MERGE mode. If the adjacent lower left block 1801 is encoded and decoded in affine mode, such as Fig. 17B As shown, motion vectors v2, v3 and v4 of the upper left corner, upper right corner and lower left corner of the CU containing sub-block 1701 are derived. Motion vector v0 of the upper left corner of the current CU 1700 is calculated based on v2, v3 and v4. Motion vector v1 of the upper right corner of the current CU can be calculated accordingly.

[0140] After calculating the CPMV v0 and v1 of the current CU according to the affine motion model in equation (1), the MVF of the current CU can be generated. In order to identify whether the current CU is encoded or decoded in AF_MERGE mode, when at least one neighboring block is encoded or decoded in affine mode, an affine flag can be signaled in the bitstream.

[0141] 2.7 Example of Pattern Matched Motion Vector Derivation (PMMVD)

[0142] PMMVD mode is a special Merge mode based on the Frame Rate Up Conversion (FRUC) method. With this mode, the motion information of the block is not signaled but derived at the decoder side.

[0143] When the Merge flag is true, the FRUC flag may be signaled for the CU. When the FRUC flag is false, the Merge index may be signaled and the regular Merge mode may be used. When the FRUC flag is true, an additional FRUC mode flag may be signaled to indicate which method (e.g., bilateral matching or template matching) will be used to derive motion information for the block.

[0144] On the encoder side, the decision on whether to use the FRUC Merge mode for a CU is based on the RD cost selection made for normal Merge candidates. For example, multiple matching modes (e.g., bilateral matching and template matching) of a CU are checked by using the RD cost selection. The matching mode that results in the minimum cost is further compared with other CU modes. If the FRUC matching mode is the most effective mode, the FRUC flag is set to true for the CU and the relevant matching mode is used.

[0145] Generally, the motion derivation process in FRUC Merge mode has two steps: first, a motion search at the CU level is performed, and then motion refinement at the sub-CU level is performed. At the CU level, the initial motion vector of the entire CU is derived based on bilateral matching or template matching. First, a list of MV candidates is generated, and the candidate that results in the minimum matching cost is selected as the starting point for further CU refinement. Then a local search based on bilateral matching or template matching around the starting point is performed. The MV that produces the minimum matching cost is regarded as the MV of the entire CU. Subsequently, the motion information is further refined at the sub-CU level, starting from the derived CU motion vector.

[0146] For example, the following derivation process is performed for W×H CU motion information derivation. In the first stage, the MV of the entire W×H CU is derived. In the second stage, the CU is further divided into M×M sub-CUs. The value of M is calculated as in equation (3), and D is the predefined partition depth, which is set to 3 by default in JEM. Then the MV of each sub-CU is derived.

[0147]

[0148] Fig.18 An example of bilateral matching used in a frame rate up conversion (FRUC) method is shown. Bilateral matching is used to derive motion information of the current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures (1810, 1811). Under the assumption of a continuous motion trajectory, the motion vectors MV0 (1801) and MV1 (1802) pointing to the two reference blocks are proportional to the temporal distance (e.g., TD0 (1803) and TD1 (1804)) between the current picture and the two reference pictures. In some embodiments, when the current picture 1800 is temporally located between the two reference pictures (1810, 1811) and the temporal distance from the current picture to the two reference pictures is the same, the bilateral matching becomes a mirror-based bilateral MV.

[0149] Fig.19 An example of template matching used in a frame rate up conversion (FRUC) method is shown. Template matching can be used to derive motion information of the current CU 1900 by finding the closest match between a template in the current picture (e.g., an adjacent block at the top and / or left of the current CU) and a block in the reference picture 1910 (e.g., the same size as the template). In addition to the above-mentioned FRUC Merge mode, template matching can also be applied to the AMVP mode. In JEM and HEVC, AMVP has two candidates. Using the template matching method, a new candidate can be derived. If the candidate newly derived by template matching is different from the first existing AMVP candidate, it is inserted at the very beginning of the AMVP candidate list, and then the list size is set to 2 (e.g., by removing the second existing AMVP candidate). When applied to the AMVP mode, only CU level search is applied.

[0150] The MV candidate set at the CU level may include the following: (1) if the current CU is in AMVP mode, the original AMVP candidate, (2) all Merge candidates, (3) several MVs in the interpolated MV field (described later), and the top and left neighboring motion vectors.

[0151] When bilateral matching is used, each valid MV of the Merge candidate can be used as input to generate an MV pair assuming bilateral matching. For example, at reference list A, a valid MV of the Merge candidate is (MVa, ref a ). Then, find the reference picture ref of its paired bilateral MV in another reference list B b , so that ref a and ref bLocated on different sides of the current image in the time domain. If this ref b is not available in reference list B, then ref b Determined to be different from ref a The reference of , and its temporal distance to the current picture is the smallest in list B. In determining ref a After that, based on the current image and ref a ,ref b MVa is scaled by the temporal distance between them to derive MVb.

[0152] In some embodiments, four MVs from the interpolated MV field may also be added to the CU level candidate list. More specifically, the interpolated MVs at positions (0,0), (W / 2,0), (0,H / 2), and (W / 2,H / 2) of the current CU are added. When FRUC is applied to AMVP mode, the original AMVP candidate is also added to the CU level MV candidate set. In some embodiments, at the CU level, 15 MVs of the AMVP CU and 13 MVs of the Merge CU may be added to the candidate list.

[0153] The MV candidate set at the sub-CU level includes: the MV determined from the CU level search, (2) the neighboring MVs at the top, left, upper left, and upper right, (3) scaled versions of the collocated MVs from the reference picture, (4) one or more ATMVP candidates (e.g., up to four), and (5) one or more STMVP candidates (e.g., up to four). The scaled MVs from the reference picture are derived as follows. The reference pictures in the two lists are traversed. The MV at the collocated position of the sub-CU in the reference picture is scaled to the reference of the starting CU level MV. The ATMVP and STMVP candidates can be the first four. At the sub-CU level, one or more MVs (e.g., up to 17) are added to the candidate list.

[0154] Generation of interpolated MV fields. Before encoding and decoding the frame, an interpolated motion field is generated for the entire picture based on single-sided ME. The motion field can then be used as a CU-level or sub-CU-level MV candidate later.

[0155] In some embodiments, the motion field of each reference picture in both reference lists is traversed at 4x4 block level. Fig. 20An example of unilateral motion estimation (ME) 2000 in the FRUC method is shown. For each 4×4 block, if the motion associated with the block passes through a 4×4 block in the current picture and the block is not assigned any interpolated motion, the motion of the reference block is scaled to the current picture according to the temporal distances TD0 and TD1 (in the same way as the MV scaling of TMVP in HEVC), and the scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to the 4×4 block, the motion of the block is marked as unavailable in the interpolated motion field.

[0156] Interpolation and matching costs. When the motion vector points to a fractional sample position, motion compensation interpolation is required. To reduce complexity, bilinear interpolation can be used instead of regular 8-tap HEVC interpolation for both bilateral matching and template matching.

[0157] The calculation of the matching cost is different in different steps. When selecting candidates from the candidate set at the CU level, the matching cost can be the absolute sum difference (SAD) of bilateral matching or template matching. After determining the starting MV, the matching cost C of the bilateral matching of the sub-CU level search is calculated as follows:

[0158]

[0159] Here, w is a weight factor. In some embodiments, w can be set to 4 based on experience. MV and MV s Indicates the current MV and the starting MV respectively. SAD can still be used as the matching cost of template matching for sub-CU level search.

[0160] In FRUC mode, MV is derived by using luma samples only. The derived motion will be used for both luma and chroma for MC inter prediction. After determining the MV, the final MV is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma.

[0161] MV refinement is a pattern-based MV search based on bilateral matching cost or template matching cost. In JEM, two search modes are supported - Unrestricted Center-Biased Diamond Search (UCBDS) and Adaptive Cross Search, which perform MV refinement at CU level and sub-CU level respectively. For both CU and sub-CU level MV refinement, MV is directly searched with quarter luma sample MV precision, followed by one-eighth luma sample MV refinement. The search range for MV refinement for CU and sub-CU steps is set equal to 8 luma samples.

[0162] In bilateral matching Merge mode, bidirectional prediction is applied because the motion information of the CU is derived based on the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. In template matching Merge mode, the encoder can choose from unidirectional prediction from list 0, unidirectional prediction from list 1, or bidirectional prediction for the CU. The selection can be based on the template matching cost as follows:

[0163] If costBi <= factor * min(cost0, cost1)

[0164] Bidirectional prediction is used;

[0165] Otherwise, if cost0 <= cost1

[0166] Then use the unidirectional prediction from list 0;

[0167] otherwise,

[0168] Then use the one-way prediction from List 1;

[0169] Here, cost0 is the SAD of list 0 template matching, cost1 is the SAD of list 1 template matching, and costBi is the SAD of bi-prediction template matching. For example, when the value of the factor is equal to 1.25, it means that the selection process is biased towards bi-prediction. Inter prediction direction selection can be applied to the CU level template matching process.

[0170] 2.8 Example of Generalized Bidirectional Prediction (GBI)

[0171] In traditional bidirectional prediction, the prediction values ​​from L0 and L1 are averaged to generate the final prediction value using equal weight 0.5. The prediction value generation formula is shown in equation (5).

[0172] P TraditionalBiPred =(P L0 +P L1 +RoundingOffset)>>shiftNum Equation (5)

[0173] In equation (5), P TraditionalBiPred is the final prediction value of the traditional two-way prediction, P L0 and P L1 are the predicted values ​​of L0 and L1 respectively, and RoundingOffset and shiftNum are used to normalize the final predicted value.

[0174] Generalized Bidirectional Prediction (GBI) proposes to allow different weights to be applied to the prediction values ​​from L0 and L1. The prediction value generation is shown in equation (6).

[0175] PGBi =((1-w1)*P L0 +w1*P L1 +RoundingOffset GBi )>>shiftNum GBi Equation (6)

[0176] In equation (6), P GBi is the final prediction value of GBi. (1-w1) and w1 are the selected GBI weights applied to the prediction values ​​of L0 and L1 respectively. RoundingOffset GBi and shiftNum GBi Used to normalize the final predicted values ​​in GBi.

[0177] The supported weights w1 are {-1 / 4, 3 / 8, 1 / 2, 5 / 8, 5 / 4}. One equal-weight set and four unequal-weight sets are supported. For the equal-weight case, the process of generating the final prediction value is exactly the same as in the traditional bi-prediction mode. For the true bi-prediction case under random access (RA) conditions, the number of candidate weight sets is reduced to three.

[0178] For Advanced Motion Vector Prediction (AMVP) mode, if the CU is encoded or decoded with bidirectional prediction, the weight selection in GBI is explicitly signaled at the CU level. For Merge mode, the weight selection is inherited from the Merge candidate. In this scheme, GBI supports the weighted average of the DMVR generated template and the final prediction value of BMS-1.0.

[0179] 2.9 Example of Multi-Hypothesis Inter-frame Prediction

[0180] In the multi-hypothesis inter-frame prediction mode, in addition to the traditional unidirectional / bidirectional prediction signal, one or more additional prediction signals are signaled. The resulting overall prediction signal is obtained by weighted superposition of samples. uni / bi and the first additional inter-frame prediction signal / hypothesis h3, the resulting prediction signal is as follows:

[0181] p3=(1-α)p uni / bi +αh3

[0182] The changes in the prediction unit syntax are as follows:

[0183]

[0184] The weight factor α is specified by the syntax element add_hyp_weight_idx according to the following mapping:

[0185] add_hyp_weight_idx α 0 1 / 4 1 -1 / 8

[0186] In some embodiments, for additional prediction signals, the concept of prediction list 0 / list 1 is abolished and a combined list is used instead. The combined list is generated by alternately inserting reference frames from list 0 and list 1 with increasing reference indices, omitting reference frames that have already been inserted, thereby avoiding double entries.

[0187] In some embodiments, and similar to the above, more than one additional prediction signal may be used. The resulting overall prediction signal is iteratively accumulated with each additional prediction signal.

[0188] p n+1 =(1-α n+1 ) n +α n+1 h n+1

[0189] The resulting overall prediction signal is obtained as the final p n (i.e. p n has the largest index n).

[0190] Note that also for inter-frame prediction blocks using Merge mode (rather than skip mode), additional inter-frame prediction signals can be specified. Also note that in the case of Merge, not only uni / bi-directional prediction parameters, but also additional prediction parameters of the selected Merge candidate can be used for the current block.

[0191] 2.10 Example of multi-hypothesis prediction for one-way prediction in AMVP mode

[0192] In some embodiments, when multi-hypothesis prediction is applied to unidirectional prediction of improved AMVP mode, a flag is signaled to enable or disable multi-hypothesis prediction for inter_dir equal to 1 or 2, where 1, 2, and 3 represent list 0, list 1, and bidirectional prediction, respectively. In addition, when the flag is true, a Merge index is signaled. In this way, multi-hypothesis prediction turns unidirectional prediction into bidirectional prediction, where one motion is obtained using the original syntax elements in AMVP mode and the other motion is obtained using the Merge scheme. The final prediction combines the two predictions into bidirectional prediction using a 1:1 weight. The Merge candidate list is first derived from the Merge mode, excluding sub-CU candidates (e.g., affine, optional temporal motion vector prediction (ATMVP)). Next, it is separated into two separate lists, one for list 0 (L0) containing all L0 motions from the candidates, and the other for list 1 (L1) containing all L1 motions. After removing redundancy and filling gaps, two Merge lists are generated for L0 and L1, respectively. When applying multi-hypothesis prediction to improve AMVP mode, there are two constraints. First, the multi-hypothesis prediction is enabled for those CUs whose luma codec block (CB) area is greater than or equal to 64. Second, the multi-hypothesis prediction is only applicable to L1 when in a low-latency B picture.

[0193] 2.11 Example of multi-hypothesis prediction in Skip / Merge mode

[0194] In some embodiments, when multi-hypothesis prediction is applied to skip or merge mode, whether multi-hypothesis prediction is enabled is explicitly signaled. In addition to the original prediction, an additional Merge index prediction is selected. Therefore, each candidate of multi-hypothesis prediction implies a Merge candidate pair, one of which is used for the first Merge index prediction and the other is used for the second Merge index prediction. However, in each pair, the Merge candidate of the second Merge index prediction is implicitly derived as a subsequent Merge candidate (i.e., the Merge index that has been signaled plus 1) without signaling any additional Merge index. After removing redundancy by excluding those pairs containing similar Merge candidates and filling in vacancies, a candidate list for multi-hypothesis prediction is formed. Then, the motion from the pair of two Merge candidates is obtained to generate a final prediction, where a 5:3 weight is applied to the first Merge index prediction and the second Merge index prediction, respectively. In addition, a Merge or skip CU with multi-hypothesis prediction enabled can save motion information of additional hypotheses in addition to the motion information of existing hypotheses for reference by subsequent adjacent CUs.

[0195] Note that sub-CU candidates (e.g., affine, ATMVP) are excluded from the candidate list, and multi-hypothesis prediction is not applicable to skip mode for low-latency B pictures. In addition, when multi-hypothesis prediction is applied to Merge or Skip mode, for those CUs whose CU width or CU height is less than 16, or those CUs whose CU width and CU height are both equal to 16, bilinear interpolation filters are used for multi-hypothesis motion compensation. Therefore, the worst-case bandwidth (access samples required for each sample point) of each Merge or Skip CU with multi-hypothesis prediction enabled is less than half of the worst-case bandwidth of each 4×4 CU with multi-hypothesis prediction disabled.

[0196] 2.12 Example of Final Motion Vector Expression (UMVE)

[0197] In some embodiments, a final motion vector expression (UMVE) is proposed, which is also called Merge with MVD (MMVD). The UMVE mode is used for Skip or Merge mode using the proposed motion vector expression method.

[0198] UMVE reuses the same Merge candidates as in VVC. Among the Merge candidates, candidates can be selected and further extended by the proposed motion vector representation method.

[0199] UMVE provides a new motion vector expression with simplified signaling. The expression method includes the starting point, motion amplitude and motion direction. This proposed technique uses the Merge candidate list as is. However, for the extension of UMVE, only candidates of the default Merge type (MRG_TYPE_DEFAULT_N) are considered.

[0200] The base candidate index (IDX) defines the starting point. The base candidate index indicates the best candidate among the candidates in the list, as shown below.

[0201] Basic Candidate IDX 0 1 2 3 The Nth MVP First MVP Second MVP Third MVP Fourth MVP

[0202] If the number of basic candidates is equal to 1, the basic candidate IDX is not signaled.

[0203] The distance index is the motion amplitude information. The distance index indicates the predefined distance from the start point information. The predefined distances are as follows:

[0204] Distance from IDX 0 1 2 3 4 5 6 7 Pixel distance 1 / 4 pixel 1 / 2 pixel 1 pixel 2 pixels 4 pixels 8 pixels 16 pixels 32 pixels

[0205] The direction index indicates the direction of the MVD relative to the starting point. The direction index can represent four directions as shown below:

[0206]

[0207] The UMVE flag is signaled immediately after the Skip flag and the Merge flag are sent. If the Skip and Merge flags are true, the UMVE flag is parsed. If the UMVE flag is equal to 1, the UMVE syntax is parsed. However, if it is not 1, the Affine flag is parsed. If the Affine flag is equal to 1, this is the Affine mode, but if it is not 1, the Skip / Merge index will be parsed for the Skip / Merge mode of the VTM.

[0208] No additional line buffer is needed for UMVE candidates. Because software's Skip / Merge candidates are used directly as base candidates. Using the input UMVE index, the complement of the MV is determined before motion compensation. No need to keep a long line buffer for this.

[0209] 2.13 Example of Affine Merge Mode with Prediction Shift

[0210] In some embodiments, UMVE is extended to affine merge mode, which we refer to as UMVE affine mode hereinafter. The proposed method selects the first available affine merge candidate as the base predictor. Then, it applies motion vector offsets to the motion vector values ​​of each control point from the base predictor. If no affine merge candidate is available, the proposed method is not used.

[0211] The inter prediction direction of the selected basic prediction value and the reference index for each direction are used without change.

[0212] In the current implementation, the affine model of the current block is assumed to be a 4-parameter model, and only 2 control points need to be derived. Therefore, only the first 2 control points of the basic prediction value will be used as the control point prediction value.

[0213] For each control point, the zero_MVD flag is used to indicate whether the control point of the current block has the same MV value as the corresponding control point prediction value. If the zero_MVD flag is true, no additional signaling is required for the control point. Otherwise, the distance index and offset direction index are signaled for the control point.

[0214] Use a distance offset table of size 5 as shown in the table below.

[0215] Distance from IDX 0 1 2 3 4 Distance-Offset 1 / 2 pixel 1 pixel 2 pixels 4 pixels 8 pixels

[0216] The distance index is signaled to indicate which distance offset to use. The mapping between distance index and distance offset value is as follows: Fig.23 shown.

[0217] The direction index can represent four directions as shown below, where only the x or y direction may have MV differences, but not both.

[0218] Offset direction IDX 00 01 10 11 x-dir-factor +1 -1 0 0 y-dir-factor 0 0 +1 -1

[0219] If inter prediction is unidirectional, the signal distance offset is applied to the offset direction of the prediction value of each control point. The result will be the MV value of each control point.

[0220] For example, when the basic prediction value is unidirectional, and the motion vector value of the control point is MVP (v px ,v py ). When the distance offset and direction index are signaled, the motion vector of the corresponding control point of the current block will be calculated as follows.

[0221] MV(v x ,v y )=MVP(v px ,v py )+MV(x-dir-factor*distance offset,y-dir-factor*distance offset)

[0222] If inter prediction is bidirectional, the signaled distance offset is applied in the signaled offset direction of the L0 motion vector of the control point prediction value; and the same distance offset with the opposite direction is applied to the L1 motion vector of the control point prediction value. The result will be the MV value of each control point in each inter prediction direction.

[0223] For example, when the basic prediction value is unidirectional, and the motion vector value of the control point on L0 is MVP L0 (v 0px ,v 0py ), and the motion vector of the control point on L1 is MVP L1 (v 1px ,v 1py ). When the distance offset and direction index are signaled, the motion vector of the corresponding control point of the current block will be calculated as follows.

[0224] MV L0 (v 0x ,v 0y )=MVP L0 (v 0px ,v 0py )+MV(x-dir-factor*distance offset,y-dir-factor*distance offset)

[0225] MV L1 (v 0x ,v 0y )=MVPL1 (v 0px ,v 0py )+MV(-x-dir-factor*distance offset,-y-dir-factor*distance offset)

[0226] 2.14 Example of Bidirectional Optical Flow (BIO)

[0227] The Bidirectional Optical Flow (BIO) method is a sample-level motion refinement that is performed on the basis of block-wise motion compensation for bidirectional prediction. In some embodiments, the sample-level motion refinement does not use signaling.

[0228] Let I (k) is the luminance value of reference k (k=0,1) after block motion compensation, and is represented by To indicate I (k) The horizontal and vertical components of the gradient. Assuming that the optical flow is valid, the motion vector field (v x , v y ) is given by:

[0229]

[0230] Combine the optical flow equation with Hermite interpolation to obtain the motion trajectory of each sample point, and finally get the function value I (k) And the derivative and The only third-order polynomial that matches is the BIO prediction:

[0231]

[0232] Fig.24 An example optical flow trajectory in the bidirectional optical flow (BIO) method is shown. Here, τ0 and τ1 represent the distance to the reference frame. The distances τ0 and τ1 are calculated based on the POC of Ref0 and Ref1: τ0 = POC(current) - POC(Ref0), τ1 = POC(Ref1) - POC(current). If the two predictions are from the same temporal direction (either both from the past or both from the future), the signals are different (e.g., τ0·τ1 < 0). In this case, BIO is only applied when the predictions are not from the same moment (e.g., τ0≠τ1), both reference regions have non-zero motion (e.g., MVx0, MVy0, MVx1, MVy1≠0), and the block motion vector is proportional to the temporal distance (e.g., MVx0 / MVx1 = MVy0 / MVy1 = -τ0 / τ1).

[0233] The motion vector field (v) is determined by minimizing the difference Δ between the values ​​at point A and point B. x , vy ). Fig.24 An example of the intersection of the motion trajectory and the reference frame plane is shown. The model only uses the first linear term of the local Taylor expansion for Δ:

[0234]

[0235] All values ​​in the above equation depend on the sample point location, denoted as (i′, j′). Assuming that the motion is consistent in the local surrounding area, we can minimize Δ within a (2M+1)×(2M+1) square window Ω centered at the current prediction point (i, j), where M is equal to 2:

[0236]

[0237] For this optimization problem, JEM uses a simplified approach, first minimizing in the vertical direction and then minimizing in the horizontal direction. This yields the following equation:

[0238]

[0239]

[0240] in,

[0241]

[0242] To avoid division by zero or very small values, regularization parameters r and m can be introduced into Equations 9 and 10, where:

[0243] r=500·4 d-8 Equation (12)

[0244] m=700·4 d-8 Equation (13)

[0245] Here, d is the bit depth of the video samples.

[0246] To keep the memory access of BIO the same as for regular bi-predictive motion compensation, all predictions and gradient values Both are calculated for the position inside the current block. Fig.25A An example of an access location external to block 2500 is shown. Fig.25A As shown in equation (9), a (2M+1)×(2M+1) square window Ω centered at the current prediction point on the boundary of the prediction block needs to access the location outside the block. In JEM, the location outside the block The value of is set equal to the nearest available value inside the block. For example, this can be implemented as filling area 2501, such as Fig.25B shown.

[0247] Using BIO, it is possible to refine the motion field for each sample. To reduce the computational complexity, a block-based BIO design is used in JEM. The motion refinement can be calculated based on a 4×4 block. In the block-based BIO, s in equation (9) can be aggregated for all samples in the 4×4 block. n value, then s n The aggregated value of is used to derive the BIO motion vector offset of the 4×4 block. More specifically, the following formula can be used for block-based BIO derivation:

[0248]

[0249] Here, b k represents the sample set of the kth 4×4 block belonging to the prediction block. n ((s n，bk )>>4) instead to derive the associated motion vector offset.

[0250] In some scenarios, the MV refinement of BIO may be unreliable due to noise or irregular motion. Therefore, in BIO, the amplitude of MV refinement is clipped to a threshold. The threshold is determined based on whether the reference pictures of the current picture are all from one direction. For example, if all the reference pictures of the current picture are from one direction, the threshold is set to 12×2. 14-d ; otherwise, it is set to 12×2 13-d .

[0251] The gradient of the BIO may be calculated simultaneously with the motion compensated interpolation using an operation consistent with the HEVC motion compensation process (e.g., a 2D separable FIR (Finite Impulse Response). In some embodiments, the input to the 2D separable FIR is the same reference frame samples as the motion compensation process and the fractional positions (fracX, fracY) according to the fractional part of the block motion vector. For the horizontal gradient The signal is first interpolated vertically using BIOfilterS corresponding to the fractional position frracY with a descaling offset d-8. Then a gradient filter BIOfilterG corresponding to the fractional position fracX with a descaling offset 18-d is applied in the horizontal direction. For the vertical gradient Gradient filter is applied vertically using BIOfilterG corresponding to fractional position fracY with descaling offset d-8. Signal displacement is then performed in horizontal direction using BIOfilterS corresponding to fractional position fracX with descaling offset up to 18-d. The length of interpolation filter BIOfilterG and signal displacement BIOfilterF used for gradient calculation can be short (e.g., 6 taps) to maintain reasonable complexity. Table 1 shows example filters for gradient calculation of different fractional positions of block motion vectors in BIO. Table 2 shows example interpolation filters that can be used for prediction signal generation in BIO.

[0252] Table 1: Exemplary filters for gradient calculations in BIO

[0253] Fractional pixel position Gradient interpolation filter (BIOfilterG) 0 {8,-39,-3,46,-17,5} 1 / 16 {8,-32,-13,50,-18,5} 1 / 8 {7,-27,-20,54,-19,5} 3 / 16 {6,-21,-29,57,-18,5} 1 / 4 {4,-17,-36,60,-15,4} 5 / 16 {3,-9,-44,61,-15,4} 3 / 8 {1,-4,-48,61,-13,3} 7 / 16 {0,1,-54,60,-9,2} 1 / 2 {-1,4,-57,57,-4,1}

[0254] Table 2: Exemplary interpolation filters for prediction signal generation in BIO

[0255] Fractional pixel position Interpolation filter for prediction signal (BIOfilterS) 0 {0,0,64,0,0,0} 1 / 16 {1,-3,64,4,-2,0} 1 / 8 {1,-6,62,9,-3,1} 3 / 16 {2,-8,60,14,-5,1} 1 / 4 {2,-9,57,19,-7,2} 5 / 16 {3,-10,53,24,-8,2} 3 / 8 {3,-11,50,29,-9,2} 7 / 16 {3,-11,44,35,-10,3} 1 / 2 {3,-10,35,44,-11,3}

[0256] In JEM, BIO can be applied to all bidirectionally predicted blocks when the two predictions come from different reference pictures. BIO can be disabled when Local Illumination Compensation (LIC) is enabled for a CU.

[0257] In some embodiments, OBMC is applied to the block after the normal MC process. To reduce computational complexity, BIO may not be applied during the OBMC process. This means that when using its own MV, BIO is applied to the MC process of the block, but when using the MV of the neighboring block during the OBMC process, BIO is not applied to the MC process.

[0258] 2.15 Example of Decoder-side Motion Vector Refinement (DMVR)

[0259] In the bidirectional prediction operation, for the prediction of a block region, two prediction blocks formed using the motion vectors (MVs) of list 0 and the MVs of list 1, respectively, are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of the bidirectional prediction are further refined by a bilateral template matching process. Bilateral template matching is applied in the decoder to perform a distortion-based search between the bilateral template and the reconstructed samples in the reference picture to obtain the refined MV without sending additional motion information.

[0260] In DMVR, bilateral templates are generated from the initial MV0 of list 0 and MV1 of list 1 as a weighted combination (i.e., average) of the two prediction blocks, as shown in Fig.26As shown. The template matching operation includes calculating a cost metric between the generated template and the sample area (around the initial prediction block) in the reference picture. For each of the two reference pictures, the MV that produces the minimum template cost is considered to be the updated MV in the list to replace the original MV. In JEM, 9 MV candidates are searched for each list. The 9 candidate MVs include the original MV and 8 surrounding MVs, where one luminance sample is offset relative to the original MV in the horizontal or vertical direction or both. Finally, the two new MVs (i.e., Fig.26 MV0′ and MV1′ shown) are used to generate the final bidirectional prediction result. The sum of absolute differences (SAD) is used as the cost metric.

[0261] DMVR is applied to Merge mode of bi-prediction, where one MV comes from a past reference picture and the other comes from a future reference picture, without transmitting additional syntax elements. In JEM, DMVR is not applied when LIC, affine motion, FRUC, or sub-CU Merge candidate is enabled for a CU.

[0262] 3. Exemplary embodiments related to the disclosed technology

[0263] For example, PCT / CN2018 / 109425 (incorporated herein by reference in its entirety) describes an embodiment including an MV update method and a two-step inter-frame prediction method. The derived MV between reference block 0 and reference block 1 in BIO is scaled and added to the original motion vectors of list 0 and list 1. At the same time, the updated MV is used to perform motion compensation, and a second inter-frame prediction is generated as the final prediction. Other embodiments include modifying the temporal gradient by removing the average difference between reference block 0 and reference block 1. In some further embodiments, the MV update method and the two-step inter-frame prediction method are extended to be performed multiple times.

[0264] 4. Disadvantages of existing implementation methods

[0265] In some existing implementations, it is not well defined how to apply BIO or / and DMVR or any other decoder-side motion vector derivation / refinement tools to CUs encoded and decoded with multiple hypothesis prediction mode or GBI mode.

[0266] In other existing implementations using generalized bidirectional prediction (GBI), when encoding weight factor indices (eg, GBI indices), all binary values ​​are encoded and decoded with contexts, which is computationally complex.

[0267] 5. Example of how the DMVD tool can be used in conjunction with other tools

[0268] Embodiments of the currently disclosed technology overcome the shortcomings of existing implementations, thereby providing video codecs with higher codec efficiency. Based on the disclosed technology, the juxtaposition of the DMVD tool with other video codec tools can enhance existing and future video codec standards, as described in the following examples described for various implementations. The examples of the disclosed technology provided below explain the general concepts and are not meant to be interpreted as limiting. In one example, unless explicitly indicated to the contrary, the various features described in these examples can be combined.

[0269] The disclosed technology describes how to apply BIO or / and DMVR or any other decoder side motion vector derivation / refinement tool to blocks encoded and decoded with multiple hypothesis prediction mode or GBI mode. In the following, DMVD (decoder side motion vector derivation) is used to represent BIO or / and DMVR or / and FRUC or / and any other decoder side motion vector derivation / refinement technology. In addition, the extension of DMVD technology to other coding and decoding methods is also proposed in this document.

[0270] The above examples may be incorporated into the context of the methods described below, for example, methods 2700 and 2750, which may be implemented at a video decoder or a video encoder.

[0271] Example 1. It is proposed that DMVD may be performed on blocks encoded with multiple hypothesis prediction modes (eg, as described in Sections 2.9, 2.10, and 2.11).

[0272] (a) In one example, for a unidirectionally predicted block with multiple hypotheses, DMVD can be performed if it is predicted with two reference blocks from different prediction directions.

[0273] (b) In one example, if three reference blocks are used to predict a block, DMVD can be performed by selecting two reference blocks. Denoting the three reference blocks as ref0, ref1, and ref2, assuming that ref0 is from prediction direction X, and ref1 and ref2 are from prediction directions 1–X, DMVD can be performed on reference block pairs (ref0, ref1) or / and (ref0, ref2).

[0274] (i) In one example, DMVD may be performed only once. In this case, (ref0, ref1) or (ref0, ref2) may be utilized in the DMVD process.

[0275] (ii) Alternatively, DMVD may be performed twice. In this case, (ref0, ref1) and (ref0, ref2) may be utilized in the first DMVD process and the second DMVD process, respectively. Alternatively, (ref0, ref2) and (ref0, ref1) may be utilized in the first DMVD process and the second DMVD process, respectively.

[0276] (1) In one example, the refinement information (e.g., the refinement motion information generated due to BIO or DMVR) of one DMVD process (e.g., the first DMVD) can be used as the input of another DMVD process (e.g., the second DMVD). That is, a sequential process (sequential mode) of two DMVD processes can be applied.

[0277] (2) Alternatively, multiple DMVD processes can be called with the same input so that multiple processes can be completed in parallel (parallel mode).

[0278] (iii) In one example, if ref0 is refined twice by BIO, denoted as ref0_r1 and ref0_r2, ref0_r1 and ref0_r2 may be used jointly (e.g., averaged or weighted averaged) to generate the final refined value of ref0. For example, ref0'=(ref0_r1+ref0_r2+1)>>1 is used as the refined ref0. Alternatively, ref0_r1 or ref0_r2 is used as the refined ref0.

[0279] (iv) In one example, if the motion information of ref0 (denoted as MV0) is refined twice by BIO or DMVR, denoted as MV0_r1 and MV0_r2, MV0_r1 and MV0_r2 may be used jointly (e.g., averaged or weighted averaged) to generate a final refined value of MV0. For example, MV0'=(MV0_r1+MV0_r2+1)>>1 is used as the refined MV0. Alternatively, MV0_r1 or MV0_r2 is used as the refined MV0.

[0280] (c) In one example, if a block is predicted with N reference blocks, and M reference blocks are from prediction direction X, and N–M reference blocks are from prediction directions 1–X, then DMVD can be performed on any one / some of the two reference blocks, one from prediction direction X and the other from prediction direction 1–X.

[0281] (i) Similar to (b)(ii), multiple DMVD processes can be called in parallel mode or sequential mode.

[0282] (ii) In one example, if the reference block is refined T times by BIO, some or all of the values ​​refined T times may be used jointly (eg, using an average or weighted average) to derive the final refined value of the reference block.

[0283] (iii) Alternatively, if the reference block is refined T times by BIO, some or all of the values ​​refined PT times can be used jointly (e.g., using an average or weighted average) to derive the final refined value of the reference block. For example, PT is equal to 1, 2, 3, 4, ..., T-1.

[0284] (iv) In one example, if the motion information of a reference block is refined T times by, for example, BIO or DMVR, some or all of these T-times refined MVs may be used jointly (for example, using an average or weighted average) to derive a final refined MV of the reference block.

[0285] (v) Alternatively, if the motion information of the reference block is refined T times, for example, by BIO or DMVR, then part or all of the MVs refined PT times can be used jointly (e.g., using an average or weighted average) to derive the final refined value of the reference block. For example, PT is equal to 1, 2, 3, 4, ..., T-1.

[0286] (d) In one example, if a block is predicted using multiple motion information sets (e.g., one motion information set is from the AMVP mode and another motion information set is from the Merge candidate (e.g., as described in Section 2.10), or both motion information sets are from the Merge candidate (e.g., as described in Section 2.11)), DMVD may be performed on each / certain motion information set.

[0287] (i) In one example, when the motion information is bidirectional motion information, DMVD is called.

[0288] (ii) In one example, in the multi-hypothesis inter prediction mode (as described in Section 2.9), DMVD is performed only on the non-additional motion information.

[0289] (e) In one example, DMVD can be performed at most once.

[0290] (i) For example, if the block is encoded and decoded using the multi-hypothesis Merge / skip mode (eg, as described in Section 2.11), DMVD is performed only on the first selected Merge candidate.

[0291] (1) Alternatively, DMVD is performed only on the second selected Merge candidate.

[0292] (2) Alternatively, the 2 / N selected Merge candidates are checked in sequence, and DMVD is performed only on the first available bidirectional Merge candidate.

[0293] (ii) For example, if the block is encoded or decoded using the multi-hypothesis AMVP mode (as described in Section 2.10), DMVD is performed only on the Merge candidates.

[0294] (iii) For example, if the block is encoded or decoded using a multi-hypothesis inter-frame prediction mode (e.g., as described in Section 2.9), the first available reference blocks from List 0 and List 1 (if any) are identified based on the signaling order of the corresponding syntax elements, and DMVD is performed only on these two reference blocks.

[0295] Example 2. It is proposed that when asymmetric weight factors are used for bidirectional prediction (such as GBI, LIC, etc.) or multi-hypothesis prediction, such weight factors can be used in the DMVD process.

[0296] (a) In one example, before calculating the temporal gradient or / and the spatial gradient in BIO, each reference block can be scaled by its corresponding weight factor.

[0297] (b) In one example, when performing bilateral matching or template matching in DMVR, each reference block can be scaled by its corresponding weight factor.

[0298] (c) In one example, if template matching is used in DMVR, such weighting factors may be used when generating the template.

[0299] Example 3. It is proposed that DMVD can be used in AMVP mode when all MVD components are zero. Alternatively, if the MVD components are zero in prediction direction X and non-zero in prediction directions 1-X, DMVD can be used to refine the motion vector in prediction direction X. In one example, in DMVR, the prediction signals in lists 1-X are used as templates to find the best motion vector in list X.

[0300] Example 4. It is proposed that DMVD can be used to refine translational motion parameters in either bidirectional affine mode or UMVE affine mode.

[0301] (a) Alternatively, in bidirectional affine inter mode, DMVD is used to refine the translational motion parameters only when their MVDs are all zero.

[0302] (b) Alternatively, in UMVE affine mode, DMVD is used to refine the translational motion parameters only when their MVDs are all zero.

[0303] Example 5. It is proposed to enable DMVD in UMVE mode.

[0304] (a) Alternatively, DMVD is disabled when there is a non-zero MVD component in UMVE mode.

[0305] (b) Alternatively, DMVD is disabled in UMVE mode.

[0306] Example 6. It is proposed that whether to enable DMVD may depend on the decoded motion information.

[0307] a. It is proposed that DMVD can be enabled when the absolute value of a block (expressed as MV and / or MVD and / or ABS(MV) of a decoded motion vector (expressed as (MV_x, MV_y)) is less than a threshold TH1, where TH1>=0.

[0308] i.ABS(MV) can be defined as:

[0309] 1. ABS(MV) = abs(MV_x);

[0310] 2. ABS(MV) = abs(MV_y);

[0311] 3.ABS(MV)=f(abs(MV_x),abs(MV_y)), where f is a function.

[0312] a.ABS(MV)=min(abs(MV_x),abs(MV_y));

[0313] b.ABS(MV)=max(abs(MV_x),abs(MV_y));

[0314] c.ABS(MV)=abs(MV_x)+abs(MV_y);

[0315] d.ABS(MV)=(abs(MV_x)+abs(MV_y)) / 2;

[0316] ii. In one example, if the ABS(MV) of a block is less than TH1 in any prediction direction, DMVD may be enabled.

[0317] iii. In one example, if the ABS(MV) of a block is less than TH1 in both prediction directions, DMVD may be enabled.

[0318] iv. In one example, both components (horizontal / vertical) of the absolute value MV and / or the absolute value MVD are required to be less than a threshold. Alternatively, two thresholds are used, one for the horizontal component and the other for the vertical component. The horizontal component is required to be less than a first threshold and the vertical component is less than a second threshold.

[0319] v. In one example, it is required that the horizontal component is less than a first threshold or the vertical component is less than a second threshold.

[0320] vi. In some examples, the first threshold and the second threshold are the same.

[0321] vii. In one example, this approach may not work for non-UMVE Merge mode.

[0322] viii. In one example, this approach may not be applicable to UMVE mode.

[0323] ix. In one example, this approach may not be applicable to AMVP mode.

[0324] x. The above MV or MVD can be derived from the AMVP model, or it can be derived from the UMVE (also known as MMVD) model.

[0325] b. It is proposed that DMVD can be enabled when the absolute value of the block (expressed as MV and / or MVD and / or the decoded motion vector (expressed as (MV_x, MV_y)) is greater than a threshold TH2, where TH2>=0.

[0326] i.ABS(MV) can be defined as:

[0327] 1. ABS(MV) = abs(MV_x);

[0328] 2. ABS(MV) = abs(MV_y);

[0329] 3.ABS(MV)=f(abs(MV_x),abs(MV_y)), where f is a function.

[0330] a.ABS(MV)=min(abs(MV_x),abs(MV_y));

[0331] b.ABS(MV)=max(abs(MV_x),abs(MV_y));

[0332] c.ABS(MV)=abs(MV_x)+abs(MV_y);

[0333] d.ABS(MV)=(abs(MV_x)+abs(MV_y)) / 2;

[0334] ii. In one example, if the ABS(MV) of a block is greater than TH2 in any prediction direction, DMVD may be disabled.

[0335] iii. In one example, if the ABS(MV) of a block is greater than TH2 in both prediction directions, DMVD may be disabled.

[0336] iv. In one example, both components (horizontal / vertical) of the absolute value MV and / or the absolute value MVD are required to be greater than a threshold. Alternatively, two thresholds are used, one for the horizontal component and the other for the vertical component. The horizontal component is required to be greater than a first threshold and the vertical component is required to be greater than a second threshold.

[0337] v. In one example, it is required that the horizontal component is greater than a first threshold or the vertical component is greater than a second threshold.

[0338] vi. In some examples, the first threshold and the second threshold are the same.

[0339] vii. In one example, this approach may not work for non-UMVE Merge mode.

[0340] viii. In one example, this approach may not be applicable to UMVE mode.

[0341] ix. In one example, this approach may not be applicable to AMVP mode.

[0342] x. The above MV or MVD can be derived from the AMVP model, or it can be derived from the UMVE (also known as MMVD) model.

[0343] c. When a block is coded and decoded using bidirectional prediction, the motion information of only one reference picture list can be used.

[0344] i. Alternatively, the motion information of both reference picture lists may be used. In one example, the MV (or MVD) is the maximum / minimum / average MV (or MVD) of the two reference picture lists.

[0345] d. It is proposed that for the case of bidirectional prediction, whether DMVD is enabled or disabled may depend on the distance between the two motion vectors of each reference picture list.

[0346] e. The determination to enable / disable DMVD can be controlled individually for each reference picture list.

[0347] Example 7. It is proposed that DMVD can be applied under certain conditions, such as based on block size, coding mode, motion information, slice / picture / slice type.

[0348] (a) In one example, when the block size contains fewer than M×H samples (e.g., 16 or 32 or 64 luma samples), DMVD is not allowed.

[0349] (b) In one example, when the block size contains more than M×H samples (e.g., 16 or 32 or 64 luma samples), DMVD is not allowed.

[0350] (c) Alternatively, when the minimum dimension of the width or height of the block is less than or not greater than X, DMVD is not allowed. In one example, X is set to 8.

[0351] (d) Alternatively, when the width of the block > th1 or ≥ th1 and / or the height of the block > th2 or ≥ th2, DMVD is not allowed. In one example, X is set to 64.

[0352] (i) For example, for a 128×128 block, DMVD is disabled.

[0353] (ii) For example, for N×128 / 128×N blocks (for N≥64), DMVD is disabled.

[0354] (iii) For example, for N×128 / 128×N blocks (for N≥4), DMVD is disabled.

[0355] (e) Alternatively, when the width of the block < th1 or ≤ th1 and / or the height of the block < th2 or ≤ th2, DMVD is not allowed. In one example, th1 or th2 is set to 8.

[0356] (f) In one example, for blocks encoded / decoded in the AMVP mode, DMVD is disabled.

[0357] (g) In one example, for blocks encoded / decoded in the skip mode, DMVD is disabled.

[0358] (h) In one example, for blocks using GBI, DMVD is disabled.

[0359] (i) In one example, for blocks using multi-hypothesis inter prediction (as described in Sections 2.9, 2.10, and 2.11), e.g., in the case of predicting a CU from more than 2 reference blocks, DMVD is disabled.

[0360] (j) In one example, for blocks / sub-blocks when the absolute average difference between two reference blocks / sub-blocks is greater than a threshold, DMVD is disabled.

[0361] Example 8. Applying DMVD at the sub-block level is proposed.

[0362] (a) In one example, DMVD may be called for each sub-block.

[0363] (b) In one example, when the width or height of a block is greater than (or equal to) a threshold value L, or both the width and the height are greater than (or equal to) the threshold value L, the block may be divided into a plurality of sub-blocks. Each sub-block is processed in the same manner as a normal codec block whose size is equal to the sub-block size.

[0364] (i) In one example, L is 64, a 64×128 / 128×64 block is divided into two 64×64 sub-blocks, and a 128×128 block is divided into four 64×64 sub-blocks. However, an N×128 / 128×N block (where N<64) is not divided into sub-blocks.

[0365] (ii) In one example, L is 64, a 64×128 / 128×64 block is divided into two 64×64 sub-blocks, and a 128×128 block is divided into four 64×64 sub-blocks. Meanwhile, an N×128 / 128×N block (where N<64) is divided into two N×64 / 64×N sub-blocks.

[0366] (c) The threshold L may be pre-defined or signaled at the SPS / PPS / picture / slice / slice group / slice level.

[0367] (d) Alternatively, the threshold may depend on certain codec information, such as block size, picture type, temporal layer index, etc.

[0368] Example 9. In one example, whether and how to apply the above methods (eg, motion refinement methods such as DMVR or / and BIO and / or other decoder-side motion refinement techniques) depends on the reference picture.

[0369] (a) In one example, if the reference picture is the current codec picture, the motion refinement method is not applied.

[0370] (b) In one example, if the reference picture is the current codec picture, the multi-time motion refinement method required in the previous bullet point is not applied.

[0371] Example 10. Even if the two reference blocks / reference pictures are from the same reference picture list, the above method as well as the existing DMVD method (eg, BIO / DMVR) can be applied.

[0372] (a) Alternatively, when two reference blocks are from the same reference picture list, the two reference pictures are required to span the current picture covering the current block. That is, one reference picture has a smaller POC value and the other reference picture has a larger POC value compared to the POC of the current picture.

[0373] (b) In one example, the conditional check for whether the prediction direction is bi-directional prediction for enabling / disabling BIO is removed. That is, whether BIO or DMVD is enabled is independent of the prediction direction value.

[0374] (c) When the product of the POC differences between the current picture and its two reference pictures (from the same reference picture list or different reference picture lists) is less than 0, BIO or DMVD may be enabled.

[0375] (d) BIO or DMVD may be enabled when the product of the POC differences between the current picture and its two reference pictures (from the same reference picture list or different reference picture lists) is less than or equal to 0 (eg, one or both reference pictures are the current picture).

[0376] Example 11. It is proposed to bypass encoding and decoding of some binary values ​​when encoding the GBI index. The maximum length of the coded binary value of the GBI index is denoted as maxGBIIdxLen.

[0377] (a) In one example, only the first binary value is decoded with context, while all other binary values ​​are bypass decoded.

[0378] (i) In one example, a context is used to encode a first binary value.

[0379] (ii) In one example, more than one context is used to encode the first binary value. For example, the following three contexts are used:

[0380] (1)ctxIdx=aboveBlockIsGBIMode+leftBlockIsGBIMode;

[0381] (2) if the above adjacent block is encoded or decoded in GBI mode, aboveBlockIsGBIMode is equal to 1, otherwise it is equal to 0; and

[0382] (3) If the adjacent block on the left is encoded or decoded in GBI mode, leftBlockIsGBIMode is equal to 1, otherwise it is equal to 0.

[0383] (b) In one example, only the first K binary values ​​are coded with context, while all other binary values ​​are bypass coded, where 0 = <K<=maxGBIIdxLen。

[0384] (i) In one example, except for the first binary value, all context-encoded binary values ​​share one context.

[0385] (ii) In one example, except for the first binary value, each context-coded binary value uses one context.

[0386] Fig.27A A flow chart of an exemplary method for video decoding is shown. The method 2700 includes, at operation 2702, for a conversion between a current block of a video and a bitstream representation of the video, making a decision regarding selective enabling of a decoder-side motion vector derivation (DMVD) tool for the current block, the DMVD tool deriving a refinement of motion information signaled in the bitstream representation, and the conversion using a Merge mode and a motion vector difference indicated by a motion direction and a motion magnitude.

[0387] The method 2700 includes, at operation 2704, performing a conversion between the current block and a bitstream representation based on the determination.

[0388] Fig.27B A flow chart of an exemplary method for video decoding is shown. The method 2750 includes, at operation 2752, making a decision regarding selective enabling of a decoder-side motion vector derivation (DMVD) tool for a current block of a video based on decoded motion information associated with the current block of the video, the DMVD tool deriving a refinement of motion information signaled in a bitstream representation of the video.

[0389] The method 2750 includes, at operation 2754, performing a conversion between the current block and a bitstream representation based on the determination.

[0390] In some embodiments, the following technical solutions may be implemented:

[0391] 1. A method for video processing (eg, Fig.27AMethod 2700 in ), comprising: for a conversion between a current block of video and a bitstream representation of the video, making (2702) a decision regarding selective enabling of a decoder-side motion vector derivation (DMVD) tool for the current block, wherein the DMVD tool derives a refinement of motion information signaled in the bitstream representation, and wherein the conversion uses a Merge mode and a motion vector difference, the motion vector difference being indicated by a motion direction and a motion magnitude; and performing (2704) the conversion between the current block and the bitstream representation based on the decision. For example, in some cases, the decision may be to enable DMVD, in which case encoding or decoding of the current block is performed using DMVD. In some cases, the decision may be to disable (or not use) DMVD, in which case encoding or decoding may be performed without using the DMVD tool.

[0392] 2. The method according to claim 1, wherein a DMVD tool is enabled.

[0393] 3. The method of claim 1, wherein the DMVD tool is disabled.

[0394] 4. The method according to scheme 1, wherein when it is determined that the current block is encoded and decoded using Merge mode, the DMVD tool is disabled.

[0395] 5. A method according to any one of schemes 1 to 4, wherein the Merge mode also includes a starting point of the motion information indicated by the Merge index, and wherein the final motion information of the current block is based on the motion vector difference and the starting point.

[0396] 6. The method according to claim 1, wherein when it is determined that the Merge mode includes a non-zero motion vector difference (MVD) component, a DMVD tool is disabled.

[0397] 7. A method for video processing (e.g., Fig.27B The method 2750 in the claims comprises: based on decoded motion information associated with a current block of a video, making (2752) a decision regarding selective enabling of a decoder-side motion vector derivation (DMVD) tool for the current block, wherein the DMVD tool derives a refinement of motion information signaled in a bitstream representation of the video; and performing (2754) conversion between the current block and the bitstream representation based on the decision.

[0398] 8. A method according to Scheme 7, wherein the DMVD tool is enabled when it is determined that (1) the absolute value of the motion vector, or (2) the absolute value of the motion vector difference, or (3) the absolute value of the decoded motion vector of the current block is (a) less than a threshold value TH1, where TH1≥0, or (b) greater than a threshold value TH2, where TH2≥0.

[0399] 9. A method according to Scheme 7, wherein, when it is determined that the absolute value of the decoded motion vector of the current block is (a) less than a threshold value THh1 or THv1, respectively, in the horizontal or vertical prediction direction, where THh1≥0 and THv1≥0, or (b) greater than a threshold value THh2 or THv2, respectively, where THh2≥0 and THv2≥0, the DMVD tool is enabled.

[0400] 10. The method of claim 8 or 9, wherein the current block is encoded using a bi-directional prediction mode, and wherein the decision is also based on motion information from only one reference picture list associated with the current block.

[0401] 11. The method of claim 8 or 9, wherein the current block is encoded using a bi-directional prediction mode, and wherein the decision is based on motion information from both reference picture lists associated with the current block.

[0402] 12. The method according to any one of schemes 7 to 9, wherein when it is determined that the current block is not encoded using a specific encoding mode, the DMVD tool is enabled.

[0403] 13. The method according to scheme 12, wherein the specific encoding mode includes a final motion vector expression (UMVE) mode or an advanced motion vector prediction (AMVP) mode.

[0404] 14. The method according to scheme 12, wherein the specific encoding mode does not include a bidirectional prediction mode, and includes a condition that a distance between two motion vectors of each reference picture list of the current block is greater than or less than a predetermined value.

[0405] 15. A method according to Scheme 7, wherein the DMVD tool is enabled when it is determined that the function of the absolute value of the decoded motion vector (MV) of the current block in the horizontal prediction direction (MV_x) and the vertical prediction direction (MV_y) is (a) less than a threshold TH1, where TH1≥0, or (b) greater than a threshold TH2, where TH2≥0.

[0406] 16. A method according to scheme 15, wherein the function is one of the following: f(abs(MV))=min(abs(MV_x),abs(MV_y)), or f(abs(MV))=max(abs(MV_x),abs(MV_y)), or f(abs(MV))=(abs(MV_x)+abs(MV_y)), or f(abs(MV))=(abs(MV_x)+abs(MV_y)) / 2.

[0407] 17. A method according to any one of schemes 1 to 16, wherein the DMVD tool includes at least one of a bidirectional optical flow (BIO) algorithm, a decoder-side motion vector refinement (DMVR) algorithm, or a frame rate up conversion (FRUC) algorithm.

[0408] 18. A method according to any one of options 1 to 17, wherein the conversion generates the current block from a bitstream representation.

[0409] 19. A method according to any one of options 1 to 17, wherein the conversion generates a bitstream representation from the current block.

[0410] 20. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement any one of the methods 1 to 19.

[0411] 21. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising a program code for executing the method according to any one of schemes 1 to 19.

[0412] 6. Example Implementations of the Disclosed Technology

[0413] Fig.28 2700. The apparatus 2800 may be used to implement one or more methods described herein. The apparatus 2800 may be included in a smartphone, a tablet, a computer, an Internet of Things (IoT) receiver, etc. The apparatus 2800 may include one or more processors 2802, one or more memories 2804, and video processing hardware 2806. The processor(s) 2802 may be configured to implement one or more methods described in this document (including but not limited to method 2700). The memory(s) 2804 may be used to store data and code for implementing the methods and techniques described herein. The video processing hardware 2806 may be used to implement some of the techniques described in this document in hardware circuits.

[0414] In some embodiments, the video encoding and decoding method can use the reference Fig.28 The method is implemented by the device implemented on the hardware platform.

[0415] Fig.292900 is a block diagram illustrating an example video processing system 2900 in which various techniques disclosed herein may be implemented. Various embodiments may include some or all of the components of system 2900. System 2900 may include an input 2902 for receiving video content. The video content may be received in a raw or uncompressed format, for example, 8 or 10 bit multi-component pixel values, or may be in a compressed or encoded format. Input 2902 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces (such as Ethernet, passive optical networks (PONs), etc.) and wireless interfaces (such as Wi-Fi or cellular interfaces).

[0416] System 2900 may include a coding component 2904, which may implement various coding methods described in this document. Coding component 2904 may reduce the average bit rate of the video from the input 2902 of coding component 2904 to the output to generate a codec representation of the video. Therefore, coding techniques are sometimes referred to as video compression or video transcoding techniques. The output of coding component 2904 may be stored or transmitted via a connected communication, as shown in component 2906. Component 2908 may use the bitstream (or encoding) representation of the stored or communicated video received at input 2902 to generate pixel values ​​or displayable video sent to display interface 2910. The process of generating user-visible video from a bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "coding" operations or tools, it should be understood that coding tools or operations are used at the encoder, and the corresponding decoding tools or operations of the inverse coding results will be performed by the decoder.

[0417] Examples of peripheral bus interfaces or display interfaces may include universal serial bus (USB) or high definition multimedia interface (HDMI) or display port, etc. Examples of storage interfaces include SATA (serial advanced technology attachments), PCI, IDE interfaces, etc. The technology described in this document may be embodied in various electronic devices, such as mobile phones, laptops, smart phones, or other devices capable of performing digital data processing and / or video display.

[0418] It will be appreciated from the foregoing that specific embodiments of the presently disclosed technology have been described herein for purposes of illustration, but that various modifications may be made without departing from the scope of the invention. Accordingly, the presently disclosed technology is not to be limited except as set forth in the appended claims.

[0419] The embodiments of the subject matter and functional operations described in this patent document can be implemented in various systems, digital electronic circuits, or in computer software, firmware or hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-transitory computer-readable medium, for execution by a data processing device or for controlling the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material combination that implements a machine-readable propagation signal, or a combination of one or more of them. The term "data processing unit" or "data processing device" covers all devices, equipment and machines for processing data, for example, including a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an operating environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0420] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one or more computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0421] The processes and logic flows described in this specification can be performed by one or more programmable processors running one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, and the apparatus can also be implemented as special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits).

[0422] For example, processors suitable for executing computer programs include general and special microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operably coupled to receive data from one or more large-capacity storage devices or to transmit data to them or both. However, a computer does not require such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, for example, including semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices. The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

[0423] The specification and drawings are to be regarded as exemplary only, where exemplary means example.As used herein, the use of "or" is intended to include "and / or" unless the context clearly dictates otherwise.

[0424] Although this patent document contains many details, these should not be interpreted as limitations on any invention or the scope of the claimed protection, but rather as descriptions of features unique to specific embodiments of specific inventions. Certain features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although the above-mentioned features may be described as working in certain combinations and even initially claimed to be protected in this way, in some cases, one or more features in the combination may be deleted from the claimed combination, and the claimed combination may be directed to a sub-combination or a variation of a sub-combination.

[0425] Similarly, although operations are described in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in sequence, or that all of the illustrated operations be performed, in order to achieve the desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0426] Only a few implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A method for video processing, comprising: for a first conversion between a current block of a video and a bitstream of the video, making a determination as to whether to enable a decoder-side motion vector derivation (DMVD) tool for the current block, wherein the DMVD tool derives refinement of motion information, and wherein the DMVD tool comprises at least one of a bidirectional optical flow algorithm or a decoder-side motion vector refinement algorithm; Based on the determination, performing the first conversion between the current block and the bitstream; and Disabling the DMVD tool for the current block, wherein the coding mode of the current block is a generalized bidirectional prediction (GBI) mode, and the height or width of the current block is less than a predetermined value; The GBI mode indicates that different weights at the coding unit level are applied to two prediction blocks generated from reference picture list 0 and reference picture list 1 to derive a final prediction block of the current block.

2. The method according to claim 1, wherein: The first conversion generates the current block from the bitstream.

3. The method according to claim 1, wherein: The first conversion generates the bitstream from the current block.

4. The method according to claim 1, wherein: The DMVD tool also includes a frame rate up-conversion algorithm.

5. The method according to claim 1, wherein: Whether to enable the DMVD tool is also based on the motion information of the current block.

6. The method according to claim 5, wherein: The DMVD tool is enabled when it is determined that (1) the absolute value of the motion vector, or (2) the absolute value of the motion vector difference, or (3) the absolute value of the motion vector of the current block is (a) less than a threshold value TH1, where TH1≥0, or (b) greater than a threshold value TH2, where TH2≥0.

7. The method according to claim 5, wherein: The DMVD tool is enabled when it is determined that the absolute value of the decoded motion vector of the current block in the horizontal or vertical prediction direction is (a) less than a threshold THh1 or THv1, respectively, where THh1≥0 and THv1≥0, or (b) greater than a threshold THh2 or THv2, respectively, where THh2≥0 and THv2≥0.

8. The method according to claim 5, wherein: The DMVD tool is enabled when a function determining the absolute value of the decoded motion vector MV of the current block in the horizontal prediction direction (MV_x) and the vertical prediction direction (MV_y) is (a) less than a threshold TH1, where TH1≥0, or (b) greater than a threshold TH2, where TH2≥0.

9. The method according to claim 8, wherein: The function is one of: f(abs(MV)) = min(abs(MV_x), abs(MV_y)), or f(abs(MV)) = max(abs(MV_x), abs(MV_y)), or f(abs(MV)) = (abs(MV_x) + abs(MV_y)), or f(abs(MV)) = (abs(MV_x) + abs(MV_y)) / 2.

10. The method according to claim 5, wherein: The current block is encoded using a bi-prediction mode, and wherein the decision is further based on motion information from only one reference picture list associated with the current block.

11. The method according to claim 5, wherein: The current block is encoded using a bi-prediction mode, and wherein the decision is further based on motion information from two reference picture lists associated with the current block.

12. The method according to claim 5, wherein: When it is determined that the current block is not encoded using a specific encoding mode, the DMVD tool is enabled, wherein the specific encoding mode includes a final motion vector expression (UMVE) mode or an advanced motion vector prediction (AMVP) mode.

13. The method according to claim 5, wherein: When it is determined that the current block is not encoded using a specific coding mode, the DMVD tool is enabled, wherein the specific coding mode does not include a bidirectional prediction mode and includes a condition that a distance between two motion vectors of each reference picture list of the current block is greater than or less than a predetermined value.

14. The method according to claim 1, further comprising: for a second conversion between a current block of video and a bitstream of the video, making a decision regarding selective enabling of a decoder-side motion vector derivation (DMVD) tool for the current block, wherein the DMVD tool derives a refinement of motion information signaled in the bitstream, and wherein the second conversion uses a Merge mode and motion vector differences indicated by a motion direction and a motion magnitude; and A second conversion between the current block and the bitstream is performed based on the determination.

15. The method according to claim 14, wherein: Enable the DMVD tool.

16. The method according to claim 14, wherein: Disable the DMVD tool.

17. The method according to claim 14, wherein: When it is determined that the current block is encoded and decoded using the Merge mode, the DMVD tool is disabled.

18. The method according to claim 14, wherein: The Merge mode further includes a start point of motion information indicated by a Merge index, and wherein final motion information of the current block is based on the motion vector difference and the start point.

19. The method according to claim 14, wherein: When it is determined that the Merge mode includes a non-zero motion vector difference (MVD) component, the DMVD tool is disabled.

20. The method according to any one of claims 14 to 19, wherein: The second conversion generates the current block from the bitstream.

21. The method according to any one of claims 14 to 19, wherein: The second conversion generates the bitstream from the current block.

22. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: For a first conversion between a current block of a video and a bitstream of the video, a decision is made as to whether to enable a decoder-side motion vector derivation (DMVD) tool for the current block, wherein: The DMVD tool derives a refinement of motion information, and wherein the DMVD tool comprises at least one of a bidirectional optical flow algorithm or a decoder-side motion vector refinement algorithm; Based on the determination, performing the first conversion between the current block and the bitstream; as well as Disabling the DMVD tool for the current block, wherein the coding mode of the current block is a generalized bidirectional prediction (GBI) mode, and the height or width of the current block is less than a predetermined value; The GBI mode indicates that different weights at the coding unit level are applied to two prediction blocks generated from reference picture list 0 and reference picture list 1 to derive a final prediction block of the current block.

23. A non-transitory computer-readable storage medium storing instructions that cause a processor to: For a first conversion between a current block of a video and a bitstream of the video, a decision is made as to whether to enable a decoder-side motion vector derivation (DMVD) tool for the current block, wherein: The DMVD tool derives a refinement of motion information, and wherein the DMVD tool comprises at least one of a bidirectional optical flow algorithm or a decoder-side motion vector refinement algorithm; Based on the determination, performing the first conversion between the current block and the bitstream; as well as Disabling the DMVD tool for the current block, wherein the coding mode of the current block is a generalized bidirectional prediction (GBI) mode, and the height or width of the current block is less than a predetermined value; The GBI mode indicates that different weights at the coding unit level are applied to two prediction blocks generated from reference picture list 0 and reference picture list 1 to derive a final prediction block of the current block.

24. A method for storing a bit stream of a video, comprising: making, for a current block of the video, a determination as to whether to enable a decoder-side motion vector derivation (DMVD) tool for the current block, wherein the DMVD tool derives refinement of motion information, and wherein the DMVD tool comprises at least one of a bidirectional optical flow algorithm or a decoder-side motion vector refinement algorithm; Based on the determination, generating the bitstream; and Disabling the DMVD tool for the current block, wherein the coding mode of the current block is a generalized bidirectional prediction (GBI) mode, and the height or width of the current block is less than a predetermined value; wherein the GBI mode indicates that different weights at the codec level are applied to two prediction blocks generated from reference picture list 0 and reference picture list 1 to derive a final prediction block of the current block; The bitstream is stored in a non-transitory computer-readable storage medium.

25. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method of any one of claims 11 to 21.

26. A non-transitory computer readable medium storing instructions, which, when executed by a processor, cause the processor to perform the method of any one of claims 11 to 21.

Images