Decoder-Side Motion Vector Derivation Based on a Reference Picture

Abstract

Devices, systems, and methods for digital video encoding and decoding are described, including a decoder-side motion vector derivation (DMVD) tool. An exemplary method for video processing includes: making a decision regarding selective enabling of the DMVD tool for a current block of video based on a determination that the current block of video is encoded and decoded using a multi-hypothesis prediction mode, where the DMVD tool derives a refinement of motion information signaled in a bitstream representation of the video; and performing a conversion between the current block and the bitstream representation based on the decision.

Description

[0001] Cross-reference to Related Applications

[0002] In accordance with applicable patent laws, this application is intended to claim the priority and benefit of International Patent Application PCT / CN2018 / 111224 filed on October 22, 2018, in a timely manner. The entire disclosure of the above application is incorporated by reference as part of the disclosure of this application. Technical Field

[0003] This patent document relates to video coding and decoding technologies, devices, and systems. Background Art

[0004] Despite the progress of video compression technologies, digital video still accounts for the largest bandwidth usage on the Internet and other digital communication networks. With the increasing number of networked user devices capable of receiving and displaying video, it is expected that the bandwidth demand for digital video usage will continue to grow. Summary of the Invention

[0005] Devices, systems, and methods related to digital video coding and decoding are described, particularly devices, systems, and methods related to the Decoder-side Motion Vector Derivation (DMVD) tool. The described methods can be applied to existing video coding and decoding standards (e.g., High Efficiency Video Coding (HEVC)) and future video coding and decoding standards or video codecs.

[0006] In one representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: making a decision on the selective enabling of a decoder-side motion vector derivation (DMVD) tool for a current block of a video based on a determination that the current block of the video is coded using a multi-hypothesis prediction mode, where the DMVD tool derives a refinement of motion information signaled in the bitstream representation of the video; and performing a conversion between the current block and the bitstream representation based on the decision, where the multi-hypothesis prediction mode is configured to generate a final prediction of the current block by applying at least one intermediate prediction value.

[0007] In another representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: determining that a current block of a video is associated with an asymmetric weight factor with respect to different reference blocks; enabling a decoder-side motion vector derivation (DMVD) tool for the current block, where the DMVD tool derives a refinement of motion information signaled in the bitstream representation of the video, and where the DMVD process is based on the asymmetric weight factor; and performing a conversion between the current block and the bitstream representation based on the enabling.

[0008] In yet another representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: determining that a current block of a video is encoded and decoded using an Advanced Motion Vector Prediction (AMVP) mode; and applying a Decoder-side Motion Vector Derivation (DMVD) tool to the current block as part of a conversion between a bitstream representation of the video and the current block, where the DMVD tool derives a refinement of motion information signaled in the bitstream representation.

[0009] In yet another representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: performing a refinement of translational motion parameters and a motion vector difference of a current block of a video encoded and decoded using a bi-affine mode or a bi-affine Merge mode, based on a Decoder-side Motion Vector Derivation (DMVD) tool, where the motion vector difference is indicated by a motion direction and a motion amplitude, and where the DMVD tool derives a refinement of motion information signaled in a bitstream representation of the video; and performing a conversion between the current block and the bitstream representation of the video based on the refinement.

[0010] In yet another representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: making a decision on a selective enabling of a Decoder-side Motion Vector Derivation (DMVD) tool for a current block of a video, based on characteristics of the current block of the video, where the DMVD tool derives a refinement of motion information signaled in a bitstream representation of the video; and performing a conversion between the current block and the bitstream representation based on the decision.

[0011] In yet another representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: making a decision on a selective enabling of a Decoder-side Motion Vector Derivation (DMVD) tool at a sub-block level, based on a determination that a current block of a video includes a plurality of sub-blocks, where the DMVD tool derives a refinement of motion information signaled in a bitstream representation of the video for each sub-block; and performing a conversion between the current block and the bitstream representation based on the decision.

[0012] In yet another representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: making a decision on a selective enabling of a Decoder-side Motion Vector Derivation (DMVD) tool for a current block of a video, based on at least one reference picture associated with the current block of the video, where the DMVD tool derives a refinement of motion information signaled in a bitstream representation of the video; and performing a conversion between the current block and the bitstream representation based on the decision.

[0013] In yet another representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: parsing a binary value string (bin string) from a bitstream representation of a current block of a video, where the binary value string includes a plurality of binary values representing a generalized bi-prediction (GBI) index of a GBI mode, and where at least one of the plurality of binary values is bypass-coded; and performing a conversion between the current block and the bitstream representation based on the parsed GBI index.

[0014] In yet another representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: encoding a binary value string into a bitstream representation of a current block of a video, where the binary value string includes a plurality of binary values representing a generalized bi-prediction (GBI) index of a GBI mode, and where at least one of the plurality of binary values is bypass-coded; and performing a conversion between the current block and the bitstream representation based on the encoded binary value string.

[0015] In yet another representative aspect, the above method is embodied in the form of processor-executable code and stored in a computer-readable program medium.

[0016] In yet another representative aspect, a device configured or operable to perform the above method is disclosed. The device may include a processor programmed to implement the method.

[0017] In yet another representative aspect, a video decoder device may implement the method described herein.

[0018] The above and other aspects and features of the disclosed technology are described in more detail in the drawings, the specification, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 An example of constructing a Merge candidate list is shown.

[0020] Figure 2 An example of the position of a spatial candidate is shown.

[0021] Figure 3 An example of a candidate pair that undergoes a redundancy check of a spatial Merge candidate is shown.

[0022] Figure 4A and Figure 4B An example of the position of a second prediction unit (PU) based on the size and shape of the current block is shown.

[0023] Figure 5Shows an example of motion vector scaling of temporal Merge candidates.

[0024] Figure 6 Shows an example of candidate positions of temporal Merge candidates.

[0025] Figure 7 Shows an example of generating combined bi - directional prediction Merge candidates.

[0026] Figure 8 Shows an example of constructing motion vector prediction candidates.

[0027] Figure 9 Shows an example of motion vector scaling of spatial motion vector candidates.

[0028] Figure 10 Shows an example of motion prediction using the Alternative Temporal Motion Vector Prediction (ATMVP) algorithm for a Coding Unit (CU).

[0029] Figure 11 Shows an example of a Coding Unit (CU) with sub - blocks and neighboring blocks used by the Spatial - Temporal Motion Vector Prediction (STMVP) algorithm.

[0030] Figure 12A and Figure 12B Shows an example snapshot of sub - blocks when using the Overlapped Block Motion Compensation (OBMC) algorithm.

[0031] Figure 13 Shows an example of neighboring samples for deriving parameters of the Local Illumination Compensation (LIC) algorithm.

[0032] Figure 14 Shows an example of a simplified affine motion model.

[0033] Figure 15 Shows an example of the Motion Vector Field (MVF) for each sub - block.

[0034] Figure 16 Shows an example of Motion Vector Prediction (MVP) for the AF_INTER affine motion mode.

[0035] Figure 17A and Figure 17B shows an example candidate of the AF_MERGE affine motion mode.

[0036] Figure 18 shows an example of bilateral matching in the Pattern Matched Motion Vector Derivation (PMMVD) mode, which is a special Merge mode based on the Frame-Rate Up Conversion (FRUC) algorithm.

[0037] Figure 19 shows an example of template matching in the FRUC algorithm.

[0038] Figure 20 shows an example of unilateral motion estimation in the FRUC algorithm.

[0039] Figure 21 shows an example of the search process of the Ultimate Motion Vector Expression (UMVE) for the current frame.

[0040] Figure 22A and Figure 22B shows an example of UMVE search points.

[0041] Figure 23 shows an exemplary mapping between the distance index and the distance offset.

[0042] Figure 24 shows an example of the optical flow trajectory used by the Bi-directional Optical flow (BIO) algorithm.

[0043] Figure 25A and Figure 25B shows an example instant image using the Bi-directional Optical flow (BIO) algorithm without block expansion.

[0044] Figure 26 shows an example of the Decoder-side Motion Vector Refinement (DMVR) algorithm based on bilateral template matching.

[0045] Figure 27A - Figure 27I shows a flowchart of an example method for video processing.

[0046] Figure 28It is a block diagram of an example of a hardware platform for implementing the visual media decoding or visual media encoding techniques described in this document.

[0047] Figure 29 It is a block diagram of an example video processing system that can implement the disclosed technology. Detailed implementation

[0048] Due to the increasing demand for high-resolution videos, video coding and decoding methods and techniques are widespread in modern technologies. Video codecs typically include electronic circuits or software for compressing or decompressing digital videos and are constantly being improved to provide higher coding and decoding efficiency. A video codec converts uncompressed video into a compressed format and vice versa. There is a complex relationship among video quality, the amount of data used to represent the video (determined by the bit rate), the complexity of the encoding and decoding algorithms, the sensitivity to data loss and errors, ease of editing, random access, and end-to-end latency (delay). Compressed formats generally conform to standard video compression specifications, such as the High Efficiency Video Coding (HEVC) standard (also known as H.265 or MPEG-H Part 2), the Versatile Video Coding (VVC) standard to be finalized, or other current and / or future video coding and decoding standards.

[0049] Embodiments of the disclosed technology can be applied to existing video coding and decoding standards (e.g., HEVC, H.265) and future standards to improve compression performance. In this document, section headings are used to enhance the readability of the description and do not limit the discussion or embodiments (and / or implementations) to the respective sections in any way.

[0050] 1. Examples of HEVC / H.265 Inter-Frame Prediction

[0051] Over the years, video coding and decoding standards have been significantly improved, and now some provide high coding and decoding efficiency and support for higher resolutions. Recent standards such as HEVC and H.265 are based on a hybrid video coding structure that utilizes temporal prediction plus transform coding and decoding.

[0052] 1.1 Examples of Prediction Modes

[0053] Each inter-frame PU (prediction unit) has motion parameters for one or two reference picture lists. In some embodiments, the motion parameters include a motion vector and a reference picture index. In other embodiments, the use of one of the two reference picture lists can also be signaled using inter_pred_idc. In still other embodiments, the motion vector can be explicitly encoded as an increment relative to a predicted value.

[0054] When a CU is encoded in skip mode, one PU is associated with the CU and there are no significant residual coefficients, no coded motion vector differences or reference picture indices. The Merge mode is specified, from which the motion parameters of the current PU are obtained from neighboring PUs, including spatial and temporal candidates. The Merge mode is applied not only to skip mode but also to any inter-predicted PU. An alternative to the Merge mode is the explicit transmission of motion parameters, where the motion vector, the corresponding reference picture index for each reference picture list, and the use of the reference picture list are signaled explicitly for each PU.

[0055] When the signaling indicates to use one of the two reference picture lists, a PU is generated from a sample block. This is called "unidirectional prediction". Unidirectional prediction can be used for P slices and B slices.

[0056] When the signaling indicates to use both of the two reference picture lists, a PU is generated from two sample blocks. This is called "bidirectional prediction". Bidirectional prediction can only be used for B slices.

[0057] 1.1.1 Example of constructing Merge mode candidates

[0058] When predicting a PU using the Merge mode, an index pointing to an entry in the Merge candidate list is parsed from the bitstream and used to retrieve the motion information. The construction of this list can be outlined according to the following series of steps:

[0059] · Step 1: Initial candidate derivation

[0060] o Step 1.1: Spatial candidate derivation

[0061] o Step 1.2: Redundancy check of spatial candidates

[0062] o Step 1.3: Temporal candidate derivation

[0063] · Step 2: Additional candidate insertion

[0064] o Step 2.1: Create bidirectional prediction candidates

[0065] o Step 2.2: Insert zero motion candidates

[0066] Figure 1An example of constructing a Merge candidate list based on the above series of steps is shown. For spatial-domain Merge candidate derivation, up to four Merge candidates are selected among the candidates located at five different positions. For temporal-domain Merge candidate derivation, up to one Merge candidate is selected among two candidates. Since a constant number of candidates is assumed for each PU at the decoder, additional candidates are generated when the number of candidates does not reach the maximum number of Merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, the index of the best Merge candidate is encoded using Truncated Unary binarization (TU). If the size of the CU is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

[0067] 1.1.2 Constructing Spatial-Domain Merge Candidates

[0068] In the derivation of spatial-domain Merge candidates, up to four Merge candidates are selected among the candidates located at the positions depicted in Figure 2 . The order of derivation is A1, B1, B0, A0, and B2. Position B2 is considered only if any of the PUs at positions A1, B1, B0, A0 is unavailable (e.g., because it belongs to another strip or slice) or is intra-coded. After adding the candidate at position A1, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving the coding efficiency.

[0069] To reduce the computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only the pairs linked by the arrows in Figure 3 are considered, and a candidate is added to the list only if the corresponding candidate for the redundancy check does not have the same motion information. Another source of duplicate motion information is the "second PU" associated with a partition different from 2N×2N. As an example, Figure 4A and Figure 4B depict the second PU in the cases of N×2N and 2N×N, respectively. When the current PU is partitioned into N×2N, the candidate at position A1 is not considered for list construction. In some embodiments, adding this candidate may result in two prediction units having the same motion information, which is redundant for a coding unit having only one PU. Similarly, when the current PU is partitioned into 2N×N, position B1 is not considered.

[0070] 1.1.3 Constructing Temporal-Domain Merge Candidates

[0071] In this step, only one candidate is added to the list. Specifically, in the derivation of the temporal Merge candidate, a scaled motion vector is derived based on the co-located PUs belonging to the picture with the smallest POC (Picture Order Count) difference from the current picture within the given reference picture list. The reference picture list to be used for deriving the co-located PUs is signaled explicitly in the slice header.

[0072] Figure 5 An example of the derivation of the scaled motion vector (as the dashed line) for the temporal Merge candidate is shown, which is scaled from the motion vector of the co-located PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal Merge candidate is set to be equal to zero. For B slices, two motion vectors (one for reference picture list 0 and the other for reference picture list 1) are obtained and combined to generate the bi-predictive Merge candidate.

[0073] In the co-located PU (Y) belonging to the reference frame, the position of the temporal candidate is selected between candidates C0 and C1, as Figure 6 depicted. If the PU at position C0 is unavailable, intra-coded, or outside the current CTU, then position C1 is used. Otherwise, position C0 is used in the derivation of the temporal Merge candidate.

[0074] 1.1.4 Construction of additional types of Merge candidates

[0075] In addition to the spatio-temporal Merge candidates, there are two additional types of Merge candidates: combined bi-predictive Merge candidates and zero Merge candidates. The combined bi-predictive Merge candidates are generated by leveraging the spatio-temporal Merge candidates. The combined bi-predictive Merge candidates are only used for B slices. The combined bi-predictive candidate is generated by combining the motion parameters of the first reference picture list of an initial candidate with the motion parameters of the second reference picture list of another initial candidate. If the two tuples provide different motion hypotheses, they will form a new bi-predictive candidate.

[0076] Figure 7 An example of the process is shown, where two candidates (which have mvL0 and refIdxL0 or mvL1 and refIdxL1) in the original list (710, on the left) are used to create the combined bi-predictive Merge candidate that is added to the final list (720, on the right).

[0077] Insert zero motion candidates to fill the remaining entries in the Merge candidate list and thus reach the MaxNumMergeCand capacity. These candidates have zero spatial displacement and a reference picture index that starts from zero and increases each time a new zero motion candidate is added to the list. The number of reference frames used by these candidates is 1 and 2 for uni - directional prediction and bi - directional prediction, respectively. In some embodiments, no redundancy check is performed on these candidates.

[0078] 1.1.5 Examples of Motion Estimation Regions for Parallel Processing

[0079] To accelerate the encoding process, motion estimation can be performed in parallel, thereby simultaneously deriving the motion vectors of all prediction units within a given region. Deriving Merge candidates from the spatial neighborhood may interfere with parallel processing because a prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is complete. To mitigate the trade - off between codec efficiency and processing latency, a Motion Estimation Region (MER) can be defined. The size of the MER is signaled in the Picture Parameter Set (PPS) using the "log2_parallel_merge_level_minus2" syntax element. When the MER is defined, Merge candidates that fall into the same region are marked as unavailable and are thus not considered in list construction.

[0080] 1.2 Embodiments of Advanced Motion Vector Prediction (AMVP)

[0081] AMVP exploits the spatio - temporal correlation of motion vectors with neighboring PUs, which is used for explicit transmission of motion parameters. A list of motion vector candidates is constructed by first checking the availability of temporally neighboring PU positions to the left and above, removing redundant candidates, and adding zero vectors to make the candidate list of a constant length. Then, the encoder can select the best prediction value from the candidate list and send the corresponding index indicating the selected candidate. Similar to Merge index signaling, the index of the best motion vector candidate is encoded using truncated unary. In this case, the maximum value to be encoded is 2 (see Figure 8 ). In the following sections, details of the derivation process of motion vector prediction candidates will be provided.

[0082] 1.2.1 Examples of Constructing Motion Vector Prediction Candidates

[0083] Figure 8 Outlines the derivation process of motion vector prediction candidates and can be implemented for each reference picture list with refidx as the input.

[0084] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For spatial motion vector candidate derivation, two motion vector candidates are finally derived based on the motion vectors of each PU located at five different positions as shown previously Figure 2 as shown.

[0085] For temporal motion vector candidate derivation, one motion vector candidate is selected from two candidates derived based on two different collocated positions. After generating the first spatio-temporal candidate list, duplicate motion vector candidates in the list are removed. If the number of potential candidates is greater than two, motion vector candidates with reference picture indices greater than 1 in the associated reference picture list are removed from the list. If the number of spatio-temporal motion vector candidates is less than two, additional zero motion vector candidates are added to the list.

[0086] 1.2.2 Construction of Spatial Motion Vector Candidates

[0087] In the derivation of spatial motion vector candidates, at most two candidates are considered among the five potential candidates derived from PUs located at the positions as shown previously Figure 2 as shown, which are the same positions as those for motion Merge. The derivation order for the left side of the current PU is defined as A0, A1, and scaled A0, scaled A1. The derivation order for the upper side of the current PU is defined as B0, B1, B2, scaled B0, scaled B1, scaled B2. Thus, for each side, there are four cases that can be used as motion vector candidates, where two cases do not require the use of spatial scaling and two cases use spatial scaling. The four different cases are outlined as follows:

[0088] · Without spatial scaling

[0089] (1) The same reference picture list and the same reference picture index (the same POC)

[0090] (2) Different reference picture lists, but the same reference picture (the same POC)

[0091] · With spatial scaling

[0092] (3) The same reference picture list, but different reference pictures (different POCs)

[0093] (4) Different reference picture lists and different reference pictures (different POCs)

[0094] First check the non-spatial scaling case, then the case where spatial scaling is allowed. Spatial scaling is considered when the POC is different between the reference pictures of the neighboring PU and the reference pictures of the current PU regardless of the reference picture list. If all PUs of the left candidate are unavailable or are intra-coded, then scaling for the upper motion vector is allowed to help parallel derivation of the left and upper MV candidates. Otherwise, spatial scaling is not allowed for the upper motion vector.

[0095] like Figure 9 As shown in the example in , for the case of spatial scaling, the motion vectors of neighboring PUs are scaled in a similar way to temporal scaling. One difference is that the reference picture list and the index of the current PU are given as input; the actual scaling process is the same as that of temporal scaling.

[0096] 1.2.3 Constructing temporal motion vector candidates

[0097] Except for the reference picture index derivation, all the processes used to derive the temporal Merge candidate are the same as those used to derive the spatial motion vector candidate (e.g. Figure 6 In some embodiments, the reference picture index is signaled to the decoder.

[0098] 2. Example of Inter Prediction Method in Joint Exploration Model (JEM)

[0099] In some embodiments, a reference software called Joint Exploration Model (JEM) is used to explore future video codec technologies. In JEM, sub-block based prediction is adopted in several codec tools, such as affine prediction, optional temporal motion vector prediction (ATMVP), spatiotemporal motion vector prediction (STMVP), bidirectional optical flow (BIO), frame rate up conversion (FRUC), locally adaptive motion vector resolution (LAMVR), overlapped block motion compensation (OBMC), local illumination compensation (LIC), and decoder side motion vector refinement (DMVR).

[0100] 2.1 Example of motion vector prediction based on sub-CU

[0101] In the Joint Exploration Model (JEM) with quadtrees plus binary trees (QTBT), each coding unit (CU) can have at most one set of motion parameters for each prediction direction. In some embodiments, two sub-CU level motion vector prediction methods are considered in the encoder by partitioning a large CU into sub-CUs and deriving the motion information of all sub-CUs of the large CU. The Adaptive Temporal Motion Vector Prediction (ATMVP) method allows each CU to extract multiple sets of motion information from multiple blocks smaller than the current CU in the collocated reference picture. In the Spatial-Temporal Motion Vector Prediction (STMVP) method, the motion vectors of sub-CUs are recursively derived by using the temporal motion vector predictors and the spatial neighboring motion vectors. In some embodiments, and in order to maintain a more accurate motion field for sub-CU motion prediction, the motion compression of reference frames can be disabled.

[0102] 2.1.1 Example of Adaptive Temporal Motion Vector Prediction (ATMVP)

[0103] In the ATMVP method, the Temporal Motion Vector Prediction (TMVP) method is modified by extracting multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU.

[0104] Figure 10 An example of the ATMVP motion prediction process for CU 1000 is shown. The ATMVP method predicts the motion vectors of sub-CU 1001 within CU 1000 in two steps. The first step is to identify the corresponding block 1051 in the reference picture 1050 using the temporal vector. The reference picture 1050 is also referred to as the motion source picture. The second step is to partition the current CU 1000 into sub-CUs 1001 and obtain the motion vectors and the reference indices of each sub-CU from the blocks corresponding to each sub-CU.

[0105] In the first step, the reference picture 1050 and the corresponding block are determined by the motion information of the spatial neighboring blocks of the current CU 1000. To avoid the repeated scanning process of neighboring blocks, the first Merge candidate in the Merge candidate list of the current CU 1000 is used. The first available motion vector and its associated reference index are set as the temporal vector and the index of the motion source picture. In this way, compared with TMVP, the corresponding block can be more accurately identified, where the corresponding block (sometimes called the collocated block) is always in the lower right or central position relative to the current CU.

[0106] In the second step, the corresponding block of sub-CU 1051 is identified by adding a temporal vector to the coordinates of the current CU and using the temporal vector in the motion source picture 1050. For each sub-CU, the motion information of its corresponding block (e.g., the smallest motion grid covering the central sample) is used to derive the motion information of the sub-CU. After identifying the motion information of the corresponding N×N block, it is converted into the motion vector and reference index of the current sub-CU in the same way as TMVP in HEVC, where motion scaling and other processes apply. For example, the decoder checks whether the low-latency condition is satisfied (e.g., the POC of all reference pictures of the current picture is less than the POC of the current picture), and may use the motion vector MV x (e.g., the motion vector corresponding to reference picture list X) to predict the motion vector MV of each sub-CU y (e.g., where X is equal to 0 or 1, and Y is equal to 1 - X).

[0107] 2.1.2 Example of Spatio-Temporal Motion Vector Prediction (STMVP)

[0108] In the STMVP method, the motion vectors of sub-CUs are derived recursively in raster scan order. Figure 11 Fig. shows a CU with 4 sub-blocks and neighboring blocks. Consider an 8×8 CU 1100, which includes 4 4×4 sub-CUs: A (1101), B (1102), C (1103), and D (1104). The neighboring 4×4 blocks in the current frame are labeled as a (1111), b (1112), c (1113), and d (1114).

[0109] The motion derivation of sub-CU A starts by identifying its two spatial neighbors. The first neighbor is the N×N block (block c 1113) above sub-CU A 1101. If this block c 1113 is not available or is intra-coded, then other N×N blocks above sub-CU A 1101 are checked (from left to right, starting from block c 1113). The second neighbor is the block (block b 1112) to the left of sub-CU A 1101. If block b 1112 is not available or is intra-coded, then other blocks to the left of sub-CU A 1101 are checked (from top to bottom, starting from block b 1112). The motion information obtained from the neighboring blocks in each list is scaled to the first reference frame of the given list. Next, the temporal motion vector predictor (TMVP) of sub-block A 1101 is derived by following the same process as the TMVP derivation specified in HEVC. The motion information of the collocated block at block D 1104 is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors are averaged separately for each reference list. The averaged motion vector is assigned as the motion vector of the current sub-CU.

[0110] 2.1.3 Examples of Sub - CU Motion Prediction Mode Signaling

[0111] In some embodiments, the sub - CU modes are enabled as additional Merge candidates, and no additional syntax elements are required to signal these modes. Two additional Merge candidates are added to the Merge candidate list of each CU to represent the ATMVP mode and the STMVP mode. In other embodiments, if the sequence parameter set indicates that ATMVP and STMVP are enabled, up to seven Merge candidates can be used. The coding logic for the additional Merge candidates is the same as that for the Merge candidates in HM, which means that for each CU in a P - slice or B - slice, two additional RD checks may be required for the two additional Merge candidates. In some embodiments, for example, in JEM, all binary values of the Merge index are context - encoded by CABAC (Context - based Adaptive Binary Arithmetic Coding). In other embodiments, for example, in HEVC, only the first binary value is context - encoded, and the remaining binary values are context - bypass - encoded.

[0112] 2.2 Examples of Adaptive Motion Vector Difference Resolution

[0113] In some embodiments, when use_integer_mv_flag in the slice header is equal to 0, the Motion Vector Difference (MVD) between the motion vector of the PU and the predicted motion vector is signaled in quarter - luminance samples. In JEM, local adaptive motion vector resolution (LAMVR) is introduced. In JEM, the MVD can be encoded and decoded in quarter - luminance samples, integer - luminance samples, or four - luminance samples. The MVD resolution is controlled at the coding unit (CU) level, and for each CU with at least one non - zero MVD component, an MVD resolution flag is signaled conditionally.

[0114] For a CU with at least one non - zero MVD component, a first flag is signaled to indicate whether quarter - luminance sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter - luminance sample MV precision is not used, another flag is signaled to indicate whether integer - luminance sample MV precision or four - luminance sample MV precision is used.

[0115] When the first MVD resolution flag of the CU is zero or the CU is not coded (meaning all MVDs in the CU are zero), the quarter-luma sample MV resolution is used for the CU. When the CU uses integer-luma sample MV precision or quad-luma sample MV precision, the MVP in the AMVP candidate list of the CU is rounded to the corresponding precision.

[0116] In the encoder, CU-level RD checks are used to determine which MVD resolution is to be used for the CU. That is, for each MVD resolution, three CU-level RD checks are performed. To speed up the encoder, the following coding scheme is applied in JEM:

[0117] -- During the RD check of a CU with normal quarter-luma sample MVD resolution, the motion information of the current CU (integer-luma sample accuracy) is stored. The stored motion information (after rounding) is used as a starting point for further small-range motion vector refinement during the RD check for the same CU with integer-luma sample and 4-luma sample MVD resolutions, so that the time-consuming motion estimation process is not repeated three times.

[0118] -- Conditionally call the RD check for a CU with 4-luma sample MVD resolution. For a CU, when the RD cost of the integer-luma sample MVD resolution is much larger than the RD cost of the quarter-luma sample MVD resolution, skip the RD check for the 4-luma sample MVD resolution of the CU.

[0119] 2.3 Examples of Higher Motion Vector Storage Precision

[0120] In HEVC, the motion vector precision is one-quarter pixel (one-quarter luma sample and one-eighth chroma sample for 4:2:0 video). In JEM, the precision of the internal motion vector storage and Merge candidates is increased to 1 / 16 pixel. The higher motion vector precision (1 / 16 pixel) is used for motion-compensated inter prediction of CUs coded in skip / Merge mode. For CUs coded in normal AMVP mode, integer pixel or one-quarter pixel motion is used.

[0121] The SHVC upsampling interpolation filter with the same filter length and normalization factor as the HEVC motion compensation interpolation filter is used as the motion compensation interpolation filter for additional fractional pixel positions. In JEM, the chroma component motion vector precision is 1 / 32 sample, and an additional interpolation filter for 1 / 32 pixel fractional positions is derived by using the average of two neighboring 1 / 16 pixel fractional position filters.

[0122] 2.4 Examples of Overlapped Block Motion Compensation (OBMC)

[0123] In JEM, overlapped block motion compensation (OBMC) has been previously used in H.263. In JEM, different from H.263, CU-level syntax can be used to turn OBMC on and off. When OBMC is used in JEM, OBMC is performed for all motion compensation (MC) block boundaries except for the right and bottom boundaries of the CU. Additionally, it is applied to both the luminance and chrominance components. In JEM, an MC block corresponds to a coding / decoding block. When a CU is coded / decoded in a sub-CU mode (including sub-CU Merge, affine, and FRUC (Frame Rate Up Conversion) mode), each sub-block of the CU is an MC block. To handle the CU boundaries in a unified manner, OBMC is performed at the sub-block level for all MC block boundaries, where the sub-block size is set to be equal to 4×4, as Figure 12A - Figure 12B shown.

[0124] Figure 12A Fig. shows the sub-blocks at the CU / PU boundary, and the shaded sub-blocks are where OBMC is applied. Similarly,[[]] Figure 12B Fig. shows the sub-PU in the ATMVP mode.

[0125] When OBMC is applied to the current sub-block, in addition to the current motion vector, the motion vectors of four adjacent neighboring sub-blocks (if available and different from the current motion vector) are also used to derive the predicted block of the current sub-block. These multiple predicted blocks based on multiple motion vectors are combined to generate the final prediction signal of the current sub-block.

[0126] The predicted block based on the motion vector of a neighboring sub-block is denoted as PN, where N represents the index for the neighboring upper, lower, left, and right sub-blocks, and the predicted block based on the motion vector of the current sub-block is denoted as PC. When PN is based on the motion information that contains the same motion information as the current sub-block for the neighboring sub-block, OBMC is not performed from PN. Otherwise, each sample in PN is added to the same sample in PC, i.e., the four rows / columns of PN are added to PC. Weight factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} are used for PN, and weight factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are used for PC. The exception is small MC blocks (i.e., when the height or width of the coding / decoding block is equal to 4 or the CU is coded / decoded in a sub-CU mode), for such blocks, only two rows / columns of PN are added to PC. In this case, weight factors {1 / 4, 1 / 8} are used for PN, and weight factors {3 / 4, 7 / 8} are used for PC. For PN generated based on the motion vectors of vertical (horizontal) neighboring sub-blocks, the samples in the same row (column) of PN are added to PC with the same weight factor.

[0127] In JEM, for a CU with a size less than or equal to 256 luma samples, a CU-level flag is signaled to indicate whether OBMC is applied to the current CU. For a CU with a size greater than 256 luma samples or not coded / decoded in AMVP mode, OBMC is applied by default. At the encoder, when OBMC is applied to a CU, its effect is taken into account during the motion estimation stage. The motion information of the upper neighboring block and the left neighboring block is used to form a prediction signal by OBMC to compensate the upper boundary and the left boundary of the original signal of the current CU, and then the normal motion estimation process is applied.

[0128] 2.5 Examples of Local Illumination Compensation (LIC)

[0129] LIC is based on a linear model of illumination change, using a scaling factor a and an offset b. And it is adaptively enabled or disabled for each coded / decoded Coding Unit (CU) in an inter-frame mode.

[0130] When LIC is applied to a CU, the least square error method is adopted to derive the parameters a and b by using the neighboring samples of the current CU and their corresponding reference samples. Figure 13 Examples of neighboring samples for deriving the IC algorithm parameters are shown. Specifically, as Figure 13 shown, the subsampled (2:1 subsampling) neighboring samples of the CU in the reference picture and the corresponding samples (identified by the motion information of the current CU or sub-CU) are used. The IC parameters are derived separately and applied to each prediction direction.

[0131] When a CU is coded / decoded in Merge mode, the LIC flag is copied from the neighboring block in a manner similar to the copying of motion information in Merge mode; otherwise, the LIC flag is signaled for the CU to indicate whether LIC is applicable.

[0132] When LIC is enabled for a picture, additional CU-level RD checks are required to determine whether to apply LIC to the CU. When LIC is enabled for a CU, the Mean-Removed Sum of Absolute Difference (MR-SAD) and the Mean-Removed Sum of Absolute Hadamard-Transformed Difference (MR-SATD) (instead of SAD and SATD) are used for integer-pixel motion search and fractional-pixel motion search, respectively.

[0133] To reduce the coding complexity, the following coding scheme is applied in JEM:

[0134] --When there is no significant illumination change between the current picture and its reference pictures, LIC is disabled for the entire picture. To identify such a situation, histograms of the current picture and each reference picture of the current picture are calculated at the encoder. If the histogram difference between the current picture and each reference picture of the current picture is less than a given threshold, LIC is disabled for the current picture; otherwise, LIC is enabled for the current picture.

[0135] 2.6 Examples of Affine Motion Compensation Prediction

[0136] In HEVC, only the translational motion model is applied to Motion Compensation Prediction (MCP). However, cameras and objects can have many types of motion, such as zooming in / out, rotation, perspective motion, and / or other irregular motions. On the other hand, JEM applies simplified affine transform motion compensation prediction. Figure 14 An example of an affine motion field of block 1400 described by two control point motion vectors V0 and V1 is shown. The motion vector field (MVF) of block 1400 is described by the following equation:

[0137]

[0138] As Figure 14 shown, (v 0x ,v 0y ) is the motion vector of the upper left control point, and (v 1x ,v 1y ) is the motion vector of the upper right control point. To simplify motion compensation prediction, sub-block based affine transform prediction can be applied. The sub-block size M×N is derived as follows:

[0139]

[0140] Here, MvPre is the motion vector fractional precision (e.g., 1 / 16 in JEM), and (v 2x ,v 2y ) is the motion vector of the lower left control point calculated according to Equation (1). If necessary, M and N can be adjusted downward to be divisors of w and h respectively.

[0141] Figure 15An example of the affine MVF for each sub-block of block 1500 is shown. To derive the motion vector for each M×N sub-block, the motion vector of the central sample of each sub-block can be calculated according to Equation (1) and rounded to the motion vector fractional precision (e.g., 1 / 16 in JEM). Then, a motion compensation interpolation filter can be applied to generate the prediction for each sub-block with the derived motion vector. After MCP, the high-precision motion vectors of each sub-block are rounded and saved with the same precision as the normal motion vectors.

[0142] 2.6.1 Embodiments of the AF_INTER mode

[0143] In JEM, there are two affine motion modes: the AF_INTER mode and the AF_MERGE mode. For a CU with both width and height greater than 8, the AF_INTER mode can be applied. An affine flag at the CU level is signaled in the bitstream to indicate whether the AF_INTER mode is used. In the AF_INTER mode, neighboring blocks are used to construct a candidate list with motion vector pairs {(v0, v1)|v0 = {v A , v B , v c}}, v1 = {v D , v E}}.

[0144] Figure 16 An example of the motion vector prediction (MVP) of block 1600 in the AF_INTER mode is shown. As Figure 16As shown, v0 is selected from the motion vectors of sub-blocks A, B, or C. The motion vectors from neighboring blocks are scaled according to the reference list. The motion vector can also be scaled according to the relationship between the picture order count (POC) of the reference of the neighboring block, the POC of the reference of the current CU, and the POC of the current CU. The method of selecting v1 from neighboring sub-blocks D and E is similar. If the number of candidate lists is less than 2, the list is filled with motion vector pairs formed by copying each AMVP candidate. When the candidate list is greater than 2, the candidates can be sorted first according to the neighboring motion vectors (e.g., based on the similarity of the two motion vectors in a pair of candidates). In some embodiments, only the first two candidates are retained. In some embodiments, rate distortion (RD) cost checking is used to determine which motion vector pair candidate is selected as the control point motion vector prediction (CPMVP) of the current CU. An index indicating the position of the CPMVP in the candidate list can be signaled in the bitstream. After determining the CPMVP of the current affine CU, affine motion estimation is applied and the control point motion vector (CPMV) is found. Then, the difference between the CPMV and the CPMVP is signaled in the bitstream.

[0145] 2.6.3 Embodiments of the AF_MERGE mode

[0146] When applying a CU in the AF_MERGE mode, it obtains the first block encoded / decoded in the affine mode from valid neighboring reconstructed blocks. Figure 17A An example of the selection order of candidate blocks of the current CU 1700 is shown. As Figure 17A shown, the selection order can be from the left (1701), top (1702), top right (1703), bottom left (1704) to top left (1705) of the current CU 1700. Figure 17B Another example of the candidate blocks of the current CU 1700 in the AF_MERGE mode is shown. If the neighboring bottom left block 1801 is encoded / decoded in the affine mode, as Figure 17B shown, then the motion vectors v2, v3, and v4 of the upper left corner, upper right corner, and lower left corner of the CU containing sub-block 1701 are derived. The motion vector v0 of the upper left corner of the current CU 1700 is calculated based on v2, v3, and v4. The motion vector v1 of the upper right of the current CU can be calculated accordingly.

[0147] After calculating the CPMVs v0 and v1 of the current CU according to the affine motion model in Equation (1), the MVF of the current CU can be generated. To identify whether the current CU is coded in AF_MERGE mode, when at least one neighboring block is coded in affine mode, the affine flag can be signaled in the bitstream.

[0148] 2.7 Examples of motion vector derivation for pattern matching (PMMVD)

[0149] The PMMVD mode is a special Merge mode based on the frame rate up-conversion (FRUC) method. With this mode, the motion information of the block is not signaled, but derived at the decoder side.

[0150] When the Merge flag is true, the FRUC flag can be signaled for the CU. When the FRUC flag is false, the Merge index can be signaled and the regular Merge mode is used. When the FRUC flag is true, an additional FRUC mode flag can be signaled to indicate which method (e.g., bilateral matching or template matching) will be used to derive the motion information of the block.

[0151] On the encoder side, the decision on whether to use the FRUC Merge mode for the CU is based on the RD cost selection made for the normal Merge candidates. For example, by using RD cost selection to check multiple matching patterns (e.g., bilateral matching and template matching) of the CU. The matching pattern resulting in the minimum cost is further compared with other CU modes. If the FRUC matching pattern is the most efficient mode, the FRUC flag is set to true for the CU and the relevant matching pattern is used.

[0152] Generally, the motion derivation process in the FRUC Merge mode has two steps: first, perform motion search at the CU level, and then perform motion refinement at the sub-CU level. At the CU level, the initial motion vector of the entire CU is derived based on bilateral matching or template matching. First, an MV candidate list is generated, and the candidate resulting in the minimum matching cost is selected as the starting point for further CU refinement. Then, a local search based on bilateral matching or template matching around the starting point is performed. The MV resulting in the minimum matching cost is regarded as the MV of the entire CU. Subsequently, starting from the derived CU motion vector, the motion information is further refined at the sub-CU level.

[0153] For example, the following derivation process is performed for the motion information derivation of a W×H CU. In the first stage, the MV of the entire W×H CU is derived. In the second stage, the CU is further divided into M×M sub-CUs. The value of M is calculated as in Equation (3), and D is a predefined partition depth, which is default set to 3 in JEM.

[0154] Then, the motion vectors (MVs) of each sub-CU are derived.

[0155]

[0156] Figure 18 An example of bilateral matching used in a frame rate up-conversion (FRUC) method is shown. Bilateral matching is used to derive the motion information of a current coding unit (CU) by finding the closest match between two blocks along the motion trajectory of the current CU (1800) in two different reference pictures (1810, 1811). Under the assumption of a continuous motion trajectory, the motion vectors MV0 (1801) and MV1 (1802) pointing to two reference blocks are proportional to the temporal distances (e.g., TD0 (1803) and TD1 (1804)) between the current picture and the two reference pictures. In some embodiments, when the current picture 1800 is temporally located between the two reference pictures (1810, 1811) and the temporal distances from the current picture to the two reference pictures are the same, the bilateral matching becomes mirror-based bilateral MVs.

[0157] Figure 19 An example of template matching used in a frame rate up-conversion (FRUC) method is shown. Template matching can be used to derive the motion information of a current CU 1900 by finding the closest match between a template in the current picture (e.g., neighboring blocks at the top and / or left of the current CU) and a block in a reference picture 1910 (e.g., having the same size as the template). In addition to the above FRUC Merge mode, template matching can also be applied to the AMVP mode. In JEM and HEVC, AMVP has two candidates. Using the template matching method, a new candidate can be derived. If the newly derived candidate by template matching is different from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and then the list size is set to 2 (e.g., by removing the second existing AMVP candidate). When applied to the AMVP mode, only CU-level search is applied.

[0158] The set of MV candidates at the CU level can include the following: (1) the original AMVP candidates if the current CU is in the AMVP mode, (2) all Merge candidates, (3) several MVs in the interpolated MV field (described later), and the neighboring motion vectors at the top and left.

[0159] When bilateral matching is used, each valid MV of a Merge candidate can be used as an input to generate an MV pair assuming bilateral matching. For example, at reference list A, one valid MV of a Merge candidate is (MVa,ref a ). Then, the reference picture ref of its paired bilateral MV is found in another reference list B b such that refa and ref b are on different sides of the current picture in the time domain. If such ref b is not available in reference list B, then ref b is determined to be different from ref a and its time-domain distance to the current picture is the smallest in list B. After determining ref a , MVa is derived by scaling it based on the time-domain distance between the current picture and ref a , ref b to obtain MVb.

[0160] In some embodiments, four MVs from the interpolated MV field can also be added to the CU-level candidate list. More specifically, the interpolated MVs at the positions (0, 0), (W / 2, 0), (0, H / 2), and (W / 2, H / 2) of the current CU are added. When FRUC is applied to the AMVP mode, the original AMVP candidates are also added to the CU-level MV candidate set. In some embodiments, at the CU level, 15 MVs of the AMVP CU and 13 MVs of the Merge CU can be added to the candidate list.

[0161] The MV candidate set at the sub-CU level includes: (1) the MVs determined from the CU-level search, (2) the neighboring MVs at the top, left, top-left, and top-right, (3) the scaled versions of the collocated MVs from the reference pictures, (4) one or more ATMVP candidates (e.g., up to four), and (5) one or more STMVP candidates (e.g., up to four). The scaled MVs from the reference pictures are derived as follows. The reference pictures in both lists are traversed. The MVs at the collocated positions of the sub-CUs in the reference pictures are scaled to the reference of the starting CU-level MV. The ATMVP and STMVP candidates can be the top four. At the sub-CU level, one or more MVs (e.g., up to 17) are added to the candidate list.

[0162] Generation of interpolated MV field. Before encoding and decoding the frame, an interpolated motion field is generated for the entire picture based on unilateral ME. Then, the motion field can be used later as CU-level or sub-CU-level MV candidates.

[0163] In some embodiments, the motion fields of each reference picture in the two reference lists are traversed at the 4×4 block level. Figure 20An example of the unilateral motion estimation (ME) 2000 in the FRUC method is shown. For each 4×4 block, if the motion associated with the block passes through the 4×4 block in the current picture and the block has not been assigned any interpolated motion, the motion of the reference block is scaled to the current picture according to the temporal distances TD0 and TD1 (in the same way as the MV scaling of TMVP in HEVC), and the scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to the 4×4 block, the motion of the block is marked as unavailable in the interpolated motion field.

[0164] Interpolation and matching cost. When the motion vector points to a fractional sample position, motion compensation interpolation is required. To reduce complexity, bilinear interpolation can be used instead of the regular 8-tap HEVC interpolation for both bilateral matching and template matching.

[0165] The calculation of the matching cost varies in different steps. When selecting a candidate from the candidate set at the CU level, the matching cost can be the Absolute Sum Difference (SAD) of bilateral matching or template matching. After determining the starting MV, the matching cost C of bilateral matching for the sub-CU level search is calculated as follows:

[0166]

[0167] Here, w is the weight factor. In some embodiments, w can be empirically set to 4. MV and MV s respectively indicate the current MV and the starting MV. SAD can still be used as the matching cost of template matching for the sub-CU level search.

[0168] In the FRUC mode, the MV is derived only by using the luminance samples. The derived motion will be used for both the luminance and chrominance of MC inter prediction. After determining the MV, an 8-tap interpolation filter for luminance and a 4-tap interpolation filter for chrominance are used to perform the final MV.

[0169] MV refinement is a pattern-based MV search based on the bilateral matching cost or the template matching cost. In JEM, two search modes are supported - Unrestricted Center-Biased Diamond Search (UCBDS) and adaptive cross search, for MV refinement at the CU level and sub-CU level respectively. For both MV refinement at the CU and sub-CU levels, the MV is directly searched with a quarter luminance sample MV accuracy, followed by an eighth luminance sample MV refinement. The search range for MV refinement used for the CU and sub-CU steps is set to be equal to 8 luminance samples.

[0170] In the bilateral matching Merge mode, bidirectional prediction is applied because the motion information of the CU is derived based on the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. In the template matching Merge mode, the encoder can select for the CU from unidirectional prediction from list 0, unidirectional prediction from list 1, or bidirectional prediction. The selection can be based on the following template matching cost:

[0171] If costBi <= factor * min(cost0, cost1)

[0172] then bidirectional prediction is used;

[0173] Otherwise, if cost0 <= cost1

[0174] then unidirectional prediction from list 0 is used;

[0175] Otherwise,

[0176] then unidirectional prediction from list 1 is used;

[0177] Here, cost0 is the SAD of template matching in list 0, cost1 is the SAD of template matching in list 1, and costBi is the SAD of bidirectional prediction template matching. For example, when the value of the factor is equal to 1.25, this means the selection process favors bidirectional prediction. The inter-frame prediction direction selection can be applied to the CU-level template matching process.

[0178] 2.8 Examples of Generalized Bidirectional Prediction (GBI)

[0179] In traditional bidirectional prediction, the predicted values from L0 and L1 are averaged to generate the final predicted value with equal weights of 0.5. The formula for generating the predicted value is shown in Equation (5).

[0180] P TraditionalBiPred = (P L0 + P L1 + RoundingOffset) >> shiftNum Equation (5)

[0181] In Equation (5), P TraditionalBiPred is the final predicted value of traditional bidirectional prediction, P L0 and P L1 are the predicted values of L0 and L1 respectively, and RoundingOffset and shiftNum are used to normalize the final predicted value.

[0182] Generalized Bi-Directional Prediction (GBI) is proposed to allow different weights to be applied to the prediction values from L0 and L1. GBI is also known as Bi-Prediction with CU Weights (BCW). The prediction value is generated as shown in Equation (6).

[0183] P GBi = ((1 - w1) * P L0 + w1 * P L1 + RoundingOffset GBi ) >> shiftNum GBi Equation (6)

[0184] In Equation (6), P GBi is the final prediction value of GBi. (1 - w1) and w1 are the selected GBI weights applied to the prediction values of L0 and L1 respectively. RoundingOffset GBi and shiftNum GBi are used to normalize the final prediction value in GBi.

[0185] The supported weights w1 are {-1 / 4, 3 / 8, 1 / 2, 5 / 8, 5 / 4}. One equal-weight set and four unequal-weight sets are supported. For the equal-weight case, the process of generating the final prediction value is exactly the same as that in the traditional bi-directional prediction mode. For the true bi-directional prediction case under random access (RA) conditions, the number of candidate weight sets is reduced to three.

[0186] For the Advanced Motion Vector Prediction (AMVP) mode, if this CU is coded / decoded by bi-directional prediction, the weight selection in GBI is explicitly signaled at the CU level. For the Merge mode, the weight selection is inherited from the Merge candidates. In this scheme, GBI supports the weighted average of the DMVR generation templates and the final prediction value of BMS-1.0.

[0187] 2.9 Examples of Multi-Hypothesis Inter-Frame Prediction

[0188] In the multi-hypothesis inter-frame prediction mode, in addition to the traditional single / bi-directional prediction signals, one or more additional prediction signals are signaled. The overall prediction signal generated is obtained by weighted superposition of samples. Through the single / bi-directional prediction signal p uni / bi and the first additional inter-frame prediction signal / hypothesis h3, the generated prediction signal is as follows:[[]]

[0189] p3 = (1 - α)p uni / bi + αh3

[0190] The changes in the prediction unit's syntax structure are as follows:

[0191]

[0192] The weight factor α is specified by the syntax element add_hyp_weight_idx according to the following mapping:

[0193] add_hyp_weight_idx α 0 1 / 4 1 -1 / 8

[0194] In some embodiments, for additional prediction signals, the concept of prediction list 0 / list 1 is abolished and a combined list is used instead. The combined list is generated by alternately inserting reference frames from list 0 and list 1 with increasing reference indices and omitting the already inserted reference frames, thus avoiding double entries.

[0195] In some embodiments, and similar to the above, more than one additional prediction signal can be used. The resulting overall prediction signal is iteratively accumulated with each additional prediction signal.

[0196] p n+1 = (1 - α n+1 ) p n + α n+1 h n+1

[0197] Obtain the resulting overall prediction signal as the last p n (i.e., p n with the largest index n).

[0198] Note that, similarly for inter-frame prediction blocks using the Merge mode (instead of the skip mode), additional inter-frame prediction signals can be specified. Also note that in the case of Merge, not only the single / directional prediction parameters, but also the additional prediction parameters of the selected Merge candidate can be used for the current block.

[0199] 2.10 Examples of multi-hypothesis prediction for unidirectional prediction in the AMVP mode

[0200] In some embodiments, when multi-hypothesis prediction is applied to unidirectional prediction in the improved AMVP mode, a flag is signaled to enable or disable multi-hypothesis prediction for inter_dir equal to 1 or 2, where 1, 2, and 3 represent list 0, list 1, and bi-prediction, respectively. Additionally, when the flag is true, a Merge index is signaled. In this way, multi-hypothesis prediction changes the unidirectional prediction to a bi-prediction, where one motion is obtained using the original syntax elements in the AMVP mode, and the other motion is obtained using the Merge scheme. The final prediction combines these two predictions into a bi-prediction using a 1:1 weight. First, the Merge candidate list is derived from the Merge mode, excluding sub-CU candidates (e.g., affine, alternative temporal motion vector prediction (ATMVP)). Next, it is separated into two separate lists, one for list 0 (L0) that contains all L0 motions from the candidates, and the other for list 1 (L1) that contains all L1 motions. After removing redundancy and filling in the gaps, two Merge lists are generated for L0 and L1, respectively. When applying multi-hypothesis prediction to improve the AMVP mode, there are two constraints. First, for those CUs with a luma coding block (CB) area greater than or equal to 64, this multi-hypothesis prediction is enabled. Second, this multi-hypothesis prediction is only applicable to L1 when in low-delay B pictures.

[0201] 2.11 Example of Multi-Hypothesis Prediction for Skip / Merge Mode

[0202] In some embodiments, when multi-hypothesis prediction is applied to the Skip or Merge mode, whether to enable multi-hypothesis prediction is explicitly signaled. In addition to the original prediction, additional Merge index predictions are selected. Thus, each candidate for multi-hypothesis prediction implies a Merge candidate pair, where one is for the first Merge index prediction and the other is for the second Merge index prediction. However, in each pair, the Merge candidate for the second Merge index prediction is implicitly derived as the subsequent Merge candidate (i.e., the signaled Merge index plus 1), without signaling any additional Merge index. After removing redundancy by excluding those pairs that contain similar Merge candidates and filling in the gaps, a candidate list for multi-hypothesis prediction is formed. Then, the motions from the pairs of two Merge candidates are obtained to generate the final prediction, where 5:3 weights are applied to the first Merge index prediction and the second Merge index prediction, respectively. Additionally, a Merge or Skip CU that enables multi-hypothesis prediction can save the motion information of additional hypotheses in addition to the motion information of the existing hypotheses for reference by later neighboring CUs.

[0203] Note that sub-CU candidates (e.g., affine, ATMVP) are excluded from the candidate list, and for low-latency B pictures, multi-hypothesis prediction is not applicable to skip mode. Additionally, when multi-hypothesis prediction is applied to Merge or skip mode, for CUs with a CU width or CU height less than 16, or those with both a CU width and CU height equal to 16, a bilinear interpolation filter is used for multi-hypothesis motion compensation. Thus, the worst-case bandwidth (the number of accessed samples per sample point) for each Merge or skip CU with multi-hypothesis prediction enabled is less than half of the worst-case bandwidth of each 4×4 CU with multi-hypothesis prediction disabled.

[0204] 2.12 Examples of Ultimate Motion Vector Representation (UMVE)

[0205] In some embodiments, an Ultimate Motion Vector Representation (UMVE) is proposed. UMVE is used for skip or Merge mode by utilizing the proposed motion vector expression method.

[0206] UMVE reuses the same Merge candidates as in VVC. Among the Merge candidates, a candidate can be selected and further extended by the proposed motion vector expression method.

[0207] UMVE provides a new motion vector representation with simplified signaling. The expression method includes a starting point, a motion magnitude, and a motion direction. This proposed technique uses the Merge candidate list as it is. However, for the extension of UMVE, only candidates of the default Merge type (MRG_TYPE_DEFAULT_N) are considered.

[0208] The base candidate index (IDX) defines the starting point. The base candidate index indicates the best candidate among the candidates in the list, as follows.

[0209] Basic candidate IDX 0 1 2 3 The Nth MVP The first MVP The second MVP The third MVP The fourth MVP

[0210] If the number of base candidates is equal to 1, the base candidate IDX is not signaled.

[0211] The distance index is the motion magnitude information. The distance index indicates a predefined distance from the starting point information. The predefined distances are as follows:

[0212] Distance IDX 0 1 2 3 4 5 6 7 Pixel distance 1 / 4 pixel 1 / 2 pixel 1 pixel 2 pixels 4 pixels 8 pixels 16 pixels 32 pixels

[0213] The direction index represents the direction of the MVD relative to the starting point. The direction index can represent the following four directions:

[0214]

[0215] The UMVE flag is signaled immediately after the skip flag and the Merge flag are sent. If the skip and Merge flags are true, the UMVE flag is parsed. If the UMVE flag equals 1, the UMVE syntax is parsed. However, if it is not 1, the affine flag is parsed. If the affine flag equals 1, this is the affine mode. However, if it is not 1, the skip / Merge index will be parsed for the skip / Merge mode of VTM.

[0216] No additional line buffer is required for UMVE candidates. Because the software skip / Merge candidates are directly used as base candidates. Using the input UMVE index, the supplement of the MV is determined before motion compensation. No long line buffer needs to be reserved for this.

[0217] 2.13 Examples of the Affine Merge Mode with Prediction Offset

[0218] In some embodiments, UMVE is extended to the affine Merge mode, which we will hereafter refer to as the UMVE affine mode. The proposed method selects the first available affine Merge candidate as the base prediction value. Then, it applies a motion vector offset to the motion vector values of each control point from the base prediction value. If no available affine Merge candidate exists, the proposed method is not used.

[0219] The inter - prediction direction of the selected base prediction value and the reference index for each direction are used without change.

[0220] In the current implementation, the affine model of the current block is assumed to be a 4 - parameter model, and only 2 control points need to be derived. Therefore, only the first 2 control points of the base prediction value will be used as the control point prediction values.

[0221] For each control point, the zero_MVD flag is used to indicate whether the control point of the current block has the same MV value as the corresponding control point prediction value. If the zero_MVD flag is true, no other signaling is required for the control point. Otherwise, a distance index and an offset direction index are signaled for the control point.

[0222] A distance offset table of size 5 is used, as shown in the following table.

[0223] Distance IDX 0 1 2 3 4 Distance - offset 1 / 2 pixel 1 pixel 2 pixels 4 pixels 8 pixels

[0224] The distance index is signaled to indicate which distance offset to use. The mapping of the distance index and the distance offset value is as Figure 23 shown.

[0225] The direction index can represent the four directions as shown below, where only the x or y direction may have an MV difference, but not both.

[0226] Offset direction IDX 00 01 10 11 x - dir - factor +1 -1 0 0 y - dir - factor 0 0 +1 -1

[0227] If the inter-frame prediction is unidirectional, the signal distance offset is applied to the offset direction of each control point prediction value. The result will be the MV value of each control point.

[0228] For example, when the base prediction value is unidirectional, and the motion vector value of the control point is MVP(v px ,v py ). When the distance offset and direction index are signaled, the motion vector of the corresponding control point of the current block will be calculated as follows.

[0229] MV(v x ,v y ) = MVP(v px ,v py ) + MV(x-dir-factor * distance offset, y-dir-factor *

[0230] distance offset)

[0231] If the inter-frame prediction is bidirectional, the signaled distance offset is applied to the signaled offset direction of the L0 motion vector of the control point prediction value; and the same distance offset with the opposite direction is applied to the L1 motion vector of the control point prediction value. The result will be the MV value of each control point in each inter-frame prediction direction.

[0232] For example, when the base prediction value is unidirectional, and the motion vector value of the control point on L0 is MVP L0 (v 0px ,v 0py ), and the motion vector of this control point on L1 is MVP L1 (v 1px ,v 1py ). When the distance offset and direction index are signaled, the motion vector of the corresponding control point of the current block will be calculated as follows.

[0233] MV L0 (v 0x ,v 0y ) = MVP L0 (v 0px ,v 0py ) + MV(x-dir-factor * distance offset, y-dir-factor

[0234] * distance offset)

[0235] MV L1 (v 0x ,v 0y ) = MVPL1 (v 0px ,v 0py ) + MV(-x - dir - factor * distance offset, -y - dir -

[0236] factor * distance offset)

[0237] 2.14 Example of Bidirectional Optical Flow (BIO)

[0238] The bidirectional optical flow (BIO) method refines the motion at the sample level and is based on block - based motion compensation for bidirectional prediction. In some embodiments, sample - level motion refinement does not use signaling.

[0239] Let I (k) be the luminance value from reference k (k = 0, 1) after block motion compensation, and let and represent the horizontal and vertical components of the gradient of I (k) respectively. Assuming the optical flow is valid, the motion vector field (v x , v y ) is given by the following formula:

[0240]

[0241] Combining this optical flow equation with Hermite interpolation to obtain the motion trajectory of each sample, and finally obtaining a unique third - order polynomial that matches the function value I (k) and its derivatives and The value of this polynomial at t = 0 is the BIO prediction:

[0242]

[0243] Figure 24 Shows an example optical flow trajectory in the bidirectional optical flow (BIO) method. Here, τ0 and τ1 represent the distances to the reference frames. The distances τ0 and τ1 are calculated based on the POC of Ref0 and Ref1: τ0 = POC(current) - POC(Ref0), τ1 = POC(Ref1) - POC(current). If the two predictions are from the same time direction (either both from the past or both from the future), the signals are different (e.g., τ0·τ1 < 0). In this case, BIO is applied only when the predictions are not from the same moment (e.g., τ0 ≠ τ1), both reference regions have non - zero motion (e.g., MVx0, MVy0, MVx1, MVy1 ≠ 0), and the block motion vectors are proportional to the time distances (e.g., MVx0 / MVx1 = MVy0 / MVy1 = -τ0 / τ1).

[0244] The motion vector field (v x ,v y ) is determined by minimizing the difference Δ between the values of point A and point B. Figure 24 An example of the intersection of the motion trajectory and the reference frame plane is shown. The model only uses the first linear term of the local Taylor expansion for Δ:

[0245]

[0246] All values in the above equation depend on the sample position, denoted as (i′, j′). Assuming that the motion is consistent in the local surrounding area, Δ within a (2M + 1)×(2M + 1) square window Ω centered at the current prediction point (i, j) can be minimized, where M is equal to 2:

[0247]

[0248] For this optimization problem, JEM uses a simplified method, first minimizing in the vertical direction and then in the horizontal direction. This results in the following equations:

[0249]

[0250] where,

[0251]

[0252] To avoid division by zero or very small values, the regularization parameters r and m can be introduced into equations (9) and (10), where:

[0253] r = 500·4 d-8 Equation (12)

[0254] m = 700·4 d-8 Equation (13)

[0255] Here, d is the bit depth of the video sample.

[0256] To keep the memory access of BIO the same as that of regular bidirectional prediction motion compensation, all prediction and gradient values I (k) , are calculated for positions within the current block. Figure 25A An example of the access position outside block 2500 is shown. As Figure 25A shown, in equation (9), a (2M + 1)×(2M + 1) square window Ω centered at the current prediction point on the boundary of the prediction block needs to access positions outside the block. In JEM, the I (k) , values outside the block are set to be equal to the nearest available value inside the block. For example, this can be implemented as filling area 2501, asFigure 25B as shown

[0257] Using BIO, it is possible to refine the motion field for each sample point. To reduce the computational complexity, a block-based BIO design is used in JEM. The motion refinement can be calculated based on 4×4 blocks. In the block-based BIO, the s in Equation (9) for all sample points in a 4×4 block can be aggregated n values, and then the aggregated value of s n is used to derive the BIO motion vector offset for the 4×4 block. More specifically, the following formula can be used for block-based BIO derivation:

[0258]

[0259] Here, b k represents the set of sample points of the k-th 4×4 block belonging to the prediction block. The s in Equation (9) and Equation (10) n is replaced by ((s n,bk ) >> 4) to derive the associated motion vector offset.

[0260] In some scenarios, due to noise or irregular motion, the MV refinement of BIO may be unreliable. Therefore, in BIO, the magnitude of MV refinement is clipped to a threshold. The threshold is determined based on whether all reference pictures of the current picture are from one direction. For example, if all reference pictures of the current picture are from one direction, the threshold is set to 12×2 14-d ; otherwise, it is set to 12×2 13-d .

[0261] Operations consistent with the HEVC motion compensation process (e.g., 2D separable FIR (Finite Impulse Response)) can be used to calculate the gradient of BIO simultaneously with motion compensation interpolation. In some embodiments, the input of the 2D separable FIR is the same reference frame sample points as the motion compensation process and the fractional position (fracX, fracY) according to the fractional part of the block motion vector. For the horizontal gradient First, the signal is vertically interpolated using BIOfilterS corresponding to the fractional position fracY with a de-scaling offset d - 8. Then, a gradient filter BIOfilterG corresponding to the fractional position fracX with a de-scaling offset 18 - d is applied in the horizontal direction. For the vertical gradient Apply the BIOfilterG corresponding to the fractional position fracY with de-scaled offset d-8 to vertically apply the gradient filter. Then, use the BIOfilterS corresponding to the fractional position fracX with de-scaled offset up to 18-d to perform signal displacement in the horizontal direction. The lengths of the interpolation filter BIOfilterG and signal displacement BIOfilterF used for gradient calculation can be short (e.g., 6 taps) to maintain reasonable complexity. Table 1 shows example filters that can be used for gradient calculation at different fractional positions of the block motion vectors in BIO. Table 2 shows example interpolation filters that can be used for prediction signal generation in BIO.

[0262] Table 1: Exemplary Filters for Gradient Calculation in BIO

[0263] Fractional pixel position Interpolation filter for gradient (BIOfilterG) 0 {8,-39,-3,46,-17,5} 1 / 16 {8,-32,-13,50,-18,5} 1 / 8 {7,-27,-20,54,-19,5} 3 / 16 {6,-21,-29,57,-18,5} 1 / 4 {4,-17,-36,60,-15,4} 5 / 16 {3,-9,-44,61,-15,4} 3 / 8 {1,-4,-48,61,-13,3} 7 / 16 {0,1,-54,60,-9,2} 1 / 2 {-1,4,-57,57,-4,1}

[0264] Table 2: Exemplary Interpolation Filters for Prediction Signal Generation in BIO

[0265] Fractional pixel position Interpolation filter for predicted signal (BIOfilterS) 0 {0,0,64,0,0,0} 1 / 16 {1,-3,64,4,-2,0} 1 / 8 {1,-6,62,9,-3,1} 3 / 16 {2,-8,60,14,-5,1} 1 / 4 {2,-9,57,19,-7,2} 5 / 16 {3,-10,53,24,-8,2} 3 / 8 {3,-11,50,29,-9,2} 7 / 16 {3,-11,44,35,-10,3} 1 / 2 {3,-10,35,44,-11,3}

[0266] In JEM, when two predictions are from different reference pictures, BIO can be applied to all bi-directional prediction blocks. When local illumination compensation (LIC) is enabled for a CU, BIO can be disabled.

[0267] In some embodiments, OBMC is applied to blocks after the normal MC process. To reduce computational complexity, BIO may not be applied during the OBMC process. This means that when using its own MV, BIO is applied to the MC process of the block, while when using the MV of neighboring blocks during the OBMC process, BIO is not applied to the MC process.

[0268] 2.15 Examples of Decoder-Side Motion Vector Refinement (DMVR)

[0269] In bi-directional prediction operations, for the prediction of a block region, two prediction blocks formed using the motion vectors (MVs) of list 0 and list 1 respectively are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors for bi-directional prediction are further refined through a bilateral template matching process. Bilateral template matching is applied in the decoder to perform a distortion-based search between the bilateral template and the reconstructed samples in the reference picture to obtain refined MVs without sending additional motion information.

[0270] In DMVR, bilateral templates are generated from the initial MV0 of list 0 and MV1 of list 1 respectively as a weighted combination (i.e., average value) of the two prediction blocks, as Figure 26As shown. The template matching operation includes calculating a cost metric between the generated template and the sample area (around the initial prediction block) in the reference picture. For each of the two reference pictures, the MV that produces the minimum template cost is considered the updated MV in the list to replace the original MV. In JEM, 9 MV candidates are searched for each list. The 9 candidate MVs include the original MV and 8 surrounding MVs, where one luma sample is offset relative to the original MV in the horizontal or vertical direction or both. Finally, two new MVs (i.e., MV0' and MV1' as shown in Figure 26 are used to generate the final bi-prediction result. The sum of absolute differences (SAD) is used as the cost metric.

[0271] DMVR is applied to the Merge mode of bi-prediction, where one MV comes from a past reference picture and the other from a future reference picture without transmitting additional syntax elements. In JEM, when LIC, affine motion, FRUC, or sub-CU Merge candidates are enabled for a CU, DMVR is not applied.

[0272] 3. Exemplary Embodiments Related to the Disclosed Technology

[0273] For example, some embodiments include an MV update method and a two-step inter-frame prediction method. The derived MV between reference block 0 and reference block 1 in BIO is scaled and added to the original motion vectors in list 0 and list 1. At the same time, the updated MV is used to perform motion compensation, and a second inter-frame prediction is generated as the final prediction. Other embodiments include modifying the temporal gradient by removing the average difference between reference block 0 and reference block 1. In still other embodiments, the MV update method and the two-step inter-frame prediction method are extended to be performed multiple times.

[0274] 4. Disadvantages of Existing Embodiments

[0275] In some existing embodiments, it has not been well defined how to apply BIO or / and DMVR or any other decoder-side motion vector derivation / refinement tool to a CU encoded / decoded in the multi-hypothesis prediction mode or the GBI mode.

[0276] In other existing embodiments using generalized bi-prediction (GBI), when encoding the weight factor index (e.g., the GBI index), all binary values are context-encoded / decoded, which is computationally complex.

[0277] 5. Example Methods of Juxtaposing the DMVD Tool with Other Tools

[0278] Embodiments of the presently disclosed technology overcome the drawbacks of existing implementations, thus providing video coding and decoding with higher coding and decoding efficiency. Based on the disclosed technology, the juxtaposition of DMVD tools with other video coding and decoding tools can enhance existing and future video coding and decoding standards, as elaborated in the following examples described for various embodiments. The examples of the disclosed technology provided below explain the general concept and are not meant to be construed as limiting. In one example, various features described in these examples can be combined unless explicitly indicated to the contrary.

[0279] The disclosed technology describes how BIO or / and DMVR or any other decoder-side motion vector derivation / refinement tool can be applied to blocks coded with the multi-hypothesis prediction mode or the GBI mode. Hereinafter, DMVD (decoder-side motion vector derivation) is used to represent BIO or / and DMVR or / and FRUC or / and any other decoder-side motion vector derivation / refinement technique. Additionally, an extension of the DMVD technology to other coding and decoding methods is also proposed in this document.

[0280] Example 1. It is proposed that DMVD (e.g., as described in Sections 2.9, 2.10, and 2.11) can be performed on blocks coded with the multi-hypothesis prediction mode.

[0281] (a) In one example, for a block coded with multi-hypothesis and uni-directional prediction AMVP, if it is predicted using two reference blocks from different prediction directions, DMVD can be performed.

[0282] (b) In one example, if a block is predicted using three reference blocks, DMVD can be performed by selecting two reference blocks. Representing the three reference blocks as ref0, ref1, and ref2, assuming ref0 is from prediction direction X and ref1 and ref2 are from prediction direction 1–X, DMVD can be performed on the reference block pairs (ref0, ref1) or / and (ref0, ref2).

[0283] (i) In one example, DMVD can be performed only once. In this case, (ref0, ref1) or (ref0, ref2) can be utilized during the DMVD process.

[0284] (ii) Alternatively, DMVD can be performed twice. In this case, (ref0, ref1) and (ref0, ref2) can be utilized during the first DMVD process and the second DMVD process, respectively. Alternatively, (ref0, ref2) and (ref0, ref1) can be utilized during the first DMVD process and the second DMVD process, respectively.

[0285] (1) In one example, the refinement information of a DMVD process (e.g., the first DMVD), such as the refined motion information generated due to BIO or DMVR, can be used as the input for another DMVD process (e.g., the second DMVD). That is, a sequential process (sequential mode) of two DMVD processes can be applied.

[0286] (2) Alternatively, multiple DMVD processes can be called with the same input such that multiple processes can be completed in parallel (parallel mode).

[0287] (iii) In one example, if ref0 is refined twice by BIO, denoted as ref0_r1 and ref0_r2, then ref0_r1 and ref0_r2 can be used jointly (e.g., averaged or weighted averaged) to generate the final refined value of ref0. For example, ref0’ = (ref0_r1 + ref0_r2 + 1) >> 1 is used as the refined ref0. Alternatively, ref0_r1 or ref0_r2 is used as the refined ref0.

[0288] (iv) In one example, if the motion information of ref0 (denoted as MV0) is refined twice by BIO or DMVR, denoted as MV0_r1 and MV0_r2, then MV0_r1 and MV0_r2 can be used jointly (e.g., averaged or weighted averaged) to generate the final refined value of MV0. For example, MV0’ = (MV0_r1 + MV0_r2 + 1) >> 1 is used as the refined MV0. Alternatively, MV0_r1 or MV0_r2 is used as the refined MV0.

[0289] (c) In one example, if a block is predicted using N reference blocks, and M reference blocks are from prediction direction X, and N – M reference blocks are from prediction direction 1 – X, then DMVD can be performed on any one / some of the two reference blocks, where one is from prediction direction X and the other is from prediction direction 1 – X.

[0290] (i) Similar to (b)(ii), the DMVD process can be called multiple times in parallel mode or sequential mode.

[0291] (ii) In one example, if a reference block is refined T times by BIO, then some or all of these T refined values can be used jointly (e.g., using an average or weighted average) to derive the final refined value of the reference block.

[0292] (iii) Alternatively, if the reference block is refined T times by BIO, some or all of the values refined PT times can be jointly used (e.g., using the average or weighted average) to derive the final refined value of the reference block. For example, PT is equal to 1, 2, 3, 4, …, T – 1.

[0293] (iv) In one example, if the motion information of the reference block is refined T times by, for example, BIO or DMVR, some or all of these MVs refined T times can be jointly used (e.g., using the average or weighted average) to derive the final refined MV of the reference block.

[0294] (v) Alternatively, if the motion information of the reference block is refined T times by, for example, BIO or DMVR, some or all of the MVs refined PT times can be jointly used (e.g., using the average or weighted average) to derive the final refined value of the reference block. For example, PT is equal to 1, 2, 3, 4, …, T – 1.

[0295] (d) In one example, if a block is predicted using multiple sets of motion information (e.g., one set of motion information comes from the AMVP mode and another set of motion information comes from a Merge candidate (e.g., as described in Section 2.10), or both sets of motion information come from Merge candidates (e.g., as described in Section 2.11)), then DMVD can be performed on each / some of the sets of motion information.

[0296] (i) In one example, DMVD is invoked when the motion information is bi - directional motion information.

[0297] (ii) In one example, in the multi - hypothesis inter - prediction mode (as described in Section 2.9), DMVD is performed only for non - additional motion information.

[0298] (e) In one example, DMVD can be performed at most once.

[0299] (i) For example, if a block is coded / decoded using the multi - hypothesis Merge / skip mode (e.g., as described in Section 2.11), then DMVD is performed only on the first selected Merge candidate.

[0300] (1) Alternatively, DMVD is performed only on the second selected Merge candidate.

[0301] (2) Alternatively, 2 / N selected Merge candidates are checked in order, and DMVD is performed only on the first available bi - directional Merge candidate.

[0302] (ii) For example, if a block is encoded / decoded using the multi-hypothesis AMVP mode (as described in Section 2.10), then DMVD is performed only on the Merge candidates.

[0303] (iii) For example, if a block is encoded / decoded using the multi-hypothesis inter prediction mode (e.g., as described in Section 2.9), then the first available reference block (if any) from list 0 and list 1 is identified according to the signaling order of the corresponding syntax element, and DMVD is performed only on these two reference blocks.

[0304] Example 2. It is proposed that when using an asymmetric weight factor for bi-directional prediction (such as GBI, LIC, etc.) or multi-hypothesis prediction, such a weight factor can be used in the DMVD process.

[0305] (a) In one example, before calculating the temporal gradient and / or spatial gradient in the BIO, each reference block can be scaled by its corresponding weight factor.

[0306] (b) In one example, when performing bilateral matching or template matching in the DMVR, each reference block can be scaled by its corresponding weight factor.

[0307] (c) In one example, if template matching is used in the DMVR, such a weight factor can be used when generating the template.

[0308] Example 3. It is proposed that when all MVD components are zero, DMVD can be used in the AMVP mode. Alternatively, if the MVD component is zero in the prediction direction X and non-zero in the prediction direction 1 - X, then DMVD can be used to refine the motion vector in the prediction direction X. In one example, in the DMVR, the prediction signal in list 1 - X is used as a template to find the best motion vector in list X.

[0309] Example 4. It is proposed that DMVD can be used to refine the translational motion parameters in the bi-directional affine mode or the UMVE affine mode.

[0310] (a) Alternatively, in the bi-directional affine inter-frame mode, DMVD is used to refine the translational motion parameters only when all MVDs of the translational motion parameters are zero.

[0311] (b) Alternatively, in the UMVE affine mode, DMVD is used to refine the translational motion parameters only when all MVDs of the translational motion parameters are zero.

[0312] Example 5. It is proposed to enable DMVD in the UMVE mode.

[0313] (a) Alternatively, when there is a non-zero MVD component in the UMVE mode, DMVD is disabled.

[0314] (b) Alternatively, DMVD is disabled in the UMVE mode.

[0315] Example 6. It is proposed that DMVD can be applied under certain conditions, such as based on block size, coding mode, motion information, slice / picture / tile type.

[0316] (a) In one example, when the block size contains fewer than M×H samples (e.g., 16 or 32 or 64 luma samples), DMVD is not allowed.

[0317] (b) In one example, when the block size contains more than M×H samples (e.g., 16 or 32 or 64 luma samples), DMVD is not allowed.

[0318] (c) Alternatively, when the minimum size of the width or height of the block is less than or not greater than X, DMVD is not allowed. In one example, X is set to 8.

[0319] (d) Alternatively, when the width of the block > th1 or ≥ th1 and / or the height of the block > th2 or ≥ th2, DMVD is not allowed. In one example, X is set to 64.

[0320] (i) For example, for a 128×128 block, DMVD is disabled.

[0321] (ii) For example, for N×128 / 128×N blocks (for N≥64), DMVD is disabled.

[0322] (iii) For example, for N×128 / 128×N blocks (for N≥4), DMVD is disabled.

[0323] (e) Alternatively, when the width of the block < th1 or ≤ th1 and / or the height of the block < th2 or ≤ th2, DMVD is not allowed. In one example, th1 or th2 is set to 8.

[0324] (f) In one example, for blocks coded / decoded in the AMVP mode, DMVD is disabled.

[0325] (g) In one example, for blocks coded / decoded in the skip mode, DMVD is disabled.

[0326] (h) In one example, for blocks using GBI, DMVD is disabled.

[0327] (i) In one example, for a block that uses multi-hypothesis inter prediction (as described in Sections 2.9, 2.10, and 2.11), for example, in the case of predicting a CU from more than 2 reference blocks, DMVD is disabled.

[0328] (j) In one example, for a block / sub-block when the absolute average difference between two reference blocks / sub-blocks is greater than a threshold, DMVD is disabled.

[0329] Example 7. Applying DMVD at the sub-block level is proposed.

[0330] (a) In one example, DMVD can be called for each sub-block.

[0331] (b) In one example, when the width or height of a block is greater than (or equal to) a threshold L or both the width and height are greater than (or equal to) the threshold L, the block can be divided into multiple sub-blocks. Each sub-block is processed in the same way as a normal coding / decoding block with a size equal to the sub-block size.

[0332] (i) In one example, L is 64, a 64×128 / 128×64 block is divided into two 64×64 sub-blocks, and a 128×128 block is divided into four 64×64 sub-blocks. However, an N×128 / 128×N block (where N < 64) is not divided into sub-blocks.

[0333] (ii) In one example, L is 64, a 64×128 / 128×64 block is divided into two 64×64 sub-blocks, and a 128×128 block is divided into four 64×64 sub-blocks. At the same time, an N×128 / 128×N block (where N < 64) is divided into two N×64 / 64×N sub-blocks.

[0334] (c) The threshold L can be predefined or signaled at the SPS / PPS / picture / strip / slice group / slice level.

[0335] (d) Alternatively, the threshold can depend on certain coding / decoding information, such as block size, picture type, temporal layer index, etc.

[0336] Example 8. In one example, whether and how to apply the above methods (e.g., motion refinement methods such as DMVR or / and BIO and / or other decoder-side motion refinement techniques) depends on the reference picture.

[0337] (a) In one example, if the reference picture is the current coded picture, the motion refinement method is not applied.

[0338] (b) In one example, if the reference picture is the current coded picture, the multi-time motion refinement method required in the previous bullet point is not applied.

[0339] Example 9. Even if the two reference blocks / reference pictures are from the same reference picture list, the above method and existing DMVD methods (e.g., BIO / DMVR) can be applied.

[0340] (a) Alternatively, in addition, when the two reference blocks are from the same reference picture list, it is required that the two reference pictures span the current picture covering the current block. That is, compared with the POC of the current picture, one reference picture has a smaller POC value and the other reference picture has a larger POC value.

[0341] (b) In one example, the condition check for bidirectional prediction for enabling / disabling BIO for the prediction direction is removed. That is, whether BIO or DMVD is enabled is independent of the prediction direction value.

[0342] (c) When the product of the POC differences between the current picture and its two reference pictures (from the same reference picture list or different reference picture lists) is less than 0, BIO or DMVD can be enabled.

[0343] (d) When the product of the POC differences between the current picture and its two reference pictures (from the same reference picture list or different reference picture lists) is less than or equal to 0 (e.g., one or both reference pictures are the current picture), BIO or DMVD can be enabled.

[0344] Example 10. When encoding the GBI index, it is proposed to bypass the encoding / decoding of some binary values. Denote the maximum length of the code binary value of the GBI index as maxGBIIdxLen.

[0345] (a) In one example, only the first binary value is context encoded while all other binary values are bypass encoded.

[0346] (i) In one example, one context is used to encode the first binary value.

[0347] (ii) In one example, more than 1 context is used to encode the first binary value. For example, the following 3 contexts are used:

[0348] (1) ctxIdx = aboveBlockIsGBIMode + leftBlockIsGBIMode;

[0349] (2) If the above adjacent block is encoded and decoded in GBI mode, then aboveBlockIsGBIMode is equal to 1, otherwise it is equal to 0; and

[0350] (3) If the left adjacent block is encoded and decoded in GBI mode, then leftBlockIsGBIMode is equal to 1, otherwise it is equal to 0.

[0351] (b) In one example, only the first K binary values are context-encoded, and all other binary values are bypass-encoded, where 0 <= K <= maxGBIIdxLen.

[0352] (i) In one example, all context-encoded binary values except the first binary value share a context.

[0353] (ii) In one example, each context-encoded binary value except the first binary value uses a context.

[0354] The above examples can be incorporated into the context of the methods described below, e.g., methods 2700, 2710, 2720, 2730, 2740, 2750, 2760, and 2770, which can be implemented at a video decoder or a video encoder.

[0355] Figure 27A A flowchart of an exemplary method for video decoding is shown. Method 2700 includes, at operation 2702, making a decision on the selective enabling of a decoder-side motion vector derivation (DMVD) tool based on a determination that the current block of a video is encoded using a multi-hypothesis prediction mode, where the DMVD tool derives a refinement of the motion information signaled in the bitstream representation of the video. In some embodiments, the multi-hypothesis prediction mode is configured to generate a final prediction of the current block by applying at least one intermediate prediction value.

[0356] Method 2700 includes, at operation 2704, performing a conversion between the current block and the bitstream representation based on the decision.

[0357] Figure 27B A flowchart of an exemplary method for video decoding is shown. Method 2710 includes, at operation 2712, determining that the current block of a video is associated with an asymmetric weight factor with respect to different reference blocks.

[0358] Method 2710 includes, at operation 2714, enabling a decoder-side motion vector derivation (DMVD) tool for the current block, where the DMVD tool derives a refinement of the motion information signaled in the bitstream representation of the video, and the DMVD process is based on the asymmetric weighting factor.

[0359] Method 2710 includes, at operation 2716, performing a conversion between the current block and the bitstream representation based on the enablement.

[0360] Figure 27C A flowchart of an exemplary method for video decoding is shown. Method 2720 includes, at operation 2722, determining that a current block of a video is coded using an advanced motion vector prediction (AMVP) mode.

[0361] Method 2720 includes, at operation 2724, applying a decoder-side motion vector derivation (DMVD) tool to the current block as part of a conversion between the bitstream representation of the video and the current block, the DMVD tool deriving a refinement of the motion information signaled in the bitstream representation.

[0362] Figure 27D A flowchart of an exemplary method for video decoding is shown. Method 2730 includes, at operation 2732, performing a refinement of translational motion parameters and a motion vector difference of a current block of a video coded using a bi-affine mode or a bi-affine Merge mode based on a decoder-side motion vector derivation (DMVD) tool, the motion vector difference being indicated by a motion direction and a motion magnitude, the DMVD tool deriving a refinement of the motion information signaled in the bitstream representation of the video.

[0363] Method 2730 includes, at operation 2734, performing a conversion between the current block and the bitstream representation of the video based on the refinement.

[0364] Figure 27E A flowchart of an exemplary method for video decoding is shown. Method 2740 includes, at operation 2742, making a decision on a selective enablement of a decoder-side motion vector derivation (DMVD) tool for the current block based on characteristics of the current block of the video, the DMVD tool deriving a refinement of the motion information signaled in the bitstream representation of the video.

[0365] Method 2740 includes, at operation 2744, performing a conversion between the current block and the bitstream representation based on the decision.

[0366] Figure 27F A flowchart of an exemplary method for video decoding is shown. Method 2750 includes, at operation 2752, making a decision on a selective enablement of a decoder-side motion vector derivation (DMVD) tool at a sub-block level based on a determination that the current block of the video includes a plurality of sub-blocks, the DMVD tool deriving a refinement of the motion information signaled in the bitstream representation of the video for each sub-block.

[0367] Method 2750 includes, at operation 2754, performing a conversion between the current block and the bitstream representation based on the decision.

[0368] Figure 27G A flowchart showing an exemplary method for video decoding is presented. Method 2760 includes, at operation 2762, making a decision on the selective enabling of a decoder-side motion vector derivation (DMVD) tool for a current block of video based on at least one reference picture associated with the current block, where the DMVD tool derives a refinement of the motion information signaled in the bitstream representation of the video.

[0369] Method 2760 includes, at operation 2764, performing a conversion between the current block and the bitstream representation based on the decision.

[0370] Figure 27H A flowchart showing an exemplary method for video decoding is presented. Method 2770 includes, at operation 2772, parsing a string of binary values from the bitstream representation of a current block of video, the string of binary values including a plurality of binary values representing a GBI index for a GBI mode, and at least one of the plurality of binary values being bypass-coded.

[0371] Method 2770 includes, at operation 2774, performing a conversion between the current block and the bitstream representation based on the parsed GBI index.

[0372] Figure 27I A flowchart showing an exemplary method for video decoding is presented. Method 2780 includes, at operation 2782, encoding a string of binary values into the bitstream representation of a current block of video, the string of binary values including a plurality of binary values representing a GBI index for a generalized bi-directional prediction (GBI) mode, and where at least one of the plurality of binary values is bypass-coded.

[0373] Method 2780 includes, at operation 2784, performing a conversion between the current block and the bitstream representation based on the encoded string of binary values.

[0374] In some embodiments, the following technical solutions may be implemented:

[0375] A1. A method for video processing (e.g., the method 2700 in Figure 27A ), includes: making (2702) a decision on the selective enabling of a decoder-side motion vector derivation (DMVD) tool for a current block of video based on a determination that the current block of video is encoded and decoded using a multi-hypothesis prediction mode, where the DMVD tool derives a refinement of the motion information signaled in the bitstream representation of the video; and performing (2704) a conversion between the current block and the bitstream representation based on the decision, where the multi-hypothesis prediction mode is configured to generate a final prediction of the current block by applying at least one intermediate prediction value.

[0376] A2. The method according to A1, wherein when determining to use two reference blocks from different prediction directions and predicting a current block using unidirectional prediction and multi - hypothesis prediction modes of the advanced motion vector prediction (AMVP) mode, the DMVD tool is enabled.

[0377] A3. The method according to A1, wherein the transformation is also performed based on N reference blocks, and N is a positive integer.

[0378] A4. The method according to A3, wherein the N reference blocks include M reference blocks from a first prediction direction and (N - M) reference blocks from a second prediction direction different from the first prediction direction, where M is a positive integer, and N > M.

[0379] A5. The method according to A4, wherein performing the transformation includes applying the DMVD tool to one of the N reference blocks and one of the (N - M) reference blocks.

[0380] A6. The method according to A4, wherein performing the transformation includes applying the DMVD tool multiple times, and for each of the multiple times, the DMVD tool is applied to two of the N reference blocks.

[0381] A7. The method according to A6, wherein the DMVD tool is applied multiple times to the first block of the M reference blocks and consecutive blocks of the (N - M) reference blocks.

[0382] A8. The method according to A6, wherein the DMVD tool is applied in parallel multiple times.

[0383] A9. The method according to A6, wherein the DMVD tool is applied sequentially multiple times.

[0384] A10. The method according to A4, further comprising: generating a (N + 1)th reference block based on a first bi - directional optical flow refinement, where the first reference block from the M reference blocks and the second reference block from the (N - M) reference blocks are used as inputs; generating a (N + 2)th reference block based on using a second bi - directional optical flow refinement, where the first reference block and a third reference block from the (N - M) reference blocks are used as inputs; and recalculating the first reference block as a weighted average of the (N + 1)th reference block and the (N + 2)th reference block.

[0385] A11. The method according to A1, wherein the multi - hypothesis prediction mode is the multi - hypothesis Merge or skip mode, and enabling the DMVD tool is also based on a first selected Merge candidate.

[0386] A12. The method according to Scheme A1, wherein the multi-hypothesis prediction mode is a multi-hypothesis Merge or skip mode, and wherein enabling the DMVD tool is further based on a second selected Merge candidate.

[0387] A13. The method according to Scheme A1, wherein the multi-hypothesis prediction mode is a multi-hypothesis Merge or skip mode, and wherein enabling the DMVD tool is further based on a first available bidirectional Merge candidate.

[0388] A14. The method according to Scheme A1, wherein the multi-hypothesis prediction mode is a multi-hypothesis advanced motion picture prediction (AMVP) mode, and wherein enabling the DMVD tool is further based on a Merge candidate.

[0389] A15. The method according to any one of Schemes A4 to A10, wherein N = 3 and M = 2.

[0390] A16. A method for video processing (e.g., Figure 27B method 2710 therein), comprising: determining (2712) that a current block of a video is associated with an asymmetric weight factor of different reference blocks; enabling (2714) a decoder-side motion vector derivation (DMVD) tool for the current block, wherein the DMVD tool derives a refinement of motion information signaled in a bitstream representation of the video, and wherein the DMVD process is based on the asymmetric weight factor; and performing (2716) a conversion between the current block and the bitstream representation based on this enabling.

[0391] A17. The method according to Scheme A16, wherein the current block is encoded and decoded using a bidirectional prediction mode or a multi-hypothesis prediction mode that uses one or more asymmetric weight factors.

[0392] A18. The method according to Scheme A16, wherein enabling the DMVD tool is further based on scaling two or more of N reference blocks with corresponding weight factors from one or more asymmetric weight factors.

[0393] A19. The method according to Scheme A16, wherein using the DMVD tool includes using bilateral matching or template matching.

[0394] A20. A method for video processing (e.g., Figure 27C method 2720 therein), comprising: determining (2722) that a current block of a video is encoded and decoded using an advanced motion vector prediction (AMVP) mode; and applying (2724) a decoder-side motion vector derivation (DMVD) tool to the current block as part of a conversion between the bitstream representation of the video and the current block, wherein the DMVD tool derives a refinement of motion information signaled in the bitstream representation.

[0395] A21. The method according to Scheme A20 further includes: determining that at least one of a plurality of motion vector difference (MVD) components of the current block is zero.

[0396] A22. The method according to Scheme A20, wherein each of the plurality of MVD components is zero.

[0397] A23. The method according to Scheme A20, wherein a first MVD component among the plurality of MVD components is zero in a first prediction direction, a second MVD component among the plurality of MVD components is non - zero in a second prediction direction, and wherein applying the DMVD tool includes refining the motion vector in the first prediction direction.

[0398] A24. The method according to Scheme A23, wherein a prediction signal in the first prediction direction is used to derive the motion vector in the second prediction direction.

[0399] A25. A method for video processing (e.g., Figure 27D the method 2730 in), includes: based on a decoder - side motion vector derivation (DMVD) tool, performing (2732) refinement of translational motion parameters and motion vector differences of a current block of a video encoded or decoded using a bi - affine mode or a bi - affine Merge mode, the motion vector difference being indicated by a motion direction and a motion amplitude, wherein the DMVD tool derives refinement of motion information signaled in a bit - stream representation of the video; and performing (2734) a conversion between the current block and the bit - stream representation of the video based on the refinement.

[0400] A26. The method according to Scheme A25, wherein the bi - affine Merge mode further includes a starting point of motion information indicated by a Merge index, and wherein the final motion information of the current block is based on the motion vector difference and the starting point.

[0401] A27. The method according to Scheme A25 or A26, wherein each motion vector difference of the translational motion parameters is zero.

[0402] A28. The method according to any one of Schemes A1 to A27, wherein the DMVD tool includes a decoder - side motion vector refinement (DMVR) tool, or a bi - directional optical flow (BDOF) tool, or a frame rate up - conversion (FRUC) tool.

[0403] A29. The method according to any one of Schemes A1 to A28, wherein the conversion generates the current block from the bit - stream representation.

[0404] A30. The method according to any one of Schemes A1 to A28, wherein the conversion generates the bit - stream representation from the current block.

[0405] A31. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when run by the processor, cause the processor to implement the method according to any one of A1 to A30.

[0406] A32. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing the method according to any one of A1 to A30.

[0407] In some embodiments, the following technical solutions may be implemented:

[0408] B1. A method for video processing, comprising: making a decision on the selective enabling of a decoder-side motion vector derivation (DMVD) tool for a current block of a video based on characteristics of the current block, wherein the DMVD tool derives a refinement of motion information signaled in a bitstream representation of the video; and performing a conversion between the current block and the bitstream representation based on the decision.

[0409] B2. The method according to solution B1, wherein the characteristics of the current block include the size or coding / decoding mode of the current block, motion information associated with the current block, slice type, picture type or tile type.

[0410] B3. The method according to solution B1, wherein the DMVD tool is disabled when it is determined that the number of luminance samples in the current block is less than K, where K is a positive integer.

[0411] B4. The method according to solution B1, wherein the DMVD tool is disabled when it is determined that the number of luminance samples in the current block is greater than K, where K is a positive integer.

[0412] B5. The method according to solution B1, wherein the DMVD tool is enabled when it is determined that the number of luminance samples in the current block is greater than K, where K is a positive integer.

[0413] B6. The method according to any one of solutions B3 to B5, wherein K = 16, 32, 64 or 128.

[0414] B7. The method according to solution B1, wherein the DMVD tool is disabled when it is determined that the minimum dimension of the height or width of the current block is less than or equal to K

[0415] B8. The method according to solution B7, wherein K = 8.

[0416] B9. The method according to Scheme B1, wherein when it is determined that the height of the current block is greater than or equal to tH or the width of the current block is greater than or equal to tW, the DMVD tool is disabled, and wherein tH and tW are positive integers.

[0417] B10. The method according to Scheme B1, wherein when it is determined that the height of the current block is greater than or equal to tH and the width of the current block is greater than or equal to tW, the DMVD tool is disabled, and wherein tH and tW are positive integers.

[0418] B11. The method according to Scheme B9 or B10, wherein tH = 64 and tW = 64.

[0419] B12. The method according to Scheme B9 or B10, wherein tH = 128 and tW = 128.

[0420] B13. The method according to Scheme B9 or B10, wherein tH = 128, and wherein tW ≥ 64 or tW ≥ 4.

[0421] B14. The method according to Scheme B9 or B10, wherein tH ≥ 64 or tH ≥ 4, and wherein tW = 128.

[0422] B15. The method according to Scheme B1, wherein when it is determined that the height of the current block is less than tH or the width of the current block is less than tW, the DMVD tool is disabled, and wherein tH and tW are positive integers.

[0423] B16. The method according to Scheme B1, wherein when it is determined that the height of the current block is not less than tH or the width of the current block is not less than tW, the DMVD tool is disabled, and wherein tH and tW are positive integers.

[0424] B17. The method according to Scheme B1, wherein when it is determined that the height of the current block is less than or equal to tH and the width of the current block is less than or equal to tW, the DMVD tool is disabled, and wherein tH and tW are positive integers.

[0425] B18. The method according to any one of Schemes B15 to B17, wherein tH = 8 and tW = 8.

[0426] B19. The method according to Scheme B1, wherein when it is determined that the coding / decoding mode of the current block is the generalized bi - directional prediction (GBI) mode, the DMVD tool is disabled, and wherein an asymmetric weight factor is applied to two reference blocks.

[0427] B20. The method according to Scheme B1, wherein when it is determined that the coding / decoding mode of the current block is the advanced motion vector prediction (AMVP) mode, the DMVD tool is disabled.

[0428] B21. The method according to Scheme B1, wherein when it is determined that the encoding / decoding mode of the current block is the skip mode or the multi-hypothesis inter-frame prediction mode, the DMVD tool is disabled.

[0429] B22. The method according to Scheme B1, wherein the current block includes at least one sub-block.

[0430] B23. The method according to Scheme B22, wherein when it is determined that the absolute average difference between two reference blocks associated with the current sub-block is greater than a threshold, the DMVD tool is disabled for the sub-block.

[0431] B24. The method according to Scheme B1, wherein when it is determined that the absolute average difference between two reference blocks associated with the current block is greater than a threshold, the DMVD tool is disabled.

[0432] B25. The method according to any one of Schemes B1 to B24, wherein the DMVD tool includes a decoder-side motion vector refinement (DMVR) tool, or a bidirectional optical flow (BDOF) tool, or a frame rate up-conversion (FRUC) tool.

[0433] B26. The method according to any one of Schemes B1 to B25, wherein the transformation generates the current block from the bitstream representation.

[0434] B27. The method according to any one of Schemes B1 to B25, wherein the transformation generates a bitstream representation from the current block.

[0435] B28. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when run by the processor, cause the processor to implement the method according to any one of Schemes B1 to B27.

[0436] B29. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing the method according to any one of Schemes B1 to B27.

[0437] In some embodiments, the following technical solutions may be implemented:

[0438] C1. A method for video processing (e.g., Figure 27F the method 2750 therein), comprising: making (2752) a decision on the selective enabling of a decoder-side motion vector derivation (DMVD) tool at the sub-block level based on the determination that the current block of the video includes a plurality of sub-blocks, wherein the DMVD tool derives a refinement of motion information signaled in the bitstream representation of the video for each sub-block; and performing (2754) a transformation between the current block and the bitstream representation based on the decision.

[0439] C2. The method according to C1, wherein the DMVD tool is enabled for each of a plurality of sub - blocks of the current block.

[0440] C3. The method according to C2, wherein the sub - block is regarded as a block, and all operations required in the DMVD tool are performed on the sub - block.

[0441] C4. The method according to C1 or C2, wherein when it is determined that the height and width of the current block are not greater than a threshold value (L), the DMVD tool is enabled at the block level, and wherein L is a positive integer.

[0442] C5. The method according to C1 or C2, wherein the height (H) or width (W) of the current block is greater than a threshold value (L), and wherein L is a positive integer.

[0443] C6. The method according to C5, wherein the width of the sub - block is min(L, W).

[0444] C7. The method according to C5, wherein the height of the sub - block is min(L, H).

[0445] C8. The method according to C5, wherein L = 64, wherein the size of the current block is 64×128, 128×64 or 128×128, and wherein the size of each of the plurality of sub - blocks is 64×64.

[0446] C9. The method according to C5, wherein L = 16.

[0447] C10. The method according to C5, wherein L = 64, wherein the size of the current block is N×128 or 128×N, and wherein the sizes of two of the plurality of sub - blocks are N×64 or 64×N respectively.

[0448] C11. The method according to C5, wherein L is pre - determined.

[0449] C12. The method according to C5, wherein L is signaled in the bit - stream representation in the sequence parameter set (SPS), picture parameter set (PPS), picture header, slice header, slice group header or picture header.

[0450] C13. The method according to C5, wherein L is based on at least one of the size of the current block or the coding / decoding mode, picture type or temporal layer index.

[0451] C14. The method according to any one of C1 to C13, wherein the transformation generates the current block from the bit - stream representation.

[0452] C15. The method according to any one of C1 to C13, wherein the conversion generates a bitstream representation from a current block.

[0453] C16. The method according to any one of C1 to C15, wherein the DMVD tool includes a decoder-side motion vector refinement (DMVR) tool, or a bidirectional optical flow (BDOF) tool, or a frame rate up-conversion (FRUC) tool.

[0454] C17. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when run by the processor, cause the processor to implement the method according to any one of C1 to C16.

[0455] C18. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing the method according to any one of C1 to C16.

[0456] In some embodiments, the following technical solutions may be implemented:

[0457] D1. A method for video processing (e.g., Figure 27G method 2760 therein), comprising: making (2762) a decision on the selective enabling of a decoder-side motion vector derivation (DMVD) tool for a current block based on at least one reference picture associated with the current block of the video, wherein the DMVD tool derives a refinement of motion information signaled in the bitstream representation of the video; and performing (2764) a conversion between the current block and the bitstream representation based on the decision.

[0458] D2. The method according to D1, wherein the DMVD tool is not enabled when it is determined that the at least one reference picture includes the current coded picture.

[0459] D3. The method according to D1, wherein the DMVD tool is enabled when it is determined that the at least one reference picture includes a first reference picture from a first reference picture list and a second reference picture from the first reference picture list.

[0460] D4. The method according to D3, wherein the picture order count (POC) value of the first reference picture is less than the POC value of the current picture including the current block, and wherein the POC value of the second reference picture is greater than the POC value of the current picture.

[0461] D5. The method according to D3, wherein the product of the POC differences between the POC value of the current picture including the current block and the POC value of the first reference picture and between the POC value of the current picture and the POC value of the second reference picture is less than or equal to zero.

[0462] D6. The method according to any one of schemes D1 to D5, wherein the DMVD tool includes a decoder-side motion vector refinement (DMVR) tool, or a bidirectional optical flow (BDOF) tool, or a frame rate up-conversion (FRUC) tool.

[0463] D7. The method according to any one of schemes D1 to D6, wherein the conversion generates a current block from a bitstream representation.

[0464] D8. The method according to any one of schemes D1 to D6, wherein the conversion generates a bitstream representation from the current block.

[0465] D9. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when run by the processor, cause the processor to implement the method according to any one of schemes D1 to D8.

[0466] D10. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing the method according to any one of schemes D1 to D8.

[0467] In some embodiments, the following technical solutions may be implemented:

[0468] 1. A method for video processing, comprising: parsing a string of binary values from a bitstream representation of a current block of a video, wherein the string of binary values includes a plurality of binary values representing a Generalized Bi-Directional Prediction (GBI) index of a GBI mode, and wherein at least one of the plurality of binary values is bypass-coded; and performing a conversion between the current block and the bitstream representation based on the parsed GBI index.

[0469] 2. A method for video processing, comprising: encoding a string of binary values into a bitstream representation of a current block of a video, wherein the string of binary values includes a plurality of binary values representing a Generalized Bi-Directional Prediction (GBI) index of a GBI mode, and wherein at least one of the plurality of binary values is bypass-coded; and performing a conversion between the current block and the bitstream representation based on the encoded string of binary values.

[0470] 3. The method according to scheme E1 or E2, wherein the GBI mode is configured to select weights from a set of weights to generate a bi-directional prediction signal for the current block, and wherein the set of weights includes a plurality of weights different from 1 / 2.

[0471] 4. The method according to any one of schemes E1 to E3, wherein a first binary value among the plurality of binary values is coded with at least one context, and wherein all other binary values among the plurality of binary values are bypass-coded.

[0472] 5. The method according to embodiment E4, wherein at least one context comprises one context.

[0473] 6. The method according to embodiment E4, wherein at least one context comprises three contexts.

[0474] 7. The method according to embodiment E6, wherein three contexts are defined as: ctxIdx = aboveBlockIsGBIMode + leftBlockIsGBIMode, where if the neighboring block above the current block is encoded / decoded using the GBI mode, then aboveBlockIsGBIMode = 1, otherwise it is equal to 0, and where if the neighboring block to the left of the current block is encoded / decoded using the GBI mode, then leftBlockIsGBIMode = 1, otherwise it is equal to 0, and wherein using the GBI mode comprises using unequal weights for two reference blocks of a bi - directionally predicted block.

[0475] 8. The method according to any one of embodiments E1 to E3, wherein each of the first K binary values among a plurality of binary values is encoded / decoded using at least one context, wherein all other binary values among the plurality of binary values are bypass - encoded / decoded, where K is a non - negative integer, where 0 ≤ K ≤ maxGBIIdxLen, and wherein maxGBIIdxLen is the maximum length of the plurality of binary values.

[0476] 9. The method according to embodiment E8, wherein, except for the first binary value among the first K binary values, a context is shared for the remaining binary values among the first K binary values.

[0477] 10. The method according to embodiment E8, wherein, except for the first binary value among the first K binary values, each of the remaining binary values among the first K binary values uses one context.

[0478] 11. The method according to any one of embodiments E1 to E10, wherein the transformation generates a current block from a bit - stream representation.

[0479] 12. The method according to any one of embodiments E1 to E10, wherein the transformation generates a bit - stream representation from a current block.

[0480] 13. An apparatus in a video system, comprising a processor and a non - transitory memory having instructions thereon, wherein the instructions, when run by the processor, cause the processor to implement the method according to any one of embodiments E1 to E12.

[0481] 14. A computer program product stored on a non - transitory computer - readable medium, the computer program product including program code for performing the method described in any one of E1 to E12.

[0482] 6. Example embodiments of the disclosed technology

[0483] Figure 28 is a block diagram of a video processing apparatus 2800. The apparatus 2800 can be used to implement one or more methods described herein. The apparatus 2800 can be included in a smart phone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. The apparatus 2800 can include one or more processors 2802, one or more memories 2804, and video processing hardware 2806. The (s) processor(s) 2802 can be configured to implement one or more methods described in this document (including but not limited to methods 2700, 2710, 2720, 2730, 2740, 2750, 2760, and 2770). The memory (memories) 2804 can be used to store data and code for implementing the methods and techniques described herein. The video processing hardware 2806 can be used to implement some of the techniques described in this document in hardware circuits.

[0484] In some embodiments, the video encoding / decoding method can be implemented using an apparatus implemented on a hardware platform as referred to Figure 28 as described.

[0485] Some embodiments of the disclosed technology include making a decision or determination to enable a video processing tool or mode. In one example, when a video processing tool or mode is enabled, the encoder will use or implement the tool or mode in the processing of video blocks, but not necessarily modify the resulting bitstream based on the use of the tool or mode. That is, when a video processing tool or mode is enabled based on a decision or determination, the conversion from the blocks of the video to the bitstream representation of the video will use the video processing tool or mode. In another example, when a video processing tool or mode is enabled, the decoder will process the bitstream knowing that the bitstream has been modified based on the video processing tool or mode. That is, the conversion from the bitstream representation of the video to the blocks of the video will be performed using the video processing tool or mode enabled based on the decision or determination.

[0486] Some embodiments of the disclosed technology include making a decision or determination to disable a video processing tool or mode. In one example, when a video processing tool or mode is disabled, the encoder will not use the tool or mode to convert the blocks of the video into a bitstream representation of the video. In another example, when a video processing tool or mode is disabled, the decoder will process the bitstream knowing that the bitstream has not been modified using the video processing tool or mode enabled based on the decision or determination.

[0487] Figure 29 FIG. 2900 is a block diagram illustrating an example video processing system in which various techniques disclosed herein may be implemented. Various embodiments may include some or all components of system 2900. System 2900 may include an input 2902 for receiving video content. The video content may be received in a raw or uncompressed format, e.g., 8- or 10-bit multi-component pixel values, or may be in a compressed or encoded format. Input 2902 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces (such as Ethernet, passive optical network (PON), etc.) and wireless interfaces (such as Wi-Fi or cellular interfaces).

[0488] System 2900 may include an encoding component 2904, which may implement various encoding methods described in this document. Encoding component 2904 may reduce the average bit rate of the video from input 2902 to output of encoding component 2904 to produce an encoded representation of the video. Thus, encoding techniques are sometimes referred to as video compression or video transcoding techniques. The output of encoding component 2904 may be stored or transmitted via a connected communication, as shown by component 2906. Component 2908 may use the bitstream (or encoded) representation of the stored or communicated video received at input 2902 to generate pixel values or a displayable video to be sent to display interface 2910. The process of generating a user-visible video from the bitstream representation is sometimes referred to as video decompression. Additionally, although certain video processing operations are referred to as "encoding" operations or tools, it should be understood that encoding tools or operations are used at an encoder and that corresponding decoding tools or operations that reverse the encoding results will be performed by a decoder.

[0489] Examples of a peripheral bus interface or a display interface may include a universal serial bus (USB), a high definition multimedia interface (HDMI), or a DisplayPort, etc. Examples of a storage interface include SATA (serial advanced technology attachment), PCI, IDE interface, etc. The techniques described in this document may be embodied in various electronic devices, such as a mobile phone, a laptop computer, a smartphone, or other devices capable of performing digital data processing and / or video display.

[0490] From the foregoing, it will be appreciated that, for purposes of illustration, specific embodiments of the presently disclosed techniques have been described herein, but various modifications may be made without departing from the scope of the invention. Accordingly, the presently disclosed techniques are not limited except as by the appended claims.

[0491] Embodiments of the subject matter and the functional operations described in this patent document can be implemented in various systems, digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory computer-readable medium for execution by, or to control the operation of, a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter implementing a machine-readable propagated signal, or a combination of one or more of them. The term "data processing unit" or "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, e.g., including programmable processors, computers, or multiple processors or computers. In addition to hardware, the apparatus can also include code that creates an execution environment for the computer programs being discussed, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0492] A computer program (also called a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. The program can be stored in a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program being discussed, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed to be executed on one or more computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0493] The processes and logical flows described in this specification can be performed by one or more programmable processors that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logical flows can also be performed by, or the apparatus can be implemented as, special purpose logic circuitry, e.g., an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0494] For example, a processor suitable for executing a computer program includes general and special purpose microprocessors, as well as any one or more processors of any type of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing the instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as, magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to receive data from or transfer data to one or more mass storage devices or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as, including semiconductor memory devices, such as, EPROM, EEPROM, and flash memory devices. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0495] This specification and the drawings are to be considered only as exemplary, where exemplary means illustrative.

[0496] Although this patent document contains many details, these should not be construed as limitations on any invention or the scope of what is claimed, but rather as descriptions of features that are specific to particular embodiments of a particular invention. Certain features that are described in the context of separate embodiments in this patent document may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. Additionally, although the above features may be described as acting in certain combinations and even initially claimed as such, in some cases, one or more features from a claimed combination may be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variation of a sub-combination.

[0497] Similarly, although operations are depicted in the drawings in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or sequentially, or that all of the operations shown be performed, to achieve a desired result. Additionally, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0498] Only a few embodiments and examples have been described, and other implementations, enhancements, and variations may be made based on what is described and illustrated in this patent document.

Claims

1. A method for video processing, comprising: Making a decision on selective enabling of a decoder-side motion vector derivation (DMVD) tool for a current block of a video based on at least one reference picture associated with the current block, wherein the DMVD tool derives a refinement of motion information signaled in a bitstream of the video; And Performing a conversion between the current block and the bitstream based on the decision, Wherein the DMVD tool is not enabled when determining that the at least one reference picture includes a current coded picture; Wherein the DMVD tool is a bidirectional optical flow (BDOF) tool or a frame rate up-conversion (FRUC) tool; Wherein the DMVD is disabled for a block coded in a skip mode.

2. The method according to claim 1, wherein The DMVD tool is enabled when determining that the at least one reference picture includes a first reference picture from a first reference picture list and a second reference picture from the first reference picture list.

3. The method according to claim 2, wherein A picture order count (POC) value of the first reference picture is less than a POC value of a current picture including the current block, and wherein a POC value of the second reference picture is greater than the POC value of the current picture.

4. The method according to claim 2, wherein A product of a POC difference between the POC value of the current picture including the current block and the POC values of the first reference picture and the second reference picture is less than or equal to zero.

5. The method according to any one of claims 1 to 4, wherein The conversion generates the current block from the bitstream.

6. The method according to any one of claims 1 to 4, wherein, The conversion generates the bitstream from the current block.

7. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when run by the processor, cause the processor to implement the method according to any one of claims 1 to 6.

8. A non-transitory computer-readable medium storing instructions, which, when executed by a processor, cause the processor to execute the method according to any one of claims 1 to 6.

Images