Asymmetric weighted bi-prediction merge

Abstract

Devices, systems, and methods for asymmetrically weighted bi-prediction merge in video coding are introduced. In a typical aspect, a method of video data decoding includes constructing a merge candidate for generating a current video block based on asymmetric weighting of a first reference merge candidate and a second reference merge candidate, wherein the asymmetric weighting includes a first weight and a second weight different from the first weight, and wherein the merge candidate indicates a motion vector and a corresponding reference picture; after the constructing, inserting the merge candidate into a merge candidate list, and generating the current block based on the merge candidate list.

Description

[0001] This is a divisional application of the following patent application:

[0002] Application No. 201910487915.2

[0003] Filing Date: June 5, 2019

[0004] Title of Invention: Asymmetrically weighted bi-prediction merge

[0005] Cross Reference to Related Applications

[0006] This application is timely made under 35 U.S.C. § 119(e) and / or the Paris Convention, and claims priority to and the benefit of International Patent Application No. PCT / CN2018 / 089919, filed June 5, 2018. The entire disclosure of International Patent Application No. PCT / CN2018 / 089919 is incorporated by reference herein as part of the disclosure of this application. TECHNICAL FIELD

[0007] This patent document relates generally to video coding techniques. BACKGROUND

[0008] Motion compensation is a technique in video processing that predicts a frame in a video by considering the motion of a camera and / or objects in the video, given a previous frame and / or a future frame. Motion compensation can be used in encoding and decoding of video data to achieve video compression. SUMMARY

[0009] Devices, systems, and methods related to asymmetrically weighted bi-prediction merge for motion compensation are described. More generally, the described techniques provide enhancements to the Merge mode of video coding.

[0010] In one typical aspect, a method of processing video data is provided, comprising: constructing a Merge motion candidate list for a current video block; determining a Merge candidate from the Merge motion candidate list; and disabling a coding tool if the Merge candidate is an asymmetric bi-prediction Merge candidate; wherein different weights indicated by the asymmetric bi-prediction Merge candidate are applied to a first prediction block and a second prediction block of the current video block to obtain a final prediction block.

[0011] In another exemplary aspect, there is provided an apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: construct a Merge motion candidate list for a current video block; determine a Merge candidate from the Merge motion candidate list; and disable a coding tool if the Merge candidate is an asymmetric bi-prediction Merge candidate; wherein different weights indicated by the asymmetric bi-prediction Merge candidate are applied to a first prediction block and a second prediction block of the current video block to obtain a final prediction block.

[0012] In yet another exemplary aspect, there is provided a non-transitory computer readable storage medium having instructions stored therein, the instructions causing a processor to: construct a Merge motion candidate list for a current video block; determine a Merge candidate from the Merge motion candidate list; and disable a coding tool if the Merge candidate is an asymmetric bi-prediction Merge candidate; wherein different weights indicated by the asymmetric bi-prediction Merge candidate are applied to a first prediction block and a second prediction block of the current video block to obtain a final prediction block.

[0013] In one exemplary aspect, the disclosed technology can be used to provide a method of decoding video data, comprising: constructing a Merge candidate for generating a current video block based on asymmetric weighting of a first reference Merge candidate and a second reference Merge candidate, the Merge candidate comprising a motion vector and a corresponding reference picture. In one example, the asymmetric weighting comprises a first weight applied to the first reference Merge candidate and a second weight different from the first weight applied to the second reference Merge candidate. The method further comprises, after the constructing, inserting the Merge candidate into a Merge candidate list, deriving motion information from at least the Merge candidate list, and generating the current block based on the Merge candidate list.

[0014] In another exemplary aspect, the disclosed technology can be used to provide a method of video coding using asymmetrically weighted bi-prediction Merge. The method comprises: constructing a Merge candidate based on asymmetric weighting of a first reference Merge candidate and a second reference Merge candidate, wherein the asymmetric weighting comprises a first weight and a second weight different from the first weight; after the constructing, inserting the Merge candidate into a Merge candidate list, deriving motion information from at least the Merge candidate list, and encoding a current block of video data that is inter coded based on the motion information.

[0015] In yet another exemplary aspect, the above-described method is implemented in the form of processor-executable code, and stored in a computer-readable program medium.

[0016] In yet another exemplary aspect, an apparatus is disclosed that is configured to or operable to perform the above-described method. The apparatus can include a processor programmed to implement the method.

[0017] In yet another exemplary aspect, a video decoder apparatus that can implement a method as described herein.

[0018] The above-described aspects, and other aspects and features of the disclosed technology are described in greater detail in the drawings, specification and claims. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 An example of constructing a Merge candidate list is shown.

[0020] Figure 2 An example of a location of a spatial candidate is shown.

[0021] Figure 3 An example of a pair of candidates that pass a redundancy check for a spatial Merge candidate is shown.

[0022] Figure 4A and Figure 4B An example of a location of a second prediction unit (PU) based on the size and shape of the current block is shown.

[0023] Figure 5 An example of a motion vector scaling for a temporal Merge candidate is shown.

[0024] Figure 6 An example of a location of a temporal Merge candidate is shown.

[0025] Figure 7 An example of generating a combined bi-predictive Merge candidate is shown.

[0026] Figure 8 An example of constructing a motion vector prediction candidate is shown.

[0027] Figure 9 An example of a motion vector scaling for a spatial motion vector candidate is shown.

[0028] Figure 10 An example of motion prediction using an optional temporal motion vector prediction (ATMVP) algorithm for a coding unit (CU) is shown.

[0029] Figure 11 An example of a coding unit (CU) with sub-blocks and neighboring blocks used by a spatial-temporal motion vector prediction (STMVP) algorithm is shown.

[0030] Figure 12 An example of a simplified affine motion model is shown.

[0031] Figure 13 An example of an affine motion vector field (MVF) for each sub-block is shown.

[0032] Figure 14 An example of motion vector prediction (MVP) for AF INTER affine motion mode is shown.

[0033] Figure 15A And Figure 15B An example candidate for AF Merge affine motion mode is shown.

[0034] Figure 16 An example of bilateral matching in the pattern matching motion vector derivation (PMMVD) mode, which is a special Merge mode based on the frame rate up conversion (FRUC) algorithm, is shown.

[0035] Figure 17 An example of template matching in the FRUC algorithm is shown.

[0036] Figure 18 An example of unidirectional motion estimation in the FRUC algorithm is shown.

[0037] Figure 19 An example of optical flow trajectories used by the bi-directional optical flow (BIO) algorithm is shown.

[0038] Figure 20A And Figure 20B An example snapshot of the bi-directional optical flow (BIO) algorithm using block-less extension is shown.

[0039] Figure 21 An example of the decoder-side motion vector refinement (DMVR) algorithm based on bilateral template matching is shown.

[0040] Figure 22 A flowchart of an example method of video decoding using asymmetrically weighted bi-predictive Merge according to the disclosed technology is shown.

[0041] Figure 23 A flowchart of an example method of video encoding using asymmetrically weighted bi-predictive Merge according to the disclosed technology is shown.

[0042] Figure 24 is a block diagram illustrating an example of the structure of a computer system or other control device that can be used to implement various portions of the technology disclosed herein.

[0043] Figure 25 is a block diagram illustrating an example embodiment of a mobile device that can be used to implement various portions of the technology disclosed herein. DETAILED DESCRIPTION

[0044] As the demand for high-resolution video increases, video coding methods and techniques are ubiquitous in modern technology. Video codecs, which are typically electronic circuits or software that compress or decompress digital video, are constantly being improved to provide higher coding efficiency. Video codecs convert uncompressed video into a compressed format, or vice versa. There is a complex relationship between video quality, the amount of data used to represent the video (determined by the bit rate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, and end-to-end delay (latency). Compression formats are typically compliant with a standard video compression specification, such as the High Efficiency Video Coding (HEVC) standard (also known as H.265 or MPEG-H Part 2), the Versatile Video Coding standard under finalization, or other current and / or future video coding standards.

[0045] Embodiments of the disclosed technology can be applied to existing video coding standards (e.g., HEVC, H.265) and future standards to improve run-time performance. It has a particular relationship with the Merge mode in video coding. In this document, section headings are used to improve readability of the description, and do not limit the discussion or embodiments (and / or implementations) to the respective section in any way. For ease of understanding, sections are used in this document, and the scope of the technology and embodiments disclosed in the respective sections is not limited to that section.

[0046] 1. Example of inter prediction in HEVC / H.265

[0047] Video coding standards have improved significantly over the years, and now provide, to some extent, high coding efficiency and support for higher resolutions. The latest standards such as HEVC and H.265 are based on a hybrid video coding structure, where temporal prediction plus transform coding is employed.

[0048] 1.1 Example of prediction modes

[0049] Each inter-predicted PU (prediction unit) has motion parameters for one or two reference picture lists. In some embodiments, the motion parameters include a motion vector and a reference picture index. In other embodiments, the use of one of the two reference picture lists can also be signaled using inter_pred_idc. In yet another embodiment, the motion vector can be explicitly coded as a delta relative to a predictor.

[0050] When a CU is coded with skip mode, the PU is associated with the CU and there are no significant residual coefficients, no coded motion vector delta or reference picture index. A Merge mode is specified by which the motion parameters of the current PU can be derived from neighboring PUs, including spatial and temporal candidates. Merge mode can be applied to any inter predicted PU, not only to skip mode. An alternative to Merge mode is explicit signaling of the motion parameters, where the motion vector of each reference picture list, the corresponding reference picture index and the usage of the reference picture list are explicitly signaled per PU.

[0051] When the signaling indicates that one of the two reference picture lists is to be used, the PU is generated from one sample block. This is called "uni-prediction". Uni-prediction is available for P slices and B slices.

[0052] When the signaling indicates that both reference picture lists are to be used, the PU is generated from two sample blocks. This is called "bi-prediction". Bi-prediction is only available for B slices.

[0053] 1.1.1 Embodiment of constructing Merge mode candidates

[0054] When a PU is predicted using Merge mode, an index pointing to an entry in the Merge candidate list is parsed from the bitstream and the motion information is retrieved using this index. The construction of this list can be summarized in the following steps order:

[0055] Step 1 : Initial candidate derivation

[0056] Step 1.1 : Spatial candidate derivation

[0057] Step 1.2 : Spatial candidate redundancy check

[0058] Step 1.3 : Temporal candidate derivation

[0059] Step 2 : Additional candidate insertion

[0060] Step 2.1 : Creation of bi-prediction candidates

[0061] Step 2.2 : Insertion of zero motion candidate

[0062] Figure 1An example of constructing the Merge candidate list based on the above outlined sequence of steps is shown. For spatial Merge candidate derivation, up to four Merge candidates are selected among the candidates located in the five different positions. For temporal Merge candidate derivation, up to one Merge candidate is selected among the two candidates. Since the number of candidates per PU is assumed to be constant at the decoder, additional candidates are generated when the number of candidates does not reach the maximum number of Merge candidates signaled in the slice header (maxNumMergeCand). Since the number of candidates is constant, the index of the best Merge candidate is coded using truncated unary (TU). If the size of the CU is equal to 8, all PUs of the current CU share one Merge candidate list, which is the same as the Merge candidate list for 2Nx2N prediction units.

[0063] 1.1.2 Construction of spatial Merge candidates

[0064] In the derivation of spatial Merge candidates, up to four Merge candidates are selected among the candidates located in the positions shown. The derivation order is Al, Bl, B0, A0 and B2. Position B2 is only considered if any of the positions Al, Bl, B0, A0 is not available (e.g., because it belongs to another slice or tile) or is intra coded. After adding the candidate in position Al, a redundancy check is performed for the remaining additions, which ensures that candidates with the same motion information are excluded from the list, thus improving the coding efficiency. Figure 2

[0065] To reduce the complexity of the calculations, not all possible pairs of candidates are considered in the mentioned redundancy check. Instead, only the pairs linked by the arrows in Figure 3 are considered, and a candidate is only added to the list if the corresponding candidate used for the redundancy check does not have the same motion information. Another source of copied motion information is the "second PU" related to partitions different from 2Nx2N. For example, Figure 4A and 4B describe the second PU for the N x 2N and 2N x N cases, respectively. When the current PU is partitioned into N x 2N, the candidate in position Al is not considered for the list construction. In some embodiments, adding this candidate can result in two prediction units with the same motion information, which is redundant for a coding unit that only has one PU. Likewise, when the current PU is partitioned into 2N x N, position Bl is not considered.

[0066] 1.1.3 Construction of temporal Merge candidates

[0067] ​In this step, only one candidate is added to the list. In particular, in the derivation of this temporal Merge candidate, the scaled motion vector is derived based on the collocated PU with the smallest POC difference to the current picture in the given reference picture list. The reference picture list used for deriving the collocated PU is explicitly signaled in the slice header.

[0068] Figure 5 An example of the derivation of the scaled motion vector of the temporal Merge candidate (as indicated by the dashed line) is shown, which scales from the motion vector of the collocated PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal Merge candidate is set to zero. For B slices, two motion vectors are obtained (one for reference picture list 0 and the other for reference picture list 1) and combined to make it a bi-predictive Merge candidate.

[0069] In the collocated PU (Y) belonging to the reference frame, the position of the temporal candidate is chosen between the candidates C0 and C1 as Figure 6 shown. If the PU at position C0 is not available, intra coded or outside the current CTU, position C1 is used. Otherwise, position C0 is used for the derivation of the temporal Merge candidate.

[0070] 1.1.4 Building additional types of Merge candidates

[0071] Besides the spatial-temporal Merge candidate, there are two additional types of Merge candidates: the combined bi-predictive Merge candidate and the zero Merge candidate. The combined bi-predictive Merge candidate is generated with the spatial-temporal Merge candidate. The combined bi-predictive Merge candidate is only used for B slices. The combined bi-predictive candidate is generated by combining the first reference picture list motion parameters of the initial candidate with the second reference picture list motion parameters of another candidate. If these two tuples provide different motion hypotheses, they will form a new bi-predictive candidate.

[0072] Figure 7 An example of this process is shown, where two candidates with MVL0 and refldxL0 or MVL1 and refldxL1 in the original list (710, on the left) are used to create a combined bi-predictive Merge candidate that is added to the final list (720, on the right).

[0073] Zero motion candidates are inserted to fill the rest of the entries in the Merge candidate list up to the capacity of MaxNumMergeCand. These candidates have zero spatial displacement and a reference picture index that starts from zero and increases each time a new zero motion candidate is added to the list. The number of reference frames used by these candidates is one frame for uni-prediction and two frames for bi-prediction. In some embodiments, no redundancy check is performed for these candidates.

[0074] 1.1.5 Example of motion estimation region for parallel processing

[0075] To speed up the encoding process, motion estimation can be performed in parallel, thus deriving motion vectors for all prediction units in a given region at the same time. Deriving Merge candidates from the spatial neighborhood can interfere with parallel processing, as one prediction unit cannot derive motion parameters from neighboring PUs until the relevant motion estimation is completed. To soften the balance between encoding efficiency and processing delay, a motion estimation region (MER) can be defined. The size of the MER can be signaled in the picture parameter set (PPS). When an MER is defined, Merge candidates falling into the same region are marked as unavailable and thus not considered in the list construction.

[0076] 1.2 Embodiment of motion vector prediction

[0077] Motion vector prediction exploits the spatio-temporal correlation of motion vectors with neighboring PUs, which is used for the explicit transmission of motion parameters. The motion vector candidate list is first constructed by checking the availability of the top-left temporal neighboring PU positions, removing redundant candidate positions, and adding zero vectors to make the candidate list length constant. Then, the encoder can select the best predictor from the candidate list and send the corresponding index indicating the selected candidate. Similar to the Merge index signaling, the index of the best motion vector candidate is coded using truncated unary.

[0078] 1.2.1 Example of constructing motion vector prediction candidates

[0079] Figure 8 The derivation process of motion vector prediction candidates is summarized and can be implemented using refidx as input for each reference picture list.

[0080] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For the derivation of spatial motion vector candidates, the motion vectors of each PU in the five different positions shown are used to finally derive two motion vector candidates. Figure 2 The motion vectors of each PU in the five different positions shown are used to finally derive two motion vector candidates.

[0081] For temporal motion vector candidates, one motion vector candidate is selected from two candidates that are derived based on two different collocated positions. After the first temporal motion vector list is made, duplicate motion vector candidates in the list are removed. If the number of potential candidates is greater than two, the motion vector candidates with reference picture list index greater than one in the associated reference picture list are removed from the list. If the number of temporal motion vector candidates is less than two, additional zero motion vector candidates are added in the list.

[0082] 1.2.2 Construction of spatial motion vector candidates

[0083] In deriving spatial motion vector candidates, at most two candidates are considered from five potential candidates, which are derived from PUs in the same position as the position of the motion Merge, in the previous Figure 2 The derivation order for the left side of the current PU is defined as A0, Al, and scaled A0, scaled Al. The derivation order for the top of the current PU is defined as B0, Bl, B2, scaled B0, scaled Bl, scaled B2. Thus, there are four cases for each side that can be used as motion vector candidates, two of which do not require the use of spatial scaling and two of which use spatial scaling. The four different cases are summarized as follows:

[0084] -- no spatial scaling

[0085] (1) same reference picture list, and same reference picture index (same POC)

[0086] (2) different reference picture list, but same reference picture (same POC)

[0087] -- spatial scaling

[0088] (3) same reference picture list, but different reference picture (different POC)

[0089] (4) different reference picture list, and different reference picture (different POC)

[0090] The no spatial scaling case is checked first, and then the spatial scaling allowed case is checked. Spatial scaling is considered for all when the POC is different between the reference picture of the neighboring PUs and the reference picture of the current PU, regardless of the reference picture list. If all PUs of the left side candidate are unavailable or intra coded, scaling of the above motion vector is allowed to help parallel derivation of the left and top MV candidates. Otherwise, spatial scaling of the above motion vector is not allowed.

[0091] As Figure 9For spatial scaling, the motion vectors of neighboring PUs are scaled in a similar way as in the temporal scaling case, as shown in the example in FIG. 6. One difference is that the reference picture list and index of the current PU are given as input; the actual scaling process is the same as the temporal scaling process.

[0092] 1.2.3 Construction of Temporal Motion Vector Candidates

[0093] Except for the derivation of the reference picture index, all derivation processes of the temporal Merge candidate are the same as those of the spatial motion vector candidate (as shown in the example in FIG. 5). In some embodiments, the reference picture index is signaled to the decoder. Figure 6

[0094] 2. Example of Inter Prediction Method in Joint Exploration Model (JEM)

[0095] In some embodiments, the reference software named Joint Exploration Model (JEM) is used to explore future video coding technologies. In JEM, subblock-based prediction is used in multiple coding tools, such as affine prediction, alternative temporal motion vector prediction (ATMVP), spatial-temporal motion vector prediction (STMVP), bi-directional optical flow (BIO), frame rate up conversion (FRUC), local adaptive motion vector resolution (LAMVR), overlapped block motion compensation (OBMC), local illumination compensation (LIC), and decoder-side motion vector refinement (DMVR).

[0096] 2.1 Example of Motion Vector Prediction Based on

[0097] In JEM with quadtree plus binary tree (QTBT), each CU can have at most one set of motion parameters for each prediction direction. In some embodiments, two sub-CU level motion vector prediction methods are considered in the encoder by splitting a large CU into sub-CUs and deriving the motion information of all sub-CUs of the large CU. The alternative temporal motion vector prediction (ATMVP) method allows each CU to obtain multiple sets of motion information from multiple smaller than the current CU's block in collocated reference pictures. In the spatial-temporal motion vector prediction (STMVP) method, the motion vectors of sub-CUs are derived recursively by exploiting temporal motion vector predictors and spatial neighboring motion vectors. In some embodiments, to maintain more accurate motion field for sub-CU motion prediction, motion compression of reference frames can be disabled.

[0098] 2.1.1 Example of Alternative Temporal Motion Vector Prediction (ATMVP)

[0099] In the ATMVP method, the temporal motion vector prediction (TMVP) method is modified by extracting multiple sets of motion information (including motion vectors and reference indices) from smaller than the current CU's block.

[0100] ​Figure 10 An example of ATMVP motion prediction processing of CU 1000 is shown. The ATMVP method predicts the motion vectors of sub-CUs 1001 within CU 1000 in two steps. The first step is to identify the corresponding block 1051 in the reference picture 1050 with the temporal vector. The reference picture 1050 is also referred to as the motion source picture. The second step is to split the current CU 1000 into sub-CUs 1001 and obtain the motion vector and reference index of each sub-CU from the corresponding block of each sub-CU.

[0101] In the first step, the reference picture 1050 and the corresponding block are determined by the motion information of the spatial neighboring blocks of the current CU 1000. To avoid the repeated scanning process of the neighboring blocks, the first Merge candidate in the Merge candidate list of the current CU 1000 is used. The first available motion vector and its associated reference index are set as the temporal vector and the index of the motion source picture. In this way, the corresponding block, sometimes referred to as the collocated block, can be more accurately identified compared to TMVP, which is always located at the right-bottom corner or the center position relative to the current CU.

[0102] In the second step, the corresponding block of a sub-CU 1051 is identified through the temporal vector in the motion source picture 1050 by adding the temporal vector to the coordinates of the current CU. For each sub-CU, the motion information of the sub-CU is derived using the motion information of its corresponding block (e.g., the smallest motion grid covering the center sample). After the motion information of the corresponding NxN block is identified, it is converted into the motion vector and reference index of the current sub-CU, the same as the TMVP method of HEVC, where motion scaling and other processing are applied. For example, the decoder checks whether the low-delay condition is satisfied (e.g., the POCs of all reference pictures are smaller than the POC of the current picture) and possibly uses the motion vector MVx (e.g., the motion vector corresponding to the reference picture list X) to predict the motion vector MVy (e.g., X equals 0 or 1 and Y equals 1-X) for each sub-CU.

[0103] 2.1.2 Space-time motion vector prediction (STMVP) example

[0104] In the STMVP method, the motion vectors of sub-CUs are derived recursively in a raster scan order. Figure 11 An example of one CU with four sub-blocks and neighboring blocks is shown. Consider a CU 1100 of 8x8, which includes four 4x4 sub-CUs A (1101), B (1102), C (1103), and D (1104). The neighboring 4x4 blocks in the current frame are labeled as a (1111), b (1112), c (1113), and d (1114).

[0105] Motion derivation for sub-CU A starts by identifying its two spatial neighbors. The first neighbor is the NxN block above sub-CU A 1101 (block c 1113). If this block c (1113) is not available or intra coded, other NxN blocks above sub-CU A (1101) are examined (from left to right, starting from block c 1113). The second neighbor is the block to the left of sub-CU A 1101 (block b 1112). If block b (1112) is not available or intra coded, other blocks to the left of sub-CU A 1101 are examined (from top to bottom, starting from block b 1112). The motion information obtained from each list from the neighboring blocks is scaled to the first reference frame of the given list. Next, the temporal motion vector prediction (TMVP) for sub-block A 1101 is derived following the same procedure as TMVP in HEVC. The motion information of the collocated block at block D 1104 is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors are averaged separately for each reference list. The averaged motion vector is assigned as the motion vector for the current sub-CU.

[0106] 2.1.3 Example of sub-CU motion prediction mode signaling

[0107] In some embodiments, the sub-CU mode is enabled as an additional Merge candidate mode and no additional syntax element is needed to signal this mode. Two additional Merge candidates are added to the Merge candidate list of each CU to represent the ATMVP mode and the STMVP mode. In other embodiments, up to seven Merge candidates can be used if the sequence parameter set indicates that ATMVP and STMVP are enabled. The coding logic for the additional Merge candidates is the same as that for the Merge candidates in HM, which means that two additional RD checks can be needed for each CU in P or B slice for the two additional Merge candidates. In some embodiments, such as JEM, all bi-directional n-files of the Merge index are context coded by CABAC (Context-based Adaptive Binary Arithmetic Coding). In other embodiments, such as HEVC, only the first bi-directional n-file is context coded and the remaining bi-directional n-files are context bypass coded.

[0108] 2.2 Example of adaptive motion vector resolution

[0109] In some embodiments, when use_integer_MV_flag is equal to 0 in the slice header, the motion vector difference (MVD) (between the motion vector of a PU and the predicted motion vector) is signaled in quarter-luma-sample units. In JEM, local adaptive motion vector resolution (LAMVR) is introduced. In JEM, the MVD can be coded in quarter-luma-sample, integer-luma-sample, or four-luma-sample units. The MVD resolution control is at the coding unit (CU) level, and the MVD resolution flags are conditionally signaled for each CU that has at least one non-zero MVD component.

[0110] For a CU that has at least one non-zero MVD component, a first flag will signal to indicate whether quarter-luma-sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter-luma-sample MV precision is not used, another flag signals to indicate whether integer-luma-sample MV precision or four-luma-sample MV precision is used.

[0111] When the first MVD resolution flag of a CU is zero or not coded for the CU (meaning that all MVDs in the CU are zero), the CU uses quarter-luma-sample MV resolution. When a CU uses integer-luma-sample MV precision or four-luma-sample MV precision, the MVPs in the AMVP candidate list of the CU are rounded to the corresponding precision.

[0112] In the encoder, CU-level RD checks are used to determine which MVD resolution will be used for a CU. That is, three CU-level RD checks are performed for each MVD resolution. To speed up the encoder, the following coding scheme is applied in JEM:

[0113] -- When performing RD check for a CU with normal quarter-luma-sampling MVD resolution, store the motion information of the current CU (integer-luma-sampling precision). When performing RD check for the same CU with integer-luma-sample and 4-luma-sample MVD resolutions, use the stored motion information (after rounding) as the starting point for further small-range motion vector refinement, so that the time-consuming motion estimation process is not repeated three times.

[0114] -- Conditionally invoke the RD check for a CU with 4-luma-sample MVD resolution. For a CU, when the RD check cost of integer-luma-sample MVD resolution is much larger than that of quarter-luma-sample MVD resolution, the RD check of 4-luma-sample MVD resolution for the CU will be skipped.

[0115] 2.3 Example of higher motion vector storage precision

[0116] In some embodiments, the motion vector precision is one quarter pixel (one quarter luma sample for 4:2:0 video and one eighth chroma sample). In JEM, the precision of the intra motion vector storage and Merge candidate is increased to 1 / 16 pixel. For CUs coded with Skip / Merge mode, motion compensated inter prediction is performed with higher motion vector precision (1 / 16 pixel). For CUs coded with normal AMVP mode, integer pixel or quarter-pixel motion is used, as described previously.

[0117] In some embodiments, the up-sampling interpolation filter is used as a motion compensated interpolation filter for additional fractional pixel positions in the scalable extension of HEVC (SHVC) with the same filter length and normalization factor as the HEVC motion compensation interpolation filter. In JEM, the motion vector precision for chroma components is 1 / 32 sample, and an additional interpolation filter for 1 / 32 pixel fractional positions is derived by using an average of the filters for two neighboring 1 / 16 pixel fractional positions.

[0118] 2.4 Example of affine motion compensated prediction

[0119] In HEVC, motion compensated prediction (MCP) applies only a translational motion model. However, cameras and objects can have multiple kinds of motion, such as zooming, rotation, perspective motion, and / or other irregular motion. On the other hand, JEM applies a simplified affine transform motion compensated prediction. Figure 12 An example of an affine motion field for a block 1400 described by two control point motion vectors V0and V1is shown. The motion vector field (MVF) for the block 1400 can be described by the following equation:

[0120]

[0121] As shown in Figure 12 (v 0x ,v 0y ) is the motion vector of the top-left control point, and (v 1x ,v 1y ) is the motion vector of the top-right control point. To simplify the motion compensated prediction, a subblock-based affine transform prediction can be applied. The subblock size M x N is derived as follows:

[0122]

[0123] Here, MVPreis the motion vector fractional precision (e.g., 1 / 16 in JEM). (v 2x ,v 2y ) is the motion vector of the bottom-left control point, which is calculated according to equation (1). If needed, M and N can be down-adjusted to be divisors of w and h, respectively.

[0124] Figure 13 An example of the affine motion vector field (MVF) for each sub-block of block 1500 is shown. To derive the motion vector for each M×N sub-block, the motion vector of the center sample of each sub-block can be calculated according to equation (1) and rounded to the fractional precision of the motion vector (e.g., 1 / 16 in JEM). A motion-compensated interpolation filter can then be applied to generate predictions for each sub-block using the derived motion vectors. After MCP, the high-precision motion vector of each sub-block is rounded and saved to the same precision as the normal motion vector.

[0125] In JEM, there are two affine motion modes: AF_INTER mode and AF_Merge mode. AF_INTER mode can be applied to CUs with both width and height greater than 8. In the bitstream, affine flags at the CU level are signaled to indicate whether AF_INTER mode is used. In AF_INTER mode, adjacent blocks are used to construct motion vector pairs {(v0,v1)|v0={v...}. A ,v B ,v c},v1={v D ,v E The candidate list of}}.

[0126] Figure 14 An example of motion vector prediction (MVP) for block 1600 in AF_INTER mode is shown. Figure 16 As shown, v0 is selected from the motion vectors of sub-blocks A, B, or C. Motion vectors of adjacent blocks can be scaled according to a reference list. Alternatively, the motion vectors can be scaled based on the relationship between the Picture Order Count (POC) of the adjacent block references, the POC of the current CU reference, and the POC of the current CU. The method for selecting v1 from adjacent sub-blocks D and E is similar. When the number of candidates in the candidate list is less than 2, the list is populated by copying the motion vector pairs that make up each AMVP candidate. When the candidate list is greater than 2, the candidates can first be sorted according to adjacent motion vectors (e.g., based on the similarity of two motion vectors in a candidate pair). In some implementations, the top two candidates are retained. In some embodiments, a rate distortion (RD) cost check is used to determine which motion vector pair candidate is selected as the Control Point Motion Vector Prediction (CPMVP) for the current CU. An exponent indicating the position of the CPMVP in the candidate list can be issued in the bitstream. After the CPMVP of the current affine CU is determined, affine motion estimation is applied, and the Control Point Motion Vector (CPMV) is found. Then, the difference between the CPMV and CPMVP is signaled in the bitstream.

[0127] When a CU is applied in AF_Merge mode, it takes the first block coded with affine mode from the valid neighboring reconstructed blocks. Figure 15A An example of the selection order of the candidate blocks of the current coding unit (CU) 1700 is shown. As shown, the selection order can be from the left (1701), top (1702), top-right (1703), bottom-left (1704) to the top-left (1707) of the current CU 1700. Figure 15A Figure 15B Another example of the candidate blocks of the current CU 1700 in AF_Merge mode is shown. If the neighboring bottom-left block 1701 is coded with affine mode, as shown, Figure 15B

[0128] After the CPMVs v0 and v1 of the current CU are calculated according to the affine motion model in equation (1), the MVFs of the current CU can be generated. To determine whether the current CU is coded with AF_Merge mode, an affine flag can be signaled in the bitstream when at least one neighboring block is coded with affine mode.

[0129] 2.5 Example of pattern matching motion vector derivation (PMMVD)

[0130] The PMMVD mode is a special Merge mode based on the frame rate up conversion (FRUC) method. In this mode, the motion information of a block is not signaled but derived at the decoder side.

[0131] For a CU, when its Merge flag is true, a FRUC flag can be signaled. When the FRUC flag is false, the Merge index can be signaled and the regular Merge mode is used. When the FRUC flag is true, another FRUC mode flag can be signaled to indicate which mode (e.g., bi-directional matching or template matching) is used to derive the motion information of the block.

[0132] At the encoder side, whether to use the FRUC Merge mode for a CU is decided based on the RD cost selection made on the normal Merge candidates. For example, multiple matching modes (e.g., bi-directional matching and template matching) are examined for the CU by using the RD cost selection. The mode that results in the lowest cost is further compared with other CU modes. If the FRUC matching mode is the most efficient mode, then for the CU, the FRUC flag is set to true and the related matching mode is used.

[0133] ​​Generally, there are two steps in the motion derivation process in FRUC Merge mode: first, CU-level motion search is performed, and then sub-CU-level motion optimization is performed. At the CU level, the initial motion vector of the whole CU is derived based on bi-directional matching or template matching. First, a list of MV candidates is generated, and the candidate that results in the lowest matching cost is selected as the starting point for further optimization of the CU level. Then, a local search based on bi-directional matching or template matching is performed around the starting point. The MV result of the minimum matching cost is taken as the MV value of the whole CU. Next, the motion information is further refined at the sub-CU level, with the derived CU motion vector as the starting point.

[0134] For example, the following derivation process is performed for WxH CU motion information derivation. In the first stage, the MV of the whole WxH CU is derived. In the second stage, the CU is further divided into MxM sub-CUs. The value of M is calculated according to Equation (3), and D is a predefined division depth, which is set to 3 by default in JEM. Then the MV value of each sub-CU is derived.

[0135]

[0136] Figure 16 An example of bi-directional matching used in a frame rate up conversion (FRUC) method is shown. Bi-directional matching is used to obtain the motion information of a current CU by finding the closest match between two blocks in two different reference pictures (1810, 1811) along the motion trajectory of the current CU (1800). Under the assumption of continuous motion trajectory, the motion vectors MV0 (1801) and MV1 (1802) pointing to the two reference blocks are proportional to the temporal distance between the current picture and the two reference pictures (e.g., TD0 (1803) and TD1 (1804). In some embodiments, when the current picture 1800 is temporarily located between the two reference pictures (1818, 1811) and the temporal distance of the current picture to the two reference pictures is the same, bi-directional matching becomes mirror-based bi-directional MV.

[0137] Figure 17An example of template matching used in a frame rate up conversion (FRUC) method is shown. Template matching can be used to obtain the motion information of a current CU 1900 by finding the closest match between a template (e.g., the top and / or left neighboring blocks of the current CU) in the current picture and a block (e.g., of the same size as the template) in the reference picture 1910. In addition to the FRUC Merge mode described above, template matching can also be applied to the AMVP mode. In both JEM and HEVC, AMVP has two candidates. With the template matching method, a new candidate can be derived. If the new candidate derived by template matching is different from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and then the list size is set to 2 (e.g., by removing the second existing AMVP candidate). When applied to the AMVP mode, only CU-level search is applied.

[0138] CU-level set MV candidates can include the following: (1) the initial AMVP candidate, if the current CU is in AMVP mode, (2) all Merge candidates, (3) a number of MVs in the interpolated MV field (described later), and the top and left neighboring motion vectors.

[0139] When using bi-directional matching, each valid MV of a Merge candidate can be used as input to generate a pair of MVs with the hypothesis of bi-directional matching. For example, a Merge candidate has one valid MV at reference list A (MVa, refA). Then its counterpart bi-directional MV is found in another reference list B (MVb, refB) such that refA and refB are on different sides of the current picture in time. If the reference refB in the reference list B is not available, the reference refB is determined to be a different reference from refA and its temporal distance to the current picture is the minimum distance in list B. After refB is determined, MVb is derived by scaling MVa based on the temporal distance between the current picture and refA, refB. a ) and refB. b ) respectively. refA and refB are on different sides of the current picture in time. If the reference refB in the reference list B is not available, the reference refB is determined to be a different reference from refA and its temporal distance to the current picture is the minimum distance in list B. After refB is determined, MVb is derived by scaling MVa based on the temporal distance between the current picture and refA, refB. a ) and refB. b ) respectively. refA and refB are on different sides of the current picture in time. If the reference refB in the reference list B is not available, the reference refB is determined to be a different reference from refA and its temporal distance to the current picture is the minimum distance in list B. After refB is determined, MVb is derived by scaling MVa based on the temporal distance between the current picture and refA, refB. b ) and refB. b ) respectively. refA and refB are on different sides of the current picture in time. If the reference refB in the reference list B is not available, the reference refB is determined to be a different reference from refA and its temporal distance to the current picture is the minimum distance in list B. After refB is determined, MVb is derived by scaling MVa based on the temporal distance between the current picture and refA, refB. a ) and refB. b ) respectively. refA and refB are on different sides of the current picture in time. If the reference refB in the reference list B is not available, the reference refB is determined to be a different reference from refA and its temporal distance to the current picture is the minimum distance in list B. After refB is determined, MVb is derived by scaling MVa based on the temporal distance between the current picture and refA, refB. a ) and refB. b ) respectively. refA and refB are on different sides of the current picture in time. If the reference refB in the reference list B is not available, the reference refB is determined to be a different reference from refA and its temporal distance to the current picture is the minimum distance in list B. After refB is determined, MVb is derived by scaling MVa based on the temporal distance between the current picture and refA, refB.

[0140] In some implementations, four MVs from the interpolated MV field can also be added to the CU-level candidate list. More specifically, the interpolated MVs at the locations (0, 0), (W / 2, 0), (0, H / 2) and (W / 2, H / 2) of the current CU are added. When FRUC is applied in AMVP mode, the original AMVP candidates are also added to the MV candidate set at the CU level. In some implementations, at the CU level, 15 MVs for AMVP CUs and 13 MVs for Merge CUs can be added to the candidate list.

[0141] The MV candidates set at the sub-CU level includes (1) MVs determined from the CU level search, (2) top, left, top-left and top-right neighboring MVs, (3) scaled versions of collocated MVs in the reference picture, (4) one or more ATMVP candidates (e.g., up to four) and (5) one or more STMVP candidates (e.g., up to four). The scaled MVs from the reference picture are derived as follows. Both lists of reference pictures are traversed. The MVs at the collocated locations of the sub-CU in the reference pictures are scaled to the reference of the starting CU level MV. The ATMVP and STMVP candidates can be the first four. At the sub-CU level, one or more MVs (e.g., up to 17) are added to the candidate list.

[0142] Generation of an interpolated MV field. Before encoding a frame, an interpolated motion field for the entire picture is generated based on the uni-directional ME. This motion field can then be used as MV candidates at the CU level or sub-CU level.

[0143] In some embodiments, the motion field of each reference picture in both reference lists is traversed at the 4x4 block level. Figure 18 An example of uni-directional motion estimation (ME) 2000 in the FRUC method is shown. For each 4x4 block, if the motion associated with the block goes through a 4x4 block in the current picture and the block is not assigned any interpolated motion, the motion of the reference block is scaled to the current picture according to the temporal distances TD0 and TD1 (same as the MV scaling in TMVP in HEVC) and the scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to a 4x4 block, the motion of the block is marked as unavailable in the interpolated motion field.

[0144] Interpolation and matching costs. When the motion vector points to a fractional sample position, motion compensation interpolation is needed. To reduce complexity, bilinear interpolation is used for both bi-directional matching and template matching instead of the regular 8-tap HEVC interpolation.

[0145] The calculation of the matching cost is a bit different at different steps. When selecting a candidate from the CU-level candidate set, the matching cost can be the absolute sum of difference (SAD) of bi-directional matching or template matching. After the initial MV is determined, the matching cost of bi-directional matching at the sub-CU level search is as follows:

[0146]

[0147] Here, w is a weight coefficient. In some embodiments, w can be empirically set to 4. MV and MV s indicate the current MV and the initial MV, respectively. The SAD can still be used as the matching cost for the pattern matching at the sub-CU level search.

[0148] In the FRUC mode, the MV is derived by using only the luma samples. The derived motion will be used for MC inter prediction of luma and chroma. After the MV is determined, the final MC is performed using an 8-taps interpolation filter for luma and a 4-taps interpolation filter for chroma.

[0149] MV refinement is a pattern-based MV search, with either bi-directional matching cost or template matching cost as the criterion. In JEM, two search patterns are supported—unrestricted center-biased diamond search (UCBDS) and adaptive cross search, at the CU level and sub-CU level, respectively, for MV refinement. For both CU-level and sub-CU-level MV refinement, MVs are searched directly at quarter luma sample precision, followed by an eighth luma sample MV refinement. The search range for MV refinement at both CU and sub-CU steps is set to 8 luma samples.

[0150] In the bi-directional matching Merge mode, bi-directional prediction is applied because the motion information of the CU is derived based on the closest match between two blocks on the current CU motion trajectory in two different reference pictures. In the template matching Merge mode, the encoder can choose for the CU either the uni-prediction from List 0, uni-prediction from List 1, or bi-prediction. The choice can be based on the template matching cost as follows:

[0151] If costBi <= factor * min(cost0, cost1)

[0152] then bi-prediction is used;

[0153] Else if cost0 <= cost1

[0154] then uni-prediction in List 0 is used;

[0155] Else,

[0156] uni-prediction in List 1 is used;

[0157] Here, cost0 is the SAD of template matching for list 0, cost1 is the SAD of template matching for list 2, and cost bidirectional is the SAD of template matching for bidirectional template matching. For example, when the value of factor equals 1.25, it means that the selection process is offset towards bidirectional prediction. Inter-frame prediction direction selection can be applied to CU-level template matching processing.

[0158] 2.6 Examples of Bidirectional Optical Flow (BIO)

[0159] The Bidirectional Optical Flow (BIO) method refines sample-oriented motion on top of block motion compensation for bidirectional prediction. In some implementations, sample-level motion refinement does not use signaling.

[0160] Let I (k) The brightness value after block motion compensation to the reference k (k = 0, 1), and These are the horizontal and vertical components of the I(k) gradient, respectively. Assuming the optical flow is effective, the motion vector field (v...) x ,v y The following formula is given:

[0161]

[0162] Combining this optical flow equation with the Hermite interpolation of each sample's trajectory yields a unique third-order polynomial that simultaneously matches the function value I at its end. (k) and its derivative The value of this polynomial at t=0 is the BIO prediction:

[0163]

[0164] Figure 19 An example optical flow trajectory in the Two-Way Optical Flow (BIO) method is shown. Here, τ0 and τ1 represent the distances to the reference frames. The distances τ0 and τ1 are calculated based on the POCs of Ref0 and Ref1: τ0 = POC(current) - POC(Ref0), τ1 = POC(Ref1) - POC(current). If both predictions originate from the same time direction (both from the past or both from the future), the signs are different (e.g., τ0·τ1 < 0). In this case, if the predictions do not originate from the same time point (e.g., τ0 ≠ τ1), BIO is applied. Both reference regions have non-zero motion (e.g., MVx0, MVy0, MVx1, MVy1 ≠ 0), and the block motion vector is proportional to the time distance (e.g., MVx0 / MVx1 = MVy0 / MVy1 = -τ0 / τ1).

[0165] The motion vector field (v) is determined by minimizing the difference Δ between the values ​​at points A and B. x ,vy ). Figures 20A-20B An example is shown where the motion trajectory intersects the reference frame plane. For Δ, the model uses only the first linear term of the local Taylor expansion:

[0166]

[0167] All values in the above equations depend on the sample position, denoted as (i', j'). Assuming that the motion is consistent in the local surrounding area, Δ can be minimized within a (2M+1)x(2M+1) square window Ω centered at the current prediction point (i, j), where M equals 2:

[0168]

[0169] For this optimization problem, JEM uses a simplified approach, first minimizing in the vertical direction and then in the horizontal direction. The results are as follows:

[0170]

[0171]

[0172] where

[0173]

[0174] To avoid division by zero or very small values, a regularization parameter r and m can be introduced in equation (9) and equation (10).

[0175] r = 500 · 4 d-8 Equation (12)

[0176] m = 700 · 4 d-8 Equation (13)

[0177] Here, d is the bit depth of the video samples.

[0178] To make the memory access for BIO the same as for regular bi-prediction motion compensation, all prediction and gradient values I (k) , Figure 20A An example of access positions outside the block 2200 is shown. As Figure 20A shown in equation (11), a (2M+1)x(2M+1) square window Ω centered at the current prediction point on the prediction block boundary requires access to positions outside the block. In JEM, the values I (k) , outside the block are set equal to the closest available value inside the block. This can be implemented, for example, as a padding region 2201, as Figure 20B shown.

[0179] Using BIO, the motion field of each sample can be refined. To reduce the computational complexity, a block-based BIO is adopted in JEM. The motion refinement can be computed based on 4x4 blocks. In the block-based BIO, the s n values in equation (11) for all samples in a 4x4 block can be aggregated, and then the aggregated value of s n is used to derive the BIO motion vector offset for the 4x4 block. More specifically, the following equation can be used for the block-based BIO derivation:

[0180]

[0181] Here, b k denotes the group of samples belonging to the k-th 4x4 block of the prediction block. The s n in equation (9) and equation (10) is replaced by ((s n,bk )>>4) to derive the associated motion vector offset.

[0182] In some cases, the MV group of BIO can be unreliable due to noise or irregular motion. Therefore, in BIO, the size of the MV group is fixed to a threshold. The threshold is determined based on whether the reference pictures of the current picture are all from one direction. For example, if all the reference pictures of the current picture are from one direction, the value of the threshold is set to 12x2 14 -d , otherwise it is set to 12x2 13-d .

[0183] The gradients of BIO can be computed simultaneously through motion compensation interpolation using operations consistent with HEVC motion compensation processing (e.g., 2D separable finite impulse response (FIR)). In some embodiments, the input to the 2D separable FIR is the same reference frame as the motion compensation processing, as well as the fractional position (fracX, fracY) according to the fractional part of the block motion vector. For the horizontal gradient is first interpolated vertically using a BIO filter S that corresponds to the fractional position fracY with a descaling scale shift of d-8. Then a gradient filter BIO filter G is applied in the horizontal direction that corresponds to the fractional position fracX with a descaling scale shift of 18-d. For the vertical gradient A gradient filter, BIOfilterG, corresponding to the fractional position fracY with a d-8 descaling scale shift is applied in the vertical direction. Then, a signal substitution is performed using a gradient filter, BIOfilterS, corresponding to the fractional position fracX with a descaling scale shift 18-d in the horizontal direction. The interpolation filters used for gradient calculation, BIOfilterG, and signal substitution, BIOfilterS, can be shorter (e.g., 6-tap) to keep reasonable complexity. Table 1 shows example filters that can be used in BIO to calculate gradients for different fractional positions of the block motion vector. Table 2 shows example interpolation filters that can be used in BIO for prediction signal generation.

[0184] Table 1 Example filters for gradient calculation in BIO

[0185] Fractional pixel position Interpolation filter for gradients (BIOfilterG) 0 {8,-39,-3,46,-17,5} 1 / 16 {8,-32,-13,50,-18,5} 1 / 8 {7,-27,-20,54,-19,5} 3 / 16 {6,-21,-29,57,-18,5} 1 / 4 {4,-17,-36,60,-15,4} 5 / 16 {3,-9,-44,61,-15,4} 3 / 8 {1,-4,-48,61,-13,3} 7 / 16 {0,1,-54,60,-9,2} 1 / 2 {-1,4,-57,57,-4,1}

[0186] Table 2 Example interpolation filters for prediction signal generation in BIO

[0187] Fractional pixel position Interpolation filter for prediction signal (BIOfilterS) 0 {0,0,64,0,0,0} 1 / 16 {1,-3,64,4,-2,0} 1 / 8 {1,-6,62,9,-3,1} 3 / 16 {2,-8,60,14,-5,1} 1 / 4 {2,-9,57,19,-7,2} 5 / 16 {3,-10,53,24,-8,2} 3 / 8 {3,-11,50,29,-9,2} 7 / 16 {3,-11,44,35,-10,3} 1 / 2 {3,-10,35,44,-11,3}

[0188] In JEM, BIO can be applied to all bi-predicted blocks when the two predictions come from different reference pictures. When local illumination compensation (LIC) is enabled for a CU, BIO can be disabled.

[0189] In some embodiments, OBMC is applied to a block after normal MC processing. To reduce the computational complexity, BIO can not be applied during OBMC processing. This means that during OBMC processing, BIO is applied to the MC processing of a block when its own MV is used, while BIO is not applied to the MC processing of a block when the MV of a neighboring block is used.

[0190] 2.7 Example of decoder-side motion vector refinement (DMVR)

[0191] In bi-prediction operation, for the prediction of a block region, two prediction blocks formed by motion vectors (MVs) of list 0 and list 1, respectively, are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of bi-prediction are further refined by a bilateral template matching process. The bilateral template matching applied in the decoder is used to perform a distortion-based search between the bilateral template and the reconstructed samples in the reference picture in order to obtain refined MVs without transmitting additional motion information.

[0192] In DMVR, a bilateral template is generated as a weighted combination (i.e., average) of two prediction blocks, one from the initial MV0 of List 0 and the other from MV1 of List 1, as shown in Figure 21 The template matching operation includes computing a cost metric between the generated template and a sample region (around the initial prediction block) in the reference picture. For each of the two reference pictures, the MV that produces the minimum template cost is considered as the updated MV of that list to replace the original MV. In JEM, nine MV candidates are searched for each list. The nine MV candidates include the original MV and 8 peripheral MVs, which have one luma sample offset in the horizontal or vertical direction or both from the original MV. Finally, the final bi-prediction result is generated using the two new MVs (i.e., MV0' and MV1') as shown in Figure 21 The sum of absolute differences (SAD) is used as the cost metric.

[0193] DMVR is applied to the bi-predictive Merge mode without transmitting additional syntax elements, where one MV comes from a past reference picture and the other MV comes from a future reference picture. In JEM, DMVR is not applied when LIC, affine motion, FRUC, or sub-CU Merge candidate is enabled for a CU.

[0194] 3. Embodiment of the correlation method using asymmetric weighting

[0195] In existing implementations, a generalized bi-prediction (G-bi) for inter coding is proposed as shown in Equation (15):

[0196] P[x] = (1 - w) - P0[x + v0] + w - P1[x + v i ] Equation (15)

[0197] Here, w is the weight value, x is the coordinate of the current block, P0 and P1 are the reference pictures in List 0 and List 1, respectively, and v0 and v1 are the motion vectors in P0 and P1. The candidate weight set in G-bi mode includes a total of 7 weights (W = {-1 / 4, 1 / 4, 3 / 8, 1 / 2, 5 / 8, 3 / 4, 5 / 4}), including 0.5 corresponding to the traditional bi-prediction mode. The index indicating the entry position of the weight value in the candidate weight set is signaled. At most one index is signaled for each CU, and the corresponding weight value is shared in all PUs and all color components in that CU.

[0198] To reduce the signaling overhead of the exponent, each coding unit (CU) is assigned at most one w, which is shared among all its bi-predicted units (PUs). If the CU contains at least one bi-predicted PU that requires signaling of a motion vector difference (MVD), the exponent w is explicitly signaled. Otherwise, no signaling of the exponent is needed. The following rules are then applied to determine the weight value for each PU in the CU:

[0199] For each bi-predicted PU that requires signaling of a MVD (i.e., normal and affine prediction modes), its weight value is set equal to the explicitly signaled w.

[0200] For each bi-predicted PU that is coded using Merge mode, advanced temporal motion vector prediction, or affine Merge mode, its weight value w is directly inferred from the weight value of the associated Merge candidate.

[0201] For the remaining bi-predicted PUs, their weight values are set equal to the default weight value, i.e., 0.5.

[0202] However, this method makes a complex selection of different weight values in the AMVP mode, and the encoding time is increased by 300% compared to the conventional method. In addition, the signaling cost can be high because it explicitly signals the weight exponent for the AMVP case.

[0203] 4. An example method of asymmetrically weighted bi-predictive Merge

[0204] Figure 22 A flowchart showing an example method 2200 of video decoding using asymmetrically weighted bi-predictive Merge is shown. The method 2200 includes receiving a current block of video data that is inter-coded. The current block can be received from a video bitstream received over a network connection or from a locally stored compressed video file. For example, the decoding process can perform parsing on the video bitstream to arrive at the bits corresponding to the current block during the parsing process.

[0205] The method 2200 includes constructing, at step 2210, a Merge candidate for generating the current video block based on asymmetric weighting of a first reference Merge candidate and a second reference Merge candidate. In some embodiments, the asymmetric weighting includes a first weight and a second weight that is different from the first weight. In some embodiments, the Merge candidate indicates a motion vector and a corresponding reference picture. In some embodiments, the asymmetric weighting for video decoding is based on a model in Equation (16) as described below.

[0206] The method 2200 includes inserting, at step 2220, the Merge candidate into a Merge candidate list after the constructing.

[0207] In some other implementations, the method 2200 can include deriving motion information from the at least the Merge candidate list. For example, the motion information can include motion vectors (and corresponding reference frames).

[0208] The method 2200 includes generating, at step 2230, the current block based on the Merge candidate list. In some embodiments of the disclosed technology, the generating can include decoding the current block based on the motion information, the motion vectors, and the corresponding reference pictures. The decoding process can use motion compensation, in which pixel values of the current block are computed based on the motion vectors and pixel values of corresponding blocks of the reference pictures.

[0209] Figure 23 A flowchart of an exemplary method of video coding using asymmetrically weighted bi-predictive Merge is shown. The method 2300 includes constructing, at step 2310, a Merge candidate based on asymmetric weighting of a first reference Merge candidate and a second reference Merge candidate. In some embodiments, the asymmetric weighting includes a first weight and a second weight different from the first weight. In some embodiments, the asymmetric weighting for video coding is based on a model in equation (16) as described below.

[0210] The method 2300 includes inserting, at step 2320, the Merge candidate into a Merge candidate list after the constructing.

[0211] The method 2300 includes deriving, at step 2330, motion information from the at least the Merge candidate list.

[0212] The method 2300 includes encoding, at step 2340, a current block of video data that is inter coded based on the motion information.

[0213] The methods 2200 and 2300 can further include different embodiments of asymmetric weighting of the first reference Merge candidate and the second reference Merge candidate. In some embodiments, both the first reference Merge candidate and the second reference Merge candidate can be bi-directional candidates, such that each candidate can include two motion vectors and corresponding motion information. In other embodiments, only one of the first or second reference Merge candidates can be a bi-directional candidate.

[0214] In some embodiments, the Merge candidate is based on asymmetric weighting of two different motion vectors (and corresponding motion information) from the first reference Merge candidate. In other embodiments, the Merge candidate is based on asymmetric weighting of two different motion vectors from the second reference Merge candidate. In other embodiments, the Merge candidate is based on asymmetric weighting of motion vectors from the first and second reference Merge candidates.

[0215] The methods 2200 and 2300 described in Figure 22 and Figure 23 may further include using the following model for bi-prediction:

[0216] P[x] = a - P0[x + v0] + b - P1[x + v i ] Equation (16)

[0217] Here, P i [x + v i ] is the prediction value associated with the pixel located at x in the reference picture list i, and the weights (a, b) are defined as symmetric if a equals b, otherwise (a does not equal b), the weights are defined as asymmetric. In some embodiments, a + b = 1.

[0218] In some embodiments, the asymmetrically weighted bi-prediction Merge candidate can derive the same motion information as the Merge candidate in existing designs such as HEVC and JEM, including the prediction direction, reference index, motion vector (and weights if the general bi-prediction tool is enabled). However, unlike the existing Merge candidate that applies symmetrically weighted bi-prediction, the asymmetrically weighted bi-prediction is used in the construction process of the Merge candidate list (if GBI is not enabled), or more Merge candidates are considered by applying different asymmetrically weighted (if GBI is enabled).

[0219] The following examples illustrate various implementations and motion compensation algorithms for generating bi-prediction Merge candidates using asymmetric weighting. The examples described below can be incorporated in the context of the methods described above, such as the methods 2200 and 2300, which can be implemented on a decoder and an encoder, respectively.

[0220] Example 1 Asymmetric weighting can be applied to one, some, or all of the existing Merge candidates (or future newly designed Merge candidates). For example, spatial Merge candidates, temporal Merge candidates, additional Merge candidates, ATMVP, STMVP, PMMVD, etc. can be used to generate asymmetrically weighted bi-prediction Merge candidates, which will be further inserted into the Merge list.

[0221] (a) In one example, asymmetric weighting can be applied to the first N available bi-prediction Merge candidates, where N is a positive integer.

[0222] (b) In one example, asymmetric weighting can only be applied to N bi-directional Merge candidates that have asymmetric picture order count (POC) (order picture) difference between the current picture and its two reference pictures. Let AbsPOCDiff(x,y) denote the function that computes the absolute POC difference between pictures x and y, let P0 and P1 denote the reference pictures from list 0 and list 1, respectively, and let P c denote the current picture. If AbsPOCDiff(P0, P c ) is not equal to AbsPOCDiff(P1, P c ), then the POC difference is defined as asymmetric.

[0223] (c) In one example, only certain types of Merge candidates (e.g., advanced temporal motion vector prediction or affine Merge mode) can be selected to apply asymmetric weighting.

[0224] (d) In one example, new Merge candidates can be derived by applying asymmetric weighting to Merge candidates that have symmetric weights. If a Merge candidate that is derived with existing methods already has asymmetric weighting, such as a spatial Merge candidate with weights inherited from neighboring blocks, then this spatial Merge candidate is not selected to apply asymmetric weighting. Alternatively, in another aspect, new Merge candidates can be derived by applying different asymmetric weightings to Merge candidates that have asymmetric weights, such as spatial Merge candidates or temporal Merge candidates that are derived with existing methods.

[0225] (e) When generating asymmetrically weighted bi-directional prediction Merge candidates, the weight value w in equation (16) is subject to the following constraints:

[0226] (i) When AbsPOCDiff(P c , P0) < AbsPOCDiff(P c , P1), only weight values greater than 1 / 2 can be assigned to a.

[0227] (ii) When AbsPOCDiff(P c , P0) > AbsPOCDiff(P c , P1), only weight values greater than 1 / 2 can be assigned to b.

[0228] Example 2 One or a set of asymmetric weightings can be applied to the selected N bi-directional Merge candidates.

[0229] (a) In one example, individual asymmetric (a, b) is applied to each of the N Merge candidates. Different (a, b) can be applied to different Merge candidates.

[0230] (b) In one example, several asymmetric (a, b) can be applied to each of the N Merge candidates. Some candidates can share the same (a, b).

[0231] (c) In one example, several asymmetric (a, b) can be applied to the first M (M < N) Merge candidates, and only one asymmetric (a, b) is applied to the remaining N - M Merge candidates. For example, M is set equal to 1.

[0232] Example 3 The asymmetric weighting set can be signaled to the decoder in VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), or slice header segment. Alternatively, the asymmetric weighting can be implicitly fixed in the encoder and decoder.

[0233] (a) In one example, the asymmetric weighting of a includes {1 / 16, 2 / 16, 3 / 16, 4 / 16, 5 / 16, 6 / 16, 7 / 16, 9 / 16, 10 / 16, 11 / 16, 12 / 16, 13 / 16, 14 / 16, 15 / 16}.

[0234] (b) In one example, the asymmetric weight of a includes {1 / 8, 2 / 8, 3 / 8, 5 / 8, 6 / 8, 7 / 8}.

[0235] (c) In a more general case, the asymmetric weighting of a includes {1 / 2 N , 2 / 2 N , …, 2 N -2 / 2 N , 2 N -1 / 2 N}, where N is a positive integer, such as 2, 3, 4, and 5.

[0236] (d) In one example, the asymmetric weighting of a and b can also be negative.

[0237] Example 4 The following rules are followed when inserting the asymmetrically weighted bi-predictive Merge candidate into the Merge list:

[0238] (a) In one example, the same rule can be applied to all blocks in a slice / tile / picture / sequence. Alternatively, the order can be adjusted from block to block.

[0239] (b) If asymmetrically weighted bi-predictive Merge candidate A is generated from symmetrically weighted bi-predictive Merge candidate B, then insert A after (possibly not directly after) B in the Merge candidate list.

[0240] (c) Assuming asymmetrically weighted bi-predictive Merge candidates A1 and A2 are generated from symmetrically weighted bi-predictive Merge candidates B1 and B2 respectively, if B1 precedes B2 in the Merge list, then A1 should also precede A2 (possibly not directly precede A2) in the Merge list, and vice versa.

[0241] (d) If multiple asymmetrically weighted bi-predictive Merge candidates with weights (a1, b1), (a2, b2), …, (aNA, bNA) are generated from one symmetrically weighted bi-predictive Merge candidate, then insert them in a given order, e.g., in ascending order of abs(ai - 1 / 2), where abs(x) computes the absolute value of x, and 1 <= i <= NA. If abs(ai - 1 / 2) is equal to abs(aj - 1 / 2), then they can be inserted in any order.

[0242] (e) In one example, two asymmetrically weighted bi-predictive Merge candidates with weight values (1 / 4, 3 / 4) and (3 / 4, 1 / 4) are inserted right after the first bi-predictive Merge candidate.

[0243] (f) In one example, the newly generated weighted bi-predictive Merge candidate can be added after some Merge candidate derived with existing methods, e.g., right after a TMVP or ATMVP or STMVP or joint bi-predictive Merge candidate.

[0244] (g) Two Merge candidate lists can be constructed first, one using existing methods and the other by applying asymmetric weighting to the first list or some Merge candidates in the first list. In this case, different orders of the two candidate lists can be invoked for different blocks.

[0245] Example 5 The weight values are stored for the blocks coded using bi-predictive Merge mode and can be reused by the blocks coded later.

[0246] (a) In one example, the asymmetric weighting values of the spatial / temporal merge candidates can be inherited from the weight values of the corresponding spatial / temporal neighboring blocks in the merge list construction process. In this case, these merge candidates are already asymmetrically weighted bi-predictive merge candidates themselves. In one example, these candidates are not used to generate new asymmetrically weighted bi-predictive merge candidates. Only symmetrically weighted ones can be used to generate asymmetrically weighted merge candidates. Alternatively, these candidates can also be used to generate new asymmetrically weighted bi-predictive merge candidates but with different weighting values.

[0247] (b) In one example, the asymmetric weight values are not inherited in the merge list construction process.

[0248] (c) The above methods can also apply when the general bi-prediction tool is enabled.

[0249] Example 6 For asymmetrically weighted bi-predictive merge candidates, the maximum merge list length is increased by L, where L is greater than or equal to 0.

[0250] (a) If at most T asymmetrically weighted bi-predictive merge candidates can be added to the merge list, then L <= T.

[0251] Example 7 Integration of asymmetrically weighted bi-predictive merge candidates with existing coding tools.

[0252] (a) When weighted prediction, local illumination compensation or bi-directional optical flow is applied, the asymmetrically weighted bi-predictive merge candidates are not inserted into the merge candidate list.

[0253] (b) The asymmetrically weighted bi-predictive merge candidates are disabled for DMVR.

[0254] (c) Alternatively, the asymmetrically weighted bi-predictive merge candidates are enabled for DMVR and the template P is generated as shown in equation (16).

[0255] (i) After motion refinement and motion compensation using refined motion information (in each reference list), symmetrically weighted averaging applies to the final bi-prediction.

[0256] (ii) After motion refinement and motion compensation using refined motion information (in each reference list), asymmetrically weighted averaging (same as the asymmetrically weighted bi-predictive merge candidates) applies to the final bi-prediction.

[0257] Example 8Whether and how to insert asymmetrically weighted bi-prediction Merge candidate into Merge candidate list can depend on the size and / or shape of the current block, and / or coding block information.

[0258] (a) In one example, assuming the current block size is M x N, if M x N <= T, then asymmetrically weighted bi-prediction Merge candidate is not inserted into Merge candidate list; or if M x N > T, then asymmetrically weighted bi-prediction Merge candidate is not inserted into Merge candidate list. T is an integer such as 32 or 64.

[0259] (b) Or, it can only apply when M is equal to N or M is not equal to N.

[0260] (c) Or, it can depend on the block mode, such as non-affine mode.

[0261] (d) Or, it can also depend on the total number of Merge candidates (excluding combined bi-prediction Merge candidate and / or zero motion Merge candidate) derived from the existing design. If the total number of allowed Merge candidates is reached, then asymmetrically weighted bi-prediction Merge candidate does not need to be added.

[0262] Example 9 Asymmetrically weighted bi-prediction can be applied to all components, including Y, Cb, Cr or R, G, B. Or, it only applies to one or some components. For example, it only applies to Y component.

[0263] Example 10 If one or more of the following conditions are met, asymmetrically weighted bi-prediction Merge candidate with weight values (a1, b1) has higher priority (or in other words, is inserted into Merge candidate list closer to the front of Merge candidate list) than asymmetrically weighted bi-prediction Merge candidate with the same motion information (MV0 with respect to reference picture R0, MV1 with respect to reference picture R1) but different weight values (a2, b2) (and a1 - b1 > a2 - b2).

[0264] (a) The quantization parameter (QP) of R0 is lower than the QP of R1

[0265] (b) R0 is closer to the current picture than R1

[0266] (c) The QP of the reference block pointed by MV0 is lower than the QP of the reference block pointed by MV1

[0267] Example 11 It should be noted that the proposed method also applies to multiple hypothesis motion prediction.

[0268] (a) In one example, for multiple prediction blocks, different weights can be applied to each prediction block.

[0269] (b) Or, if bi-prediction, for each reference picture list, two temporary prediction blocks can be first obtained (can be generated from multiple reference blocks associated with one reference picture list), and two weights can be further applied.

[0270] (c) Or, if uni-prediction but from two or more reference blocks, asymmetric weighting can also be applied.

[0271] Example 12 Pruning process in merge mode can also consider weights in addition to motion information.

[0272] 5. Example Implementations of the Disclosed Technology

[0273] Figure 24 is a block diagram illustrating an example of a computer system or other control device architecture that can be used to implement various portions of the disclosed technology, including, but not limited to, methods 2200 and 2300. In Figure 24 In particular, computer system 2400 includes one or more processors 2405 and memory 2410 connected by an interconnect 2425. Interconnect 2425 can represent any one or more single physical buses, point to point connections, or both, connected by appropriate bridges, adapters, or controllers. Therefore, interconnect 2425 can include, for example, a system bus, a peripheral component interconnect (PCI) bus, a HyperTransport, or Industry Standard Architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 674 bus (sometimes referred to as "Firewire").

[0274] Processor 2405 can include a central processing unit (CPU) to control, for example, the overall operation of the host. In some embodiments, processor 2405 accomplishes this by executing software or firmware stored in memory 2410. Processor 2405 can be or include one or more programmable general- purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), programmable logic devices (PLDs), or the like, or a combination of these devices.

[0275] Memory 2410 can be or include a main memory of the computer system. Memory 2410 represents any appropriate form of random access memory (RAM), read only memory (ROM), flash memory, or the like, or a combination of these devices. In use, memory 2410 can contain, among other things, a set of machine instructions that, when executed by processor 2405, cause processor 2405 to perform operations to implement embodiments of the disclosed technology.

[0276] Also connected to processor 2405 by interconnect 2425 is (optional) network adapter 2415. Network adapter 2415 provides computer system 2400 with the ability to communicate with remote devices, such as storage clients and / or other storage servers, and can be, for example, an Ethernet adapter or a Fibre Channel adapter.

[0277] Figure 25 A block diagram illustrating an example embodiment of a mobile device 2500 that can be used to implement various portions of the disclosed technology, including but not limited to methods 2200 and 2300, is shown. Mobile device 2500 can be a laptop, a smartphone, a tablet, a video camera, or other device capable of processing video. Mobile device 2500 includes a processor or controller 2501 to process data and a memory 2502 in communication with processor 2501 to store and / or buffer data. For example, processor 2501 can include a central processing unit (CPU) or a microcontroller unit (MCU). In some implementations, processor 2501 can include a field programmable gate array (FPGA). In some implementations, mobile device 2500 includes or is in communication with a graphics processing unit (GPU), a video processing unit (VPU), and / or a wireless communication unit to implement various visual and / or communication data processing functions of a smartphone device. For example, memory 2502 can include and store processor executable code that, when executed by processor 2501, configures mobile device 2500 to perform various operations, such as receiving information, commands, and / or data, processing information and data, and sending or providing processed information / data to another data device, such as an actuator or an external display.

[0278] To support the various functions of the mobile device 2500, the memory 2502 can store information and data, such as instructions, software, values, images, and other data used or generated by the processor 2501. For example, the memory 2502 can be implemented using various types of Random Access Memory (RAM) devices, Read Only Memory (ROM) devices, flash memory devices, and other suitable storage media. In some implementations, the mobile device 2500 includes an input / output (I / O) unit 2503 to interface the processor 2501 and / or the memory 2502 with other modules, units, or devices. For example, the I / O unit 2503 can interface the processor 2501 and the memory 2502 to utilize various wireless interfaces compatible with typical data communication standards, such as between one or more computers in the cloud and a user device. In some implementations, the mobile device 2500 can interface with other devices using wired connections through the I / O unit 2503. The mobile device 2500 can also be connected with other external interfaces, such as data storage, and / or visual or audio display devices 2504 to retrieve and transmit data and information that can be processed by the processor, stored by the memory, or displayed by the display devices 2504 or output units of external devices. For example, the display devices 2504 can display video frames containing blocks (CUs, PUs, or TUs) that have applied Intra Block Copy based on whether the blocks were encoded using a motion compensation algorithm.

[0279] In some embodiments, a video decoder device can implement a method of video decoding using asymmetric weighted bi-prediction Merge as described herein. The method can be similar to the method 2200 described above.

[0280] In some embodiments, a decoder-side method of video decoding can use asymmetric weighted bi-prediction Merge and improve video quality by receiving a current block of video data that is inter coded; constructing a Merge candidate based on asymmetric weighting of a first reference Merge candidate and a second reference Merge candidate, wherein the asymmetric weighting includes a first weight and a second weight that is different from the first weight; inserting the Merge candidate into a Merge candidate list after the constructing; deriving motion information from at least the Merge candidate list; and decoding the current block based on the motion information, a motion vector, and a corresponding reference picture.

[0281] In some embodiments, a method of video decoding can be implemented using a decoding device implemented on a hardware platform as described with respect to Figure 24 and Figure 25 In some embodiments, video decoding operations can be performed by a video encoder during an encoding process in which the video encoder determines reconstructed video pictures as a decoder would. In some embodiments, a method of video decoding can be implemented using a decoding device implemented on a hardware platform as described with respect to

[0282] From the foregoing, it will be appreciated that specific embodiments of the disclosed technology have been described herein for purposes of illustration, but well-known modifications can be made by those skilled in the art. Accordingly, the disclosed technology is not limited to that precisely as shown and described. In addition, it is to be understood that the present technology can assume various alternative variations and step sequences, except where expressly

[0283] Implementations of the subject matter described in this specification can be implemented in a variety of systems, digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible and non-transitory computer-readable medium for execution by, or to control the operation of, data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine- readable propagated signal, or a combination of one or more of them. The term “data processing apparatus” or “data processing device” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0284] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and are interconnected by a communication network.

[0285] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit), and that apparatus can also be implemented as special purpose logic circuitry. Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions from a memory or like storage device or from a computer-readable medium. The instructions can include one or more programs to implement methods described herein such as the procedures of a data processing application. Generally, a computer will also include, or be operatively coupled to receive data from, or transfer data to, or both, one or more machine- readable storage devices or storage media. Non-limiting examples of

[0286] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0287] It is intended that the specification and examples be considered exemplary only, with the exemplary meaning one of skill in the art being taught to follow in the examples as being one of skill in the art would employ in the performance of the claimed application. As used herein, "or" is intended to mean "and / or" unless the context clearly indicates otherwise.

[0288] While this patent document contains many specifics, these should not be construed as limiting the scope of any invention or claim, but as merely descriptive of some embodiments thereof. Some features described in the context of separate embodiments can also be implemented within a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in a sub-combination of embodiments, or in a combination of embodiments. Furthermore, although the foregoing features can be described in the context of only a single embodiment, various features of the embodiments can be combined in any suitable sub-combination, or in a combination of sub-combinations.

[0289] Likewise, a variety of inventive processes are described and illustrated herein in the context of example embodiments. However, the exemplary processes are not limited to the specific sequences, or order, of steps, or inferences, where defined, set forth in the requirements, as such precise sequences can vary. Further, some steps or inferences can be performed at the same time or concurrently, equivalents should also be considered. Additionally, unless otherwise specified, steps or inferences can be performed in any order or sequence.

[0290] Only a few implementations and examples are described and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A method for coding video data, comprising: for a conversion between a current video block of a video and a bitstream of the video, constructing a Merge motion candidate list for the current video block; determining a Merge candidate from the Merge motion candidate list; and performing the conversion based on the determination, wherein at least one coding tool is disabled when the Merge candidate is an asymmetric bi-prediction Merge candidate, wherein different weights indicated by the asymmetric bi-prediction Merge candidate are applied to first and second prediction blocks of the current video block to obtain a final prediction block, and wherein the at least one coding tool comprises a decoder-side motion vector refinement, DMVR.

2. The method of claim 1, wherein, in response to the Merge candidate not being an asymmetric bi-prediction Merge candidate and the DMVR being enabled, the method further comprises: refining motion information derived from the Merge candidate and performing motion compensation for the current video block to obtain two prediction blocks; and applying the same weight to the two prediction blocks to obtain a final prediction block.

3. The method of claim 1, wherein, the coding tool comprises at least one of: weighted prediction, bi-directional optical flow for refining prediction blocks of bi-prediction.

4. The method of claim 1, wherein, the Merge candidate is one of: a spatial domain Merge candidate derived from a spatial domain neighboring block, a temporal domain Merge candidate derived from a temporal domain neighboring block, or an alternative temporal motion vector prediction candidate.

5. The method of claim 1, wherein, the weight applied to the current video block is inherited from a block from which the asymmetric bi-prediction Merge candidate is derived.

6. The method of claim 1, wherein, the Merge motion candidate list corresponds to a component that is a luma component.

7. The method of claim 1, wherein, a value of the weight is selected from a predefined set of weights, wherein the predefined set of weights comprises a negative weight value.

8. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any of claims 1-7.

9. A non-transitory computer-readable storage medium having stored therein instructions, the instructions causing a processor to: for a conversion between a current video block of a video and a bitstream of the video, construct a Merge motion candidate list for the current video block; determine a Merge candidate from the Merge motion candidate list; and perform the conversion based on the determination, wherein, wherein at least one coding tool is disabled when the Merge candidate is an asymmetric bi-prediction Merge candidate, wherein different weights indicated by the asymmetric bi-prediction Merge candidate are applied to first and second prediction blocks of the current video block to obtain a final prediction block, and wherein the at least one coding tool comprises a decoder-side motion vector refinement, DMVR.

10. A method for coding video data, comprising: For a first conversion between a current video block of a video and a bitstream of the video, a first Merge candidate is constructed based on asymmetric weighting of a first reference Merge candidate and a second reference Merge candidate, wherein the asymmetric weighting comprises a first weight and a second weight different from the first weight, and wherein the first Merge candidate indicates a motion vector and a corresponding reference picture; after the constructing, the first Merge candidate is inserted into a Merge candidate list; and the first conversion is performed based on the Merge candidate list, wherein the method further comprises: determining a Merge candidate from the Merge candidate list; and performing a second conversion between the current video block of the video and the bitstream of the video based on the determining, wherein at least one coding tool is disabled when the determined Merge candidate is an asymmetric bi-prediction Merge candidate, wherein different weights indicated by the asymmetric bi-prediction Merge candidate are applied to a first prediction block and a second prediction block of the current video block to obtain a final prediction block, and wherein the at least one coding tool comprises decoder-side motion vector refinement DMVR.

11. The method of claim 10, wherein, the first reference Merge candidate or the second reference Merge candidate is a spatial Merge candidate, a temporal Merge candidate, a combined bi-prediction Merge candidate, a zero Merge candidate, or a Merge candidate generated using affine motion compensation prediction, alternative temporal motion vector prediction ATMVP, spatial-temporal motion vector prediction STMVP, pattern matching motion vector derivation PMMVD, bi-directional optical flow BIO, or decoder-side motion vector refinement DMVR.

12. The method of claim 10 or 11, wherein, the first reference Merge candidate and the second reference Merge candidate are selected from a top N available Merge candidates of the Merge candidate list.

13. The method of claim 10 or 11, wherein, a first picture order count difference between a current picture and the first reference Merge candidate is different from a second picture order count difference between the current picture and the second reference Merge candidate.

14. The method of claim 13, wherein, a sum of the first weight and the second weight is equal to 1, and the first weight is greater than 0.5 in a case that the first picture order count difference is smaller than the second picture order count difference; or, wherein a sum of the first weight and the second weight is equal to 1, and the second weight is greater than 0.5 in a case that the first picture order count difference is greater than the second picture order count difference.

15. The method of claim 10, wherein, the first reference Merge candidate and the second reference Merge candidate are generated using a common motion compensation algorithm, and wherein the common motion compensation algorithm is affine motion compensation prediction, alternative temporal motion vector prediction ATMVP, spatial-temporal motion vector prediction STMVP, pattern matching motion vector derivation PMMVD, bi-directional optical flow BIO, or decoder-side motion vector refinement DMVR.

16. The method of claim 10 or 11, wherein, the first reference Merge candidate or the second reference Merge candidate is generated based on symmetric weighting.

17. The method of claim 10, wherein, The asymmetric weights are selected from an asymmetric weight set, and wherein the first reference Merge candidate and the second reference Merge candidate are selected from N bi-directional Merge candidates of the Merge candidate list.

18. The method of claim 17, wherein, applying a different asymmetric weight from the asymmetric weight set to each of the N bi-directional Merge candidates, wherein the asymmetric weight is applied to at least two of the N bi-directional Merge candidates.

19. The method of claim 17, wherein, applying a different asymmetric weight from the asymmetric weight set to each of M of the N bi-directional Merge candidates, wherein M is less than N, and wherein (N-M) of the N bi-directional Merge candidates are applied a common asymmetric weight, wherein M is equal to 1.

20. The method of claim 10, further comprising: signaling signaling information containing the asymmetric weights, wherein the signaling information is a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), or a slice header.

21. The method of claim 10, wherein, The asymmetric weights are predetermined and known to both an encoder and a decoder.

22. The method of claim 10, wherein, The first weight is selected from the list {1 / 16, 2 / 16, 3 / 16, 4 / 16, 5 / 16, 6 / 16, 7 / 16, 9 / 16, 10 / 16, 11 / 16, 12 / 16, 13 / 16, 14 / 16, 15 / 16}, and wherein the sum of the first weight and the second weight is equal to one; or wherein the first weight is selected from the list {1 / 8, 2 / 8, 3 / 8, 5 / 8, 6 / 8, 7 / 8}, and wherein the sum of the first weight and the second weight is equal to one; or wherein the first weight is selected from the list {1 / 2 N , 2 / 2 N , …, (2 N -2) / 2 N , (2 N -1) / 2 N }, where N = {2, 3, 4, 5}, and wherein the sum of the first weight and the second weight equals one.

23. The method of claim 10, wherein, The first weight or the second weight is less than zero.

24. The method of claim 10, wherein, The Merge candidate list includes a first reference Merge candidate having a first index, and wherein the first Merge candidate is inserted into the Merge candidate list and has an index greater than the first index.

25. The method of claim 10, wherein, Each of the first reference Merge candidate and second reference Merge candidate is generated based on a symmetric weight, wherein the Merge candidate list includes the first reference Merge candidate having a first index and the second reference Merge candidate having a second index greater than the first index, and wherein the method further comprises: constructing a second Merge candidate based on the first reference Merge candidate; constructing a third Merge candidate based on the second reference Merge candidate; inserting the second Merge candidate into the Merge candidate list and having a third index; and inserting the third Merge candidate into the Merge candidate list and having a fourth index, the fourth index being greater than the third index.

26. The method of claim 10, wherein, inserting the first Merge candidate into the Merge candidate list having an index, and wherein the index is based on the first weight.

27. The method of claim 10, wherein, inserting the first Merge candidate in the Merge candidate list after a first bi-predictive Merge candidate, and wherein the first weight and the second weight are { 1 / 4, 3 / 4} or { 3 / 4, 1 / 4} respectively.

28. The method of claim 10, wherein, inserting the first Merge candidate in the Merge candidate list after a Merge candidate generated using affine motion compensation prediction, alternative temporal motion vector prediction (ATMVP), spatial-temporal motion vector prediction (STMVP), pattern matching motion vector derivation (PMMVD), bi-directional optical flow (BIO), or decoder-side motion vector refinement (DMVR).

29. The method of claim 10, wherein, the asymmetric weighting is the same as an asymmetric weighting used for generating a Merge candidate for a neighboring block of the current video block.

30. The method of claim 10, wherein, inserting the first Merge candidate in the Merge candidate list is based on a size or a shape of the current video block, and wherein the size of the current video block is MxN.

31. The method of claim 30, wherein, M*N is less than or equal to T = { 32, 64}; or wherein M*N is greater than or equal to T = { 32, 64}; or wherein M is equal to N; or wherein M is not equal to N.

32. The method of claim 10, wherein, inserting the first Merge candidate in the Merge candidate list with an index, and wherein the index is based on a quantization parameter (QP) of the first reference Merge candidate.

33. A method for coding video data, comprising: for a first conversion between a current video block of a video and a bitstream of the video, constructing one or more asymmetrically weighted bi-predictive Merge candidates based on one or more symmetrically weighted bi-predictive Merge candidates; after the constructing, inserting the one or more asymmetrically weighted bi-predictive Merge candidates and the one or more symmetrically weighted bi-predictive Merge candidates into a Merge candidate list so as to insert the asymmetrically weighted bi-predictive Merge candidates after the one or more symmetrically weighted bi-predictive Merge candidates in the Merge candidate list; deriving motion information from at least the asymmetrically weighted bi-predictive Merge candidates; and performing the first conversion based on the motion information, wherein the method further comprises: determining a Merge candidate from the Merge candidate list; and performing a second conversion between a current video block of the video and a bitstream of the video based on the determining, wherein at least one coding tool is disabled when the determined Merge candidate is an asymmetrically bi-predictive Merge candidate, wherein different weights indicated by the asymmetrically bi-predictive Merge candidate are applied to first and second prediction blocks of the current video block to obtain a final prediction block, and wherein the at least one coding tool comprises decoder-side motion vector refinement (DMVR).

34. The method of claim 33, wherein, the one or more asymmetrically weighted bi-predictive Merge candidates comprise a first weight and a second weight different from the first weight. the one or more asymmetrically weighted bi-predictive Merge candidates comprise a first weight and a second weight different from the first weight.

35. The method of claim 33, wherein, Each of the asymmetrically weighted bi-predictive merge candidates is constructed based on one of the symmetrically weighted bi-predictive merge candidates, and wherein the asymmetrically weighted bi-predictive merge candidates are inserted into the same sequence as the corresponding symmetrically weighted bi-predictive merge candidates.

36. The method of claim 33, wherein, Each of the asymmetrically weighted bi-predictive merge candidates includes a derived weighting value for the motion information associated with the current video block of the video data, wherein the weighting value is reused by blocks coded after the current video block of the video.

37. A method for coding video data, comprising: for a first conversion between a current video block of a video and a bitstream of the video, constructing one or more asymmetrically weighted bi-predictive merge candidates based on one or more reference asymmetrically weighted bi-predictive merge candidates such that the one or more asymmetrically weighted bi-predictive merge candidates have different weighting values from the corresponding reference asymmetrically weighted bi-predictive merge candidates; after the constructing, inserting the one or more asymmetrically weighted bi-predictive merge candidates and the one or more reference asymmetrically weighted bi-predictive merge candidates into a merge candidate list; deriving motion information from at least the asymmetrically weighted bi-predictive merge candidates; and performing the first conversion based on the motion information, wherein the method further comprises: determining a merge candidate from the merge candidate list; and performing a second conversion between a current video block of a video and a bitstream of the video based on the determining, wherein at least one coding tool is disabled when the determined merge candidate is an asymmetric bi-predictive merge candidate, wherein different weights indicated by the asymmetric bi-predictive merge candidate are applied to first and second prediction blocks of the current video block to obtain a final prediction block, and wherein the at least one coding tool includes decoder-side motion vector refinement (DMVR).

38. A device in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the processor implements the method of any of claims 10-37 when the processor executes the instructions.

Images