History update based on motion vector prediction table

Abstract

A method for processing a video, comprising: maintaining one or more tables, wherein each table includes motion candidates derived from a previously processed video block that was processed before a first video block of the video; deriving motion information for the first video block; applying refinement to the motion information when at least one decoder-side motion vector derivation (DMVD) technique is enabled; converting between a bitstream representation of the first video block and the first video block using the refined motion information; and determining whether to update the one or more tables using the refined motion information or the unrefined motion information.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of PCT / CN2019 / 074964, entitled “Motion Prediction under Shared Merge List,” filed on February 13, 2019, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] This document deals with video and image coding technologies. Background Art

[0004] Digital video still accounts for the largest bandwidth usage on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, it is expected that the bandwidth requirements for digital video usage will continue to grow. Summary of the invention

[0005] The disclosed techniques may be used by video or image decoder or encoder embodiments in which motion prediction under a shared Merge list is used.

[0006] In one example aspect, a method for processing a video is disclosed. The method includes: determining that a first video unit of a video is an intra-block copy (IBC) Merge shared node, and dividing the first video unit into a plurality of sub-blocks, wherein the sub-blocks encoded and decoded in the IBC Merge mode under the first video unit are processed in parallel; deriving motion information of the first sub-block under the first video unit; and using the derived motion information to convert between a bitstream representation of the video and the first sub-block, wherein in the derivation of the motion information of the first sub-block, the motion information of any other sub-block under the first video is omitted.

[0007] In another example aspect, a method of video processing includes: determining that a first video unit of a video is an intra block copy (IBC) advanced motion vector prediction (AMVP) shared node, and dividing the first video unit into a plurality of sub-blocks, wherein the sub-blocks encoded and decoded in the IBC AMVP mode under the first video unit are processed in parallel; deriving motion information of a first sub-block under the first video unit; and using the derived motion information to convert between a bitstream representation of the video and the first sub-block, wherein in the derivation of the motion information of the first sub-block, the motion information of any other sub-block under the first video is omitted.

[0008] In another example aspect, a method of video processing includes: determining that a first video unit of a video is an inter-frame advanced motion vector prediction (AMVP) motion sharing node, and dividing the first video unit into a plurality of sub-blocks, wherein the sub-blocks encoded and decoded in the inter-frame AMVP mode under the first video unit are processed in parallel; deriving motion information of a first sub-block of the plurality of sub-blocks encoded and decoded in the inter-frame AMVP mode under the first video unit; and based on the derived motion information, converting between a first sub-block of the plurality of sub-blocks and a bitstream representation of the video including the first video unit, wherein in the derivation of the motion information of the first sub-block, the motion information of any other sub-blocks under the first video unit is omitted.

[0009] In another example aspect, a method of video processing includes: determining that a first video unit of a video is a parameter sharing node, and the first video unit is divided into a plurality of sub-blocks; deriving parameters of a first sub-block among the plurality of sub-blocks under the first video unit during conversion between the first sub-block and a bitstream representation of the video including the first video unit, wherein the first sub-block is encoded and decoded using a codec tool; and performing the conversion based on the parameters, wherein the parameters associated with the codec tool derived during the conversion are independent of information of a second sub-block under the same parameter sharing node.

[0010] In another example aspect, a method of video processing includes: determining that a first video unit of a video is a motion sharing node, and dividing the first video unit into multiple sub-blocks as leaf nodes; deriving a motion candidate list for a first sub-block under the first video unit; and based on the motion candidate list, converting between the first sub-block of the multiple sub-blocks encoded and decoded in a first mode and a bitstream representation of the video including the first video unit, wherein motion information of a second sub-block of the multiple sub-blocks encoded and decoded in the above-mentioned first mode is omitted in the construction process of the motion candidate list; and wherein, when encoding or decoding the first sub-block, the motion candidate derived from the history-based motion vector prediction (HMVP) table is not added to the motion candidate list.

[0011] In another example aspect, a method of video processing includes: during a conversion between a first video block and a bitstream representation of a video containing the first video block, predicting motion information of the first video block by using a motion list constructed by a first candidate list construction technique for the first video block when the first video block is outside a motion sharing node; during a conversion between a second video block of a first video unit and a bitstream representation of the video, predicting motion information of the second video block by using a Merge list constructed by a second candidate list construction technique for the second video block, the second candidate list construction technique being different from the first candidate list construction technique, and the first video unit being a sharing node block; and performing the conversion based on the predicted motion information of the first video block and the second video block.

[0012] In another example aspect, a method of video processing includes: maintaining one or more tables, wherein each table includes motion candidates derived from a previously processed video block that was processed before a first video block of a video; deriving motion information for the first video block; applying refinement to the motion information when at least one decoder-side motion vector derivation (DMVD) technique is enabled; converting between a bitstream representation of the first video block and the first video block using the refined motion information; and determining whether to use the refined motion information or the unrefined motion information for updating the one or more tables.

[0013] In another example aspect, a method of video processing includes: maintaining one or more tables, wherein each table includes motion candidates derived from a previously processed video block that was processed before a first video block of a video; deriving motion information for the first video block; applying refinement to the motion information when at least one decoder-side motion vector derivation (DMVD) technique is enabled; converting between a bitstream representation of the first video block and the first video block using the refined motion information; and updating the one or more tables with unrefined motion information for the first video block.

[0014] In another example aspect, a method of video processing includes: maintaining one or more tables, wherein each table includes motion candidates derived from a previously processed video block that was processed before a first video block in a video; deriving motion information for the first video block; determining, based on whether the motion information is derived from the one or more tables, using at least one decoder-side motion vector derivation (DMVD) technique to enable or disable refinement of the motion information for the first video block; and converting between a bitstream representation of the video and the first video block based on the determination.

[0015] In another example aspect, the above method may be implemented by a video encoder device including a processor.

[0016] In yet another example aspect, the methods may be embodied in the form of processor-executable instructions and stored on a computer-readable program medium.

[0017] These and other aspects are described further in this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 An example of the derivation process for Merge candidate list construction is shown.

[0019] Figure 2 An example of the location of spatial merge candidates is shown.

[0020] Figure 3 An example of candidate pairs considered for redundancy check of spatial merge candidates is shown.

[0021] Figure 4 Examples of the location of the second PU for N×2N and 2N×N partitions are shown.

[0022] Figure 5 An example of motion vector scaling for temporal merge candidates is shown.

[0023] Figure 6 An example of candidate positions of the time-domain Merge candidates C0 and C1 is shown.

[0024] Figure 7 An example of a combined bi-predictive Merge candidate is shown.

[0025] Figure 8 An example of a derivation process of motion vector prediction candidates is shown.

[0026] Fig. 9 An example of motion vector scaling for spatial motion vector candidates is shown.

[0027] Fig. 10A An example of a 135 degree segmentation type (divided from the upper left corner to the lower right corner) is shown.

[0028] Fig. 10B An example of dividing a CU into two triangular prediction units (two division modes) is shown.

[0029] Fig.11 An example of the positions of neighboring blocks is shown.

[0030] Fig.12 An example of adjacent blocks (A and L) used for context selection in TPM flag encoding and decoding is shown.

[0031] Fig.13A and 13B An example of a simplified affine motion model is shown.

[0032] Fig.14 An example of the affine MVF for each sub-block is shown.

[0033] Fig.15 Examples of a 4-parameter affine model (a) and a 6-parameter affine model (b) are shown.

[0034] Fig.16 An example of MVP of AF_INTER for inherited affine candidates is shown.

[0035] Fig.17 An example of MVP of AF_INTER for a constructed affine candidate is shown.

[0036] Fig.18 An example of a candidate for AF_MERGE is shown.

[0037] Fig.19 An example of candidate positions for the affine merge mode is shown.

[0038] Fig. 20 An example of intra-picture block duplication is shown.

[0039] Fig.21 An example of candidate positions for the affine merge mode is shown.

[0040] Fig. 22 An example of a modified Merge list building process is shown.

[0041] Fig.23 An example of merging shared nodes is shown.

[0042] Fig.24 is a block diagram of an example of a video processing device.

[0043] Fig.25 A block diagram of an example implementation of a video encoder is shown.

[0044] Fig.26 is a flow chart of an example of a video processing method.

[0045] Fig. 27 is a flow chart of an example of a video processing method.

[0046] Fig.28 is a flow chart of an example of a video processing method.

[0047] Fig.29 is a flow chart of an example of a video processing method.

[0048] Fig.30 is a flow chart of an example of a video processing method.

[0049] Fig.31 is a flow chart of an example of a video processing method.

[0050] Fig.32 is a flow chart of an example of a video processing method.

[0051] Fig.33 is a flow chart of an example of a video processing method.

[0052] Fig.34 is a flow chart of an example of a video processing method.

[0053] Fig.35 is a flow chart of an example of a video processing method.

[0054] Fig.36 is a flow chart of an example of a video processing method.

[0055] Fig.37 is a flow chart of an example of a video processing method.

[0056] Fig.38 is a flow chart of an example of a video processing method.

[0057] Fig.39 is a flow chart of an example of a video processing method. DETAILED DESCRIPTION

[0058] This document provides various techniques that can be used by a decoder of an image or video bitstream to improve the quality of a decompressed or decoded digital video or image. For simplicity, the term "video" is used herein to include both a sequence of pictures (traditionally referred to as video) and a single image. In addition, a video encoder may also implement these techniques during the encoding process in order to reconstruct a decoded frame for further encoding.

[0059] The section headings used in this document are for ease of understanding and do not limit the embodiments and techniques to the corresponding sections. Thus, the embodiments of one section can be combined with the embodiments of other sections.

[0060] 1. Summary

[0061] The present invention relates to video coding technology. Specifically, it relates to motion vector coding in video coding. It can be applied to existing video coding standards such as HEVC or standards to be finalized (universal video coding). It can also be applied to future video coding standards or video codecs.

[0062] 2. Background

[0063] Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T produced the H.261 and H.263 standards, ISO / IEC produced the MPEG-1 and MPEG-4 Visual standards, and the two organizations jointly produced the H.262 / MPEG-2 Video standard and the H.264 / MPEG-4 Advanced Video Coding (AVC) standard and the H.265 / HEVC standard. Starting with H.262, video codec standards are based on a hybrid video codec structure that utilizes temporal prediction plus transform coding. In order to explore future video codec technologies beyond HEVC, the Joint Video Exploration Team (JVET) was jointly established by VCEG and MPEG in 2015. Since then, JVET has adopted many new methods and incorporated them into reference software called the Joint Exploration Model (JEM). In April 2018, the Joint Video Experts Team (JVET) between VCEG (Q6 / 16) and ISO / IEC JTC1SC29 / WG11 (MPEG) was established to work on the VVC standard, which reduces the bit rate by 50% compared to HEVC.

[0064] Fig.25 is a block diagram of an example implementation of a video encoder. Fig.25 An encoder implementation with a built-in feedback path is shown, where the video encoder also performs a video decoding function (reconstructing a compressed representation of the video data for use in the encoding of the next video data).

[0065] 2.1 Inter-frame prediction in HEVC / H.265

[0066] Each inter-predicted PU has motion parameters for one or two reference picture lists. Motion parameters include motion vectors and reference picture indices. The use of one of the two reference picture lists may also be signaled using inter_pred_idc. Motion vectors may be explicitly encoded as deltas relative to the predictor.

[0067] When a CU is encoded using skip mode, a PU is associated with the CU and there are no significant residual coefficients, no coded motion vector increments or reference picture indices. A Merge mode is specified, whereby the motion parameters for the current PU are obtained from neighboring PUs, including spatial and temporal candidates. Merge mode can be applied to any inter-predicted PU, not just skip mode. An alternative to Merge mode is explicit transmission of motion parameters, where the motion vector (more precisely, the motion vector difference (MVD) compared to the motion vector predictor), the corresponding reference picture index for each reference picture list, and the reference picture list usage are explicitly signaled for each PU. Such a mode is named advanced motion vector prediction in the present disclosure.

[0068] When signaling indicates that one of the two reference picture lists is to be used, the PU is generated from one sample block. This is called "unidirectional prediction". Unidirectional prediction is available for P slices and B slices.

[0069] When signaling indicates that two reference picture lists are to be used, the PU is generated from two sample blocks. This is called "bi-prediction". Bi-prediction is only available for B slices.

[0070] The following provides detailed information on the inter prediction modes specified in HEVC. The description will start with the Merge mode.

[0071] 2.1.1 Reference Image List

[0072] In HEVC, the term inter prediction is used to refer to predictions derived from data elements (e.g., sample values ​​or motion vectors) of reference pictures other than the currently decoded picture. As in H.264 / AVC, a picture can be predicted from multiple reference pictures. Reference pictures used for inter prediction are organized in one or more reference picture lists. A reference index identifies which reference pictures in the list should be used to create a prediction signal.

[0073] A single reference picture list, List 0, is used for P slices, and two reference picture lists, List 0 and List 1, are used for B slices. It should be noted that the reference pictures contained in List 0 / 1 can be pictures from the past and the future in terms of capture / display order.

[0074] 2.1.2 Merge Mode

[0075] 2.1.2.1 Merge Mode Candidate Derivation

[0076] When using Merge mode to predict a PU, an index pointing to an entry in the Merge candidate list is parsed from the bitstream and used to retrieve motion information. The construction of this list is specified in the HEVC standard and can be summarized in the following order:

[0077] Step 1: Initial candidate derivation

[0078] Step 1.1: Spatial Candidate Derivation

[0079] Step 1.2: Redundancy check of spatial candidates

[0080] Step 1.3: Time Domain Candidate Derivation

[0081] Step 2: Additional candidate insertions

[0082] Step 2.1: Create bidirectional prediction candidates

[0083] Step 2.2: Insert zero motion candidates

[0084] exist Figure 1 These steps are also schematically depicted in . For spatial Merge candidate derivation, up to four Merge candidates are selected from candidates located at five different positions. For temporal Merge candidate derivation, at most one Merge candidate is selected from two candidates. Since the number of candidates for each PU is assumed to be constant at the decoder, additional candidates are generated when the number of candidates obtained from step 1 does not reach the maximum number of Merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, binary unary truncation (TU) is used to encode the index of the best Merge candidate. If the size of the CU is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

[0085] Hereinafter, operations associated with the aforementioned steps are described in detail.

[0086] 2.1.2.2 Derivation of Spatial Candidates

[0087] In the derivation of spatial merge candidates, Figure 2Up to four Merge candidates are selected from the candidates at the positions depicted in . The order of derivation is A1, B1, B0, A0 and B2. Position B2 is considered only when any PU at position A1, B1, B0, A0 is not available (for example, because the PU belongs to another slice or tile) or is intra-coded. After adding the candidate at position A1, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only pairs with Figure 3 The arrows in are linked pairs, and only if the corresponding candidate for redundancy check has different motion information, the candidate is added to the list.

[0088] 2.1.2.2.1Merge Estimation Region (MER)

[0089] Another source of repeated motion information is a "second PU" associated with a partition other than 2N×2N. As an example, Figure 4 The second PU is depicted for the N×2N and 2N×N cases, respectively. When the current PU is partitioned into N×2N, the candidate at position A1 is not considered for list construction. In fact, adding this candidate would result in two prediction units with the same motion information, which is redundant for the case with only one PU in the codec unit. Similarly, position B1 is not considered when the current PU is partitioned into 2N×N.

[0090] 2.1.2.3 Derivation of time domain candidates

[0091] In this step, only one candidate is added to the list. In particular, in the derivation of the temporal Merge candidate, the scaled motion vector is derived based on the co-located PU, which belongs to the picture with the smallest POC difference relative to the current picture within a given reference picture list. The reference picture list used for the derivation of the co-located PU is explicitly signaled in the slice header. Figure 5As shown by the dashed line in the figure, the derivation of the scaled motion vector for the temporal Merge candidate is obtained, which is scaled from the motion vector of the collocated PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal Merge candidate is set equal to zero. The actual implementation of the scaling process is described in the HEVC specification. For B slices, two motion vectors are obtained and combined to produce a bidirectional prediction Merge candidate, one of which is used for reference picture list 0 (list 0) and the other for reference picture list 1 (list 1).

[0092] like Figure 6 As shown, in the collocated PU (Y) belonging to the reference frame, the position for the time domain candidate is selected between candidates C0 and C1. If the PU at position C0 is not available, intra-coded, or outside the current codec tree unit (CTU, i.e., LCU, the largest codec unit), position C1 is used. Otherwise, position C0 is used in the derivation of the time domain Merge candidate.

[0093] 2.1.2.4 Additional Candidate Insertion

[0094] In addition to spatial and temporal Merge candidates, there are two additional types of Merge candidates: combined bidirectional prediction Merge candidates and zero Merge candidates. Combined bidirectional prediction Merge candidates are generated by utilizing spatial and temporal Merge candidates. Combined bidirectional prediction Merge candidates are only used for B slices. A combined bidirectional prediction candidate is generated by combining the first reference picture list motion parameters of an initial candidate with the second reference picture list motion parameters of another candidate. If the two tuples provide different motion hypotheses, they will form a new bidirectional prediction candidate. As an example, Figure 7 The following situation is shown, where two candidates with mvL0 and refIdxL0 or mvL1 and refIdxL1 in the original list (on the left) are used to create a combined bi-predictive Merge candidate, which is added to the final list (on the right). There are many rules about the combinations that are considered to generate these additional Merge candidates.

[0095] Zero motion candidates are inserted to fill the remaining entries in the Merge candidate list and thus reach the MaxNumMergeCand capacity. These candidates have zero spatial displacement and a reference picture index that starts at zero and increases each time a new zero motion candidate is added to the list.

[0096] More specifically, the following steps are performed in sequence until the Merge list is full:

[0097] Setting the variable numRef to the number of reference pictures associated with list 0 for a P slice, or to the minimum number of reference pictures in both lists for a B slice;

[0098] Add non-repeating zero-motion candidates:

[0099] For variable i being 0...numRef-1, a default motion candidate with MV set to (0, 0) and reference picture index set to i is added for list 0 (if P slice) or both lists (if B slice).

[0100] Add repeated zero motion candidates with MV set to (0, 0), reference picture index of list 0 set to 0 (if P slice), and reference picture index of both lists set to 0 (if B slice).

[0101] Finally, no redundant checks are performed on these candidates.

[0102] 2.1.3 AMVP

[0103] AMVP exploits the spatial-temporal correlation of motion vectors with neighboring PUs, which is used for explicit transmission of motion parameters. For each reference picture list, a motion vector candidate list is constructed by first checking the availability of the left and top neighboring PU positions in the time domain, removing redundant candidates, and adding zero vectors to make the candidate list a constant length. The encoder can then select the best predictor from the candidate list and transmit the corresponding index indicating the selected candidate. Similar to Merge index signaling, unary truncation is used to encode the index of the best motion vector candidate. The maximum value to be encoded in this case is 2 (see Figure 8 ). In the following sections, details about the derivation process of motion vector prediction candidates are provided.

[0104] 2.1.3.1 Derivation of AMVP Candidates

[0105] Figure 8 The derivation process for motion vector prediction candidates is summarized.

[0106] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. Figure 2 As shown, for spatial motion vector candidate derivation, two motion vector candidates are finally derived based on the motion vectors of each PU located at five different positions.

[0107] For temporal motion vector candidate derivation, select one motion vector candidate from two candidates derived based on two different collocated positions. After making the first list of spatial-temporal candidates, remove duplicate motion vector candidates in the list. If the number of potential candidates is greater than 2, remove motion vector candidates whose reference picture index within the associated reference picture list is greater than 1 from the list. If the number of spatial-temporal motion vector candidates is less than 2, add an additional zero motion vector candidate to the list.

[0108] 2.1.3.2 Spatial Motion Vector Candidates

[0109] In the derivation of spatial motion vector candidates, at most two candidates are considered among five potential candidates from the locations such as Figure 2 PUs at the positions shown are the same as those of the motion Merge. The derivation order for the left side of the current PU is defined as A0, A1, and scaled A0, scaled A1. The derivation order for the upper side of the current PU is defined as B0, B1, B2, scaled B0, scaled B1, scaled B2. Therefore, for each side, there are four cases that can be used as motion vector candidates, two of which do not require the use of spatial domain scaling, and two use spatial domain scaling. The four different cases are summarized as follows.

[0110] No airspace scaling

[0111] (1) Same reference picture list and same reference picture index (same POC)

[0112] (2) Different reference picture lists, but the same reference picture (same POC)

[0113] Airspace Scaling

[0114] (3) Same reference picture list, but different reference pictures (different POC)

[0115] (4) Different reference picture lists and different reference pictures (different POCs)

[0116] First check for no spatial scaling, then check for spatial scaling. Regardless of the reference picture list, consider spatial scaling when the POC is different between the reference picture of the neighboring PU and the reference picture of the current PU. If all PUs of the left candidate are unavailable or intra-coded, scaling of the upper motion vector is allowed to help parallel derivation of the left and upper MV candidates. Otherwise, spatial scaling is not allowed for the upper motion vector.

[0117] In the spatial domain scaling process, the motion vectors of neighboring PUs are scaled in a similar way to the temporal domain scaling, such as Fig. 9As shown, the main difference is that the reference picture list and index of the current PU are given as input; the actual scaling process is the same as the temporal scaling process.

[0118] 2.1.3.3 Temporal Motion Vector Candidates

[0119] Except for the reference picture index derivation, all the processes for the derivation of the temporal Merge candidate are the same as those for the derivation of the spatial motion vector candidate (see Figure 6 ). The reference picture index is signaled to the decoder.

[0120] 2.2 Inter-frame prediction method in VVC

[0121] There are several new codec tools for inter prediction improvements, such as adaptive motion vector difference resolution (AMVR) for signaling MVD for affine blocks and translational motion blocks, affine prediction mode, triangular prediction mode (TPM), alternative TMVP (ATMVP), generalized bidirectional prediction (GBI), bidirectional optical flow (BIO), intra block copy (IBC), etc.

[0122] 2.2.1 Adaptive motion vector difference resolution

[0123] In HEVC, when use_integer_mv_flag is equal to 0 in the slice header, the motion vector difference (MVD) (between the PU's motion vector and the predicted motion vector) is signaled in units of quarter luma samples. In VVC, Local Adaptive Motion Vector Resolution (LAMVR) is introduced.

[0124] 2.2.1.1 AMVR for Translational Motion Blocks

[0125] In VVC, MVD can be encoded and decoded in units of quarter luma samples, integer luma samples, or four luma samples (i.e., 1 / 4 pixel, 1 pixel, 4 pixels). The MVD resolution is controlled at the codec unit (CU) level, and the MVD resolution flag is conditionally signaled to each CU with at least one non-zero MVD component.

[0126] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter luma sample MV precision is not used, another flag is signaled to indicate whether integer luma sample MV precision or four luma sample MV precision is used.

[0127] When the first MVD resolution flag of a CU is zero, or is not encoded for a CU (meaning that all MVDs in the CU are zero), a quarter luma sample MV resolution is used for the CU. When a CU uses integer luma sample MV precision or four luma sample MV precision, the MVP in the AMVP candidate list of the CU is rounded to the corresponding precision.

[0128] 2.2.1.2 AMVR of Affine Motion Block

[0129] Similar to the design for translational motion block (also known as normal AMVR mode), the AMVR mode is extended to affine codec block (also known as affine AMVR mode).

[0130] For the affine AMVR mode, three MV / MVD resolutions are supported, namely {1 / 16, 1 / 4, 1} pixels.

[0131] 2.2.2 Triangle prediction model

[0132] The concept of triangular prediction mode (TPM) is to introduce a new triangular partition for motion compensated prediction. As shown in Figure 10, it divides the CU into two triangular prediction units in the diagonal direction or the reverse diagonal direction. Each triangular prediction unit in the CU uses its own unidirectional prediction motion vector and reference frame index for inter-frame prediction, which is derived from a single unidirectional prediction candidate list. After predicting the triangular prediction unit, the diagonal edges are adaptively weighted. Then, the transformation and quantization process is applied to the entire CU. Note that this mode is only applicable to Merge mode (Note: Skip mode is regarded as a special Merge mode).

[0133] 2.2.2.1 TPM’s One-Way Prediction Candidate List

[0134] The unidirectional prediction candidate list, called the TPM motion candidate list, includes five unidirectional prediction motion vector candidates. It is derived from seven neighboring blocks, including five spatial neighboring blocks (1 to 5) and two temporally collocated blocks (6 to 7), such as Fig.11 As shown. The motion vectors of seven neighboring blocks are collected and put into the unidirectional prediction candidate list according to the following order: the order of unidirectional prediction motion vector, L0 motion vector of bidirectional prediction motion vector, L1 motion vector of bidirectional prediction motion vector, and the average motion vector of L0 and L1 motion vector of bidirectional prediction motion vector. If the number of candidates is less than five, the zero motion vector is added to the list. The motion candidates added for TPM in this list are called TPM candidates, and the motion information derived from the spatial / temporal blocks is called regular motion candidates.

[0135] More specifically, the following steps are involved:

[0136] When adding normal motion candidates from spatial neighboring blocks, normal motion candidates (corresponding to Fig.11 1-7 in the .

[0137] Set variable numCurrMergeCand=0.

[0138] For each normal motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2, if not pruned and numCurrMergeCand is less than 5, if the normal motion candidate is uni-directionally predicted (from list 0 or list 1), it is directly added to the Merge list as a TPM candidate, and numCurrMergeCand is increased by 1. Such a TPM candidate is called an "original uni-directionally predicted candidate".

[0139] Apply full trim.

[0140] For each normal motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2 and for which numCurrMergeCand is less than 5, if the normal motion candidate is bi-directionally predicted, the motion information from list 0 is added to the TPM Merge list as a new TPM candidate (i.e., modified to be uni-directionally predicted from list 0), and numCurrMergeCand is increased by 1. Such a TPM candidate is referred to as a "truncated list 0-predicted candidate".

[0141] Apply full trim.

[0142] For each normal motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2 and for which numCurrMergeCand is less than 5, if the normal motion candidate is bi-directionally predicted, the motion information from list 1 is added to the TPM Merge list (i.e., modified to be uni-directionally predicted from list 1), and numCurrMergeCand is increased by 1. Such a TPM candidate is referred to as a "truncated list 1-predicted candidate."

[0143] Apply full trim.

[0144] For each normal motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2 and for which numCurrMergeCand is less than 5, if the normal motion candidate is bi-directionally predicted,

[0145] If the slice QP of the list 0 reference picture is less than the slice QP of the list 1 reference picture, the motion information of list 1 is first scaled to the list 0 reference picture, and the average of the two MVs (one from the original list 0 and the other is the scaled MV from list 1) is added to the TPM Merge list, such a candidate is called an averaged uni-directional prediction from list 0 motion candidate, and numCurrMergeCand is increased by 1.

[0146] Otherwise, the motion information of list 0 is first scaled to the reference picture of list 1, and then the average of the two MVs (one from the original list 1 and the other is the scaled MV from list 0) is added to the TPM Merge list, such a TPM candidate is called the average unidirectional prediction from the list 1 motion candidate, and numCurrMergeCand is increased by 1.

[0147] Perform a complete pruning.

[0148] If numCurrMergeCand is less than 5, a zero motion vector candidate is added.

[0149] When a candidate is inserted into the list, if it must be compared to all previously added candidates to see if it is identical to one of them, this process is called full pruning.

[0150] 2.2.2.2 Adaptive weighting process

[0151] After predicting each triangular prediction unit, an adaptive weighting process is applied to the diagonal edges between two triangular prediction units to derive the final prediction for the entire CU. Two weighting factor groups are defined as follows:

[0152] The first weighting factor group: {7 / 8, 6 / 8, 4 / 8, 2 / 8, 1 / 8} and {7 / 8, 4 / 8, 1 / 8} are used for luma and chroma samples respectively;

[0153] The second set of weighting factors: {7 / 8, 6 / 8, 5 / 8, 4 / 8, 3 / 8, 2 / 8, 1 / 8} and {6 / 8, 4 / 8, 2 / 8} are used for luma and chroma samples respectively.

[0154] The weighting factor set is selected based on the comparison of the motion vectors of the two triangular prediction units. When the reference pictures of the two triangular prediction units are different from each other or the difference of their motion vectors is greater than 16 pixels, the second weighting factor set is used. Otherwise, the first weighting factor set is used.

[0155] 2.2.2.3 Signaling of Triangular Prediction Mode (TPM)

[0156] A one-bit flag indicating whether to use TPM may be signaled first, and then two partitioning mode indications (as shown in FIG. 10 ) and a Merge index selected for each of the two partitions may be further signaled.

[0157] 2.2.2.3.1 TPM Flag Signaling Notification

[0158] Denote the width and height of a luma block by W and H respectively. If W*H<64, the triangular prediction mode is disabled.

[0159] When encoding or decoding a block in affine mode, triangular prediction mode is also disabled.

[0160] When encoding or decoding a block in Merge mode, a one-bit flag may be signaled to indicate whether triangular prediction mode is enabled or disabled for the block.

[0161] The flags are encoded and decoded with 3 contexts based on the following equations:

[0162] Ctx index=((left block Lavailable&&Lis coded with TPM?)1:0)

[0163] +((Above block Aavailable&&A is coded with TPM?)1:0).

[0164] 2.2.3 Affine Motion Compensated Prediction

[0165] In HEVC, only the translation motion model is applied for motion compensated prediction (MCP). In the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, simplified affine transformation motion compensated prediction is applied to 4-parameter affine model and 6-parameter affine model. As shown in Figure 13, the affine motion field of the block is described by two control point motion vectors (CPMV) of the 4-parameter affine model and 3 CPMVs of the 6-parameter affine model.

[0166] The motion vector field (MVF) of a block is described by the following equations in terms of a 4-parameter affine model in equation (1) (where the 4-parameters are defined as variables a, b, e, and f) and a 6-parameter affine model in equation (2) (where the 4-parameters are defined as variables a, b, c, d, e, and f), respectively:

[0167]

[0168]

[0169] Where (mvh 0,mv h 0) is the motion vector of the left top corner control point, and (mv h 1, mv h 1) is the motion vector of the right top corner control point, and (mv h 2.mv h 2) is the motion vector of the lower left control point. All three motion vectors are called control point motion vectors (CPMVs). (x, y) represents the coordinates of the representative point relative to the upper left sample in the current block, and (mv h (x,y),mv v (x, y)) is the motion vector derived for the sample at (x, y). The CP motion vector can be signaled (as in affine AMVP mode) or derived on the fly (as in affine Merge mode). w and h are the width and height of the current block. In practice, the division is performed by rounding and right shifting. In VTM, the representative point is defined as the center position of the sub-block, for example, when the coordinates of the upper left corner of the sub-block relative to the upper left sample in the current block are (xs, ys), the coordinates of the representative point are defined as (xs+2, ys+2). For each sub-block (i.e., 4×4 in VTM), a representative point is used to derive the motion vector for the entire sub-block.

[0170] To further simplify motion compensation prediction, sub-block based affine transformation prediction is applied. To derive the motion vector of each M×N (in the current VVC, both M and N are set to 4) sub-block, as Fig.14 As shown, the motion vector of the center sample of each sub-block is calculated according to equations (1) and (2) and rounded to a fractional precision of 1 / 16. Then, a 1 / 16 pixel motion compensation interpolation filter can be applied to generate a prediction for each sub-block using the derived motion vector. The 1 / 16 pixel interpolation filter is introduced through an affine mode.

[0171] After MCP, the high-precision motion vector of each sub-block is rounded and saved with the same precision as the normal motion vector.

[0172] 2.2.3.1 Signaling of Affine Prediction

[0173] Similar to the translational motion model, due to affine prediction, there are also two modes for signaling auxiliary information. They are affine_INTER and affine_MERGE modes.

[0174] 2.2.3.2 AF_INTER mode

[0175] AF_INTER mode can be applied for CUs with width and height both greater than 8. A CU-level affine flag is signaled in the bitstream to indicate whether AF_INTER mode is used.

[0176] In this mode, for each reference picture list (list 0 or list 1), an affine AMVP candidate list is constructed from the three types of affine motion predictors in the following order, where each candidate includes the estimated CPMV of the current block. Fig.17 The difference between the best CPMV found by (in mv0 mv1 mv2) and the estimated CPMV is signaled. In addition, the index of the affine AMVP candidate from which the estimated CPMV is derived is further signaled.

[0177] Inherited affine motion predictor

[0178] The checking order is similar to the checking order of spatial MVP in HEVC AMVP list construction. First, the left inherited affine motion predictor is derived from the first block in {A1, A0} that is affine-coded and has the same reference picture as the current block. Second, the above inherited affine motion predictor is derived from the first block in {B1, B0, B2} that is affine-coded and has the same reference picture as the current block. Fig.16 Five blocks A1, A0, B1, B0, B2 are shown in FIG.

[0179] Once a neighboring block is found to be coded in affine mode, the CPMV of the codec covering the neighboring block is used to derive the predictor of the CPMV of the current block. For example, if A1 is coded in non-affine mode, and A0 is coded in 4-parameter affine mode, the left-inherited affine MV predictor will be derived from A0. In this case, the CPMV of the CU covering A0 is, Fig.18 (b) The upper left CPMV and the upper right CPMV The marked ones are used to derive the estimated CPMV of the current block, using Mark the upper left (coordinate (x0, y0)), upper right (coordinate (x1, y1)), and lower right positions (coordinate (x2, y2)) of the current block.

[0180] Affine motion predictor constructed

[0181] The constructed affine motion predictor includes control point motion vectors (CPMVs) derived from neighboring inter-coded blocks with the same reference picture, such as Fig.17 If the current affine motion model is 4-parameter affine, the number of CPMVs is 2, otherwise, if the current affine motion model is 6-parameter affine, the number of CPMVs is 3. Upper left CPMV Derived from the MV at the first block in the {A, B, C} group that is decoded in inter-frame and has the same reference picture as the current block. Top right CPMV Derived from the MV at the first block in the {D, E} group that is decoded in inter-frame and has the same reference picture as the current block. Bottom-left CPMV Derived by the MV at the first block in the {F, G} group that is decoded with inter-frame coding and has the same reference picture as the current block.

[0182] If the current affine motion model is 4-parameter affine, then only and When both are present, the constructed affine motion predictor is inserted into the candidate list, that is, and Used as the estimated CPMV of the upper left (coordinate (x0, y0)) and upper right (coordinate (x1, y1)) positions of the current block.

[0183] If the current affine motion model is a 6-parameter affine, then only all and The constructed affine motion predictor is inserted into the candidate list only when and Used as the estimated CPMV of the top left (coordinate (x0, y0)), top right (coordinate (x1, y1)), and bottom right (coordinate (x2, y2)) positions of the current block.

[0184] When inserting the constructed affine motion predictor into the candidate list, no pruning process is applied.

[0185] Normal AMVP motion predictor

[0186] The following applies until the number of affine motion predictors reaches a maximum.

[0187] By setting all CPMVs equal to (if available) to derive the affine motion predictor.

[0188] By setting all CPMVs equal to (if available) to derive the affine motion predictor.

[0189] By setting all CPMVs equal to (if available) to derive the affine motion predictor.

[0190] The affine motion predictor is derived by setting all CPMVs equal to the HEVC TMVP (if available).

[0191] The affine motion predictor is derived by setting all CPMVs to zero MV.

[0192] Note that we have already derived

[0193] In AF_INTER mode, when using the 4 / 6 parameter affine mode, 2 / 3 control points are required, and therefore 2 / 3 MVDs need to be encoded and decoded for these control points, such as Fig.15 It is proposed that MV can be derived as follows, that is, predicting mvd1 and mvd2 from mvd0.

[0194]

[0195]

[0196]

[0197] in, mvd i and mv1 are the predicted motion vector, motion vector difference, and motion vector of the top left pixel (i=0), the top right pixel (i=1), or the bottom left pixel (i=2), respectively. Fig.15 (b) Note that the addition of two motion vectors (e.g., mvA(xA, yA) and mvB(xB, yB)) is equal to the sum of the two components individually. That is, newMV = mvA + mvB and the two components of newMV are set to (xA + xB) and (yA + yB), respectively.

[0198] 2.2.3.3AF_MERGE Mode

[0199] When a CU is applied in AF_MERGE mode, it gets the first block coded in affine mode from the valid neighboring reconstructed blocks. And, Fig.18 As shown in (a), the order of selecting candidate blocks is from left, top, top right, bottom left to top left (represented by A, B, C, D, E, respectively). Fig.18 As shown in A0 in (b), the adjacent lower left block is encoded and decoded in affine mode, and the control point (CP) motion vector mv0 of the lower left corner of the upper left corner and the upper right corner of the adjacent CU / PU containing block A is retrieved N , mv1 N and mv2 N . And based on mv0 N , mv1 N and mv2 N Calculate the motion vector mv0 of the upper left corner / upper right / lower left corner of the current CU / PU C , mv1 C and mv2 C(Only for 6-parameter affine model). Note that in VTM-2.0, the sub-block at the top left corner (e.g., 4×4 block in VTM) stores mv0, and if the current block is affine coded, the sub-block at the top right corner stores mv1. If the current block is coded using the 6-parameter affine model, the sub-block at the bottom left corner will store mv2; otherwise (using the 4-parameter affine model), LB stores mv2'. The other sub-blocks store the MV for MC.

[0200] The current CU mv0 is derived according to the simplified affine motion model in equations (1) and (2). C , mv1 C and mv2 C After the CPMV of the current CU is obtained, the MVF of the current CU is generated. In order to identify whether the current CU is encoded and decoded in AF_MERGE mode, when there is at least one neighboring block encoded and decoded in affine mode, the affine flag can be signaled in the bitstream.

[0201] As an example, the affine merge candidate list is constructed by the following steps:

[0202] Insert inheritance affine candidates

[0203] Inherited affine candidates are candidates derived from the affine motion model of the block whose valid region is affine encoded. Up to two inherited affine candidates are derived from the affine motion models of neighboring blocks and inserted into the candidate list. The scan order is {A0, A1} for the left predictor and {B0, B1, B2} for the top predictor.

[0204] Insert the constructed affine candidate

[0205] If the number of candidates in the affine Merge candidate list is less than MaxNumAffineCand (eg, 5), the constructed affine candidate is inserted into the candidate list. The constructed affine candidate refers to a candidate constructed by combining adjacent motion information of each control point.

[0206] First, from Fig.19 The motion information of the control points is derived in the specified spatial and temporal domains as shown. CPk (k = 1, 2, 3, 4) represents the kth control point. A0, A1, A2, B0, B1, B2 and B3 are the spatial positions (k = 1, 2, 3) used to predict CPk; T is the temporal position used to predict CP4.

[0207] The coordinates of CP1, CP2, CP3 and CP4 are (0, 0), (W, 0), (H, 0) and (W, H), respectively, where W and H are the width and height of the current block.

[0208] The motion information for each control point is obtained according to the following priority order:

[0209] - For CP1, the priority of checking is B2->B3->A2. If B2 is available, B2 is used.

[0210] Otherwise, if B2 is not available, use B3. If both B2 and B3 are not available, use A2. If none of the three candidates are available, the motion information of CP1 cannot be obtained.

[0211] - For CP2, check priority is B1->B0.

[0212] - For CP3, check priority is A1->A0.

[0213] -For CP4, use T.

[0214] Second, affine merge candidates are constructed using combinations of control points.

[0215] The motion information of three control points is required to construct a 6-parameter affine candidate. Three control points can be selected from one of the following four combinations: ({CP1, CP2, CP4}, {CP1, CP2, CP3}, {CP2, CP3, CP4}, {CP1, CP3, CP4}). The combination of {CP1, CP2, CP3}, {CP2, CP3, CP4}, {CP1, CP3, CP4} will be converted into a 6-parameter motion model represented by the top left, top right and bottom left control points.

[0216] The motion information of two control points is required to construct a 4-parameter affine candidate. The two control points can be selected from one of the following two combinations: ({CP1, CP2}, {CP1, CP3}). These two combinations will be converted into a 4-parameter motion model represented by the top left and top right control points.

[0217] Insert the combinations of constructed affine candidates into the candidate list in the following order:

[0218] {CP1, CP2, CP3}, {CP1, CP2, CP4}, {CP1, CP3, CP4}, {CP2, CP3, CP4}, {CP1, CP2}, {CP1, CP3}.

[0219] For each combination, check the reference index of list X for each CP, if they are all the same, then the combination has a valid CPMV for list X. If the combination does not have a valid CPMV for both list 0 and list 1, then this combination is marked as invalid. Otherwise, it is valid and the CPMV is put into the sub-block Merge list.

[0220] Fill with zero motion vectors

[0221] If the number of candidates in the affine merge candidate list is less than 5, zero motion vectors with zero reference index are inserted into the candidate list until the list is full.

[0222] More specifically, for the sub-block Merge candidate list, the MV of the 4-parameter Merge candidate is set to (0, 0) and the prediction direction is unidirectional prediction (for P slices) and bidirectional prediction (for B slices) from list 0.

[0223] 2.2.4 Current Image Reference

[0224] Intra-block copying (i.e., IBC or intra-picture block compensation), also known as current picture reference (CPR), is employed in the HEVC Screen Content Codec extension (SCC). This tool is very effective for the encoding and decoding of screen content videos, since repeated patterns in the same picture often occur in text- and graphics-rich content. Using previously reconstructed blocks with the same or similar patterns as predictors can effectively reduce the prediction error and thus improve the codec efficiency. An example of intra-block compensation is shown in Fig. 20 Shown in.

[0225] Similar to the design of CRP in HEVC SCC, in VVC, the use of IBC mode is signaled at the sequence and picture levels. When IBC mode is enabled at the sequence parameter set (SPS), it can be enabled at the picture level. When IBC mode is enabled at the picture level, the currently reconstructed picture will be considered as a reference picture. Therefore, no syntax changes are required at the block level on top of the existing VVC inter-frame mode to signal the use of IBC mode.

[0226] Key Features:

[0227] Treat it as a special inter mode. Therefore, Merge and Skip modes can also be used in IBC mode. The Merge candidate list construction is unified and contains Merge candidates from neighboring positions that are coded or decoded in IBC mode or HEVC inter mode. Depending on the selected Merge index, the current block in Merge or Skip mode can be merged into the field coded in IBC mode or into the field coded in normal inter mode with different pictures as reference pictures.

[0228] The IBC mode motion vectors (also called block vectors) are encoded and decoded with integer pixel precision, but are stored in memory with 1 / 16 pixel precision after decoding because quarter pixel precision is required during the interpolation and deblocking stages. When used for IBC mode motion vector prediction, the stored vector predictor is right-shifted by 4.

[0229] Search scope: limited to the current CTU.

[0230] When Affine Mode / Triangle Mode / GBI / Weighted Prediction is enabled, CPR is not allowed.

[0231] 2.2.5 Merge List Design in VVC

[0232] VVC supports three different Merge list construction processes:

[0233] Sub-block Merge candidate list: It includes ATMVP and affine merge candidates. Both affine mode and ATMVP mode share a merge list construction process. Here, ATMVP and affine merge candidates can be added in sequence. The size of the sub-block merge list is signaled in the stripe header, and the maximum value is 5.

[0234] Unidirectional prediction TPM Merge list: For triangular prediction mode, even though the two partitions can choose their own Merge candidate indexes, one Merge list construction process is shared for the two partitions. When building the Merge list, the spatial neighboring blocks and two temporal blocks of the block are checked. The motion information derived from the spatial and temporal blocks is called regular motion candidates in our IDF. These regular motion candidates are further used to derive multiple TPM candidates. Note that the transformation is performed at the whole block level, even though the two partitions can use different motion vectors to generate their own prediction blocks. The unidirectional prediction TPM Merge list size is fixed to 5.

[0235] Regular Merge list: For the remaining codec blocks, a Merge list construction process is shared. Here, spatial / temporal / HMVP, paired bidirectional prediction Merge candidates, and zero motion candidates can be inserted in sequence. The size of the regular Merge list is signaled in the slice header, and the maximum value is 6. MMVD,

[0236] IBC Merge List: It is done in a similar way as the regular Merge List.

[0237] 2.2.5.1 Sub-block Merge Candidate List

[0238] It is proposed to put all sub-block related motion candidates into a separate Merge list in addition to the regular Merge list of non-sub-block Merge candidates.

[0239] The motion candidates associated with the sub-blocks are put into a separate Merge list called "Sub-block Merge Candidate List".

[0240] In one example, the sub-block Merge candidate list includes affine Merge candidates, and ATMVP candidates and / or sub-block-based STMVP candidates.

[0241] The sub-block Merge candidate list is filled with candidates in the following order:

[0242] ATMVP candidates (may be available or unavailable);

[0243] Affine Merge list (including inherited affine candidates and constructed affine candidates)

[0244] Zero-filled MV 4-parameter affine model

[0245] 2.2.5.1.1 ATMVP (also known as sub-block temporal motion vector predictor, SbTMVP)

[0246] The basic idea of ​​ATMVP is to derive multiple sets of temporal motion vector prediction values ​​for a block. Each sub-block is assigned a set of predictors. When generating ATMVP Merge candidates, the following process is applied:

[0247] a. Check in order Figure 2 The neighboring block A1 shown is used to find the first inter-coded but not CPR-coded block, marked as block X;

[0248] b. Initialize TMV=(0,0). If there is an MV for block X (labeled as MV'), then TMV is set equal to MVn with reference to the collocated reference picture (as signaled in the slice header).

[0249] c. Assuming the center point of the current block is (x0, y0), locate the corresponding position (x0, y0) in the collocated picture as M = (x0 + MV'x, y0 + MV'y). Find the block Z covering M.

[0250] i. If Z is intra-coded, ATMVP is not available;

[0251] ii. If Z is inter-coded, MVZ_0 and MVZ_1 of the two lists of block Z are scaled to (Reflist0index 0) and (Reflist 1index 0) as MVdefault0 and MVdefault1, and stored.

[0252] d. For each 8×8 sub-block, assume its center point is (x0S, y0S), and then locate the corresponding (x0S, y0S) position in the collocated image as MS=(x0S+MV'x, y0S+MV'y). Find the block ZS covering MS.

[0253] i. If ZS is intra-frame coded and decoded, MVdefault0 and MVdefault1 are assigned to the sub-block;

[0254] ii. If the ZS is inter-coded, MVZS_0 and MVZS_1 of the two lists of the block ZS are scaled to (Reflist 0index 0) and (Reflist 1index 0) and allocated to the sub-blocks.

[0255] 2.2.5.2 Conventional Merge List

[0256] Different from the Merge list design, in VVC, a history-based motion vector prediction (HMVP) method is adopted.

[0257] In HMVP, the motion information of the previously coded blocks is stored. The motion information of the previously coded blocks is defined as HMVP candidates. Multiple HMVP candidates are stored in a table called HMVP table, and the table is maintained during the real-time encoding / decoding process. When encoding / decoding of a new slice / LCU row / slice starts, the HMVP table is cleared. Whenever there is an inter-coded block and non-sub-block, non-TPM mode, the associated motion information is added as the last entry of the table as a new HMVP candidate. The entire encoding and decoding process is as follows Fig.21 shown.

[0258] HMVP candidates can be used in the construction process of both AMVP and Merge candidate lists. Fig. 22 A modified Merge candidate list construction process is shown. When the Merge candidate list is not full after the TMVP candidate is inserted, the HMVP candidates stored in the HMVP table can be used to fill the Merge candidate list. Considering that a block usually has a higher correlation with the nearest neighboring block in terms of motion information, the HMVP candidates in the table are inserted in descending order of the index. The last entry in the table is added to the list first, and the first entry is added last. Similarly, redundancy removal is applied to the HMVP candidates. Once the total number of available Merge candidates reaches the maximum number of Merge candidates allowed to be signaled, the Merge candidate list construction process terminates.

[0259] Note that all spatial / temporal / HMVP candidates should be coded and decoded in non-IBC mode. Otherwise, they are not allowed to be added to the regular Merge candidate list.

[0260] The HMVP table contains up to 5 regular motion candidates, and each candidate is unique.

[0261] 2.2.5.3 IBC Merge List

[0262] When IBC was added, HMVP candidates were also eligible for the IBC Merge list.

[0263] More specifically, another 5 IBC candidates can be stored. In the current implementation, conventional candidates and IBC candidates are stored in the same HMVP table. However, they are used and updated independently. The first M (M<=5) candidates are used for the use of the conventional Merge / AMVP list; and the remaining N candidates (N<=5) are used for the use of the IBC mode. Two counters are maintained to indicate how many conventional motion candidates and how many IBC motion candidates there are in the HMVP table. Therefore, it is equivalent to using 2 HMVP tables, one for the conventional Merge mode and the other for the IBC mode.

[0264] Shares the same process as regular MV Merge, but TMVP is not allowed, and a zero vector means unavailable because it is invalid. Note that for spatial neighboring blocks, the associated motion information can be added to the IBC Merge list only when it is encoded and decoded in IBC mode. At the same time, for the HMVP part, only the last few HMVP candidates (which are stored IBC motion candidates) can be considered in the IBC Merge list.

[0265] 2.2.6 Shared Merge List

[0266] In order to reduce decoder complexity and support parallel encoding, it is proposed that all leaf codec units (CUs) of an ancestor node in the CU partition tree share the same Merge candidate list to achieve parallel processing of small skip / merge coded CUs. The ancestor node is called a Merge shared node. In the case where the Merge shared node is a leaf CU, a shared Merge candidate list is generated at the Merge shared node.

[0267] More specifically, the following may apply:

[0268] If the block has luma samples no greater than 32 (e.g., 8×4 or 4×8) and is divided into two 4×4 sub-blocks, a Merge list between very small blocks (e.g., two adjacent 4×4 blocks) is used.

[0269] If a block has more than 32 luma samples, but, after partitioning, at least one subblock is smaller than a threshold (32), then all subblocks of the partition share the same Merge list (e.g., 16×4 or 4×16 ternary partitioning or 8×8 quad partitioning).

[0270] This restriction applies only to regular Merge mode, not to TPM and IBC modes.

[0271] 2.2.6.1 Update and use of HMVP table of sub-blocks under Merge shared nodes

[0272] A temporary HMVP table is allocated. Before that, the original HMVP table must be copied to the temporary HMVP table before entering the first CU of a shared region.

[0273] exist Fig.23 Example shown: Example of Merge shared mode, dividing an 8×8 QT into 4 4×4 CUs. Before executing the first CU (for example, the 4×4 CU on the upper left), the history list must be copied to the temporary HMVP table (HmvpCandListShared[]), and for each 4×4 in the shared parent (Merge shared node in the figure), if the regular Merge mode is used for encoding and decoding, it requires the history list of the Merge mode and uses the candidate motion in the temporary HMVP table.

[0274] However, after decoding a 4×4CU, if it is Merge or AMVP or IBC mode, the original HMVP table will be updated accordingly. That is, for each 4×4 in the shared parent (Merge shared node in the figure), if an unconventional Merge mode, such as AMVP mode, is used for encoding and decoding, it requires a history list of AMVP mode and uses motion candidates in the original HMVP table, which may contain motion information of other sub-blocks under the same Merge shared node.

[0275] 3. Examples of Problems Solved by Embodiments

[0276] The current design of the shared Merge list has the following problems:

[0277] The shared Merge list only applies to regular Merge mode. If a block is encoded or decoded in IBC mode, it may still use the motion information of its neighboring blocks, which conflicts with the concept of shared Merge list.

[0278] For smaller blocks, if the same cross-component linear model (CCLM) / local illumination compensation (LIC) is enabled, they will rely on the neighboring reconstructed samples of a sub-block to encode and decode another sub-block. This design reduces the throughput of the pipeline.

[0279] Two HMVP tables are needed, one is the original HMVP table (which includes motion candidates for both normal and IBC modes) and one is a temporary HMVP table for encoding and decoding the shared Merge list. The cache size is doubled.

[0280] If the shared Merge list is applied to all types of modes, then for a Merge sharing node, multiple shared Merge lists may need to be built, and each shared Merge list corresponds to a given mode (such as regular Merge mode, TPM, IBC, etc.) Such a design is still very complicated.

[0281] 4. Examples of Embodiments

[0282] The following detailed techniques should be considered as examples to explain the general concept. These techniques should not be interpreted narrowly. In addition, these techniques can be combined in any way.

[0283] 1. All sub-blocks under the IBC Merge shared node can share the same IBC Merge candidate list.

[0284] a. In one example, any sub-block, if encoded or decoded in IBC Merge mode, uses the same IBC Merge candidate list built for the IBCMercure shared node.

[0285] b. In one example, adding the motion information of a sub-block to the IBC Merge candidate list of another sub-block under the same motion sharing node is not allowed.

[0286] c. In one example, an IBC Merge candidate list is built for the entire shared node block, and the IBC Merge candidate list is shared by all sub-blocks partitioned from the shared node block.

[0287] d. In one example, two HMVP tables may be maintained, a global table and a temporary table.

[0288] i. Before decoding the first sub-block under the motion sharing node, copy the global HMVP table to the temporary HMVP table.

[0289] ii. The motion information of sub-blocks under the same motion sharing node can be used to update the global HMVP table, but is not allowed to be used to update the temporary HMVP table.

[0290] iii. The motion candidates in the temporary HMVP table can be used to construct the IBC Merge candidate list, but the motion candidates in the global HMVP table are not allowed to be used to construct the IBC Merge candidate list.

[0291] iv. For blocks that are not under a motion sharing node (such as blocks with size equal to 8×16), the motion candidates in the global HMVP table can be used to construct the IBC Merge candidate list, and the decoded motion information of these blocks can be used to update the global HMVP table.

[0292] 2. All sub-blocks under the motion sharing node can share the same motion candidate list, such as the AMVP candidate list and the TPM candidate list.

[0293] a) In one example, a candidate list is constructed for the shared node block as a whole.

[0294] b) In one example, for such a motion candidate list construction process, the motion information of any of these sub-blocks is not allowed to be considered in the motion candidate list.

[0295] c) These methods can also be applied to IBC mode, such as IBC AMVP candidate list, IBC TPM candidate list.

[0296] 2. All sub-blocks under the parameter sharing node can share the same cross-component linear model (CCLM) / local illumination compensation (LIC) parameters or other parameters derived at the decoder side.

[0297] a) In one example, shared parameters are derived for the shared node block as a whole.

[0298] b) In one example, reconstructed samples of any of the sub-blocks are not allowed to be used in the parameter derivation process of another of the sub-blocks.

[0299] c) In one example, reference samples of any of the sub-blocks are not allowed to be used in the parameter derivation process of another of the sub-blocks.

[0300] 3. It is proposed that when encoding / decoding a sub-block under the Motion / Merge / IBC Merge shared node, adding motion candidates derived from the HMVP table to the shared Merge / AMVP candidate list is not allowed.

[0301] a) In one example, the shared Merge / AMVP candidate list does not contain any HMVP candidates.

[0302] b) In one example, when a Motion / Merge / IBC Merge shared node is encountered, checking / adding motion candidates from the HMVP table is not allowed.

[0303] c)

[0304] 4. For the above bullet points, the IBC Merge shared node / motion shared node and / or parameter shared node may be defined to be the same as the Merge shared node used for the regular Merge mode.

[0305] a) In one example, if the number of luma samples within a block is not greater than (or less than) a threshold and it is further divided, the block is called an IBC Merge shared node / motion shared node and / or a parameter shared node.

[0306] b) If the number of luma samples within a block is greater than a threshold, but, after partitioning, at least one sub-block is less than the threshold, such a block is called an IBC Merge shared node / motion shared node and / or a parameter shared node.

[0307] c) In one example, the threshold is defined as 32 or 64 or 128.

[0308] d) Alternatively, if a block is divided into multiple sub-blocks, and the width of at least one sub-block is equal to M, and / or the height of at least one sub-block is equal to N, then the block is called an IBC Merge shared node / motion shared node and / or parameter shared node. In one example, both M and N are set to 4.

[0309] e) Alternatively, different thresholds / settings / definitions of M and N for IBC Merge sharing nodes and / or motion sharing nodes and / or parameter sharing nodes may be used.

[0310] 5. Motion information refined by DMVD techniques (eg, DMVR) is not allowed to be used to update the HMVP table.

[0311] 6. It is proposed to disable DMVD processing for motion candidates derived from the HMVP table.

[0312] a) In one example, if the Merge candidate is copied directly from the HMVP table, the DMVD process is skipped.

[0313] b) In one example, if the Merge candidate is generated by motion candidates in the HMVP table, such as a pair of bi-directional prediction candidates derived from at least one candidate in the HMVP table, the DMVD process is skipped.

[0314] c) Alternatively, in addition, the refined motion information of a block obtained by the DMVD process can be used to update the HMVP table.

[0315] 7. It is proposed to disable HMVP table update processing for all sub-blocks under Merge / Motion / IBC Merge / Parameter Sharing node. In this case, the temporary HMVP table will be completely removed.

[0316] a) In one example, before encoding / decoding the first sub-block under a shared node and after encoding / decoding the last sub-block under the shared node, the HMVP table remains unchanged.

[0317] b) Alternatively, the HMTP table update process may be enabled only for a certain sub-block (eg, the last block in the encoding / decoding order).

[0318] c) Alternatively, the motion information of all sub-blocks may be stored first, and the HMVP table may be updated with one or more of the plurality of representative motion information.

[0319] 8. After encoding or decoding a block using a shared Merge / Motion List, how to update the HMVP table may depend on the codec mode information.

[0320] a) In one example, the codec mode information includes a normal Merge mode and an IBC mode.

[0321] 9. The consistency bitstream should meet the following conditions: When a Merge / IBC Merge / Motion Shared node is found, all Merge-encoded sub-blocks share the same mode.

[0322] a) In one example, if a sub-block is encoded or decoded in normal Merge mode, all sub-blocks should be encoded or decoded in normal Merge mode.

[0323] b) In one example, for all sub-blocks, if one sub-block is encoded and decoded in TPM mode, all sub-blocks should be encoded and decoded in TPM mode.

[0324] c) In one example, for all sub-blocks, if one sub-block is encoded or decoded in IBC Merge mode, all sub-blocks should be encoded or decoded in IBC mode.

[0325] 10. It is proposed that when a Merge shared node is found, all sub-blocks share the same Merge list and the same motion candidates in the list.

[0326] a) In one example, the selected motion candidate index may be encoded and decoded only once for all sub-blocks.

[0327] i. In one example, the motion candidate index selected may be encoded only for the sub-blocks of the first Merge encoding in the encoding / decoding order.

[0328] b) Alternatively, multiple motion candidate indices can still be encoded and decoded as in the current design, however, the conforming bitstreams should satisfy the constraint that all are the same.

[0329] 11. It is proposed that when a Merge shared node is found, for a specific mode type (e.g., regular Merge mode, IBC Merge mode, triangle mode), all sub-blocks share the same Merge list and the same motion candidates in the list.

[0330] a) In one example, the selected mode index may be encoded and decoded only once for all sub-blocks.

[0331] i. In one example, only the selected mode index of the first Merge-encoded sub-block may be encoded and decoded in the encoding / decoding order.

[0332] b) Alternatively, multiple mode indices can still be encoded and decoded as in the current design, however, the conforming bitstreams should all be constrained to be the same.

[0333] c) Alternatively, only one specific Merge mode type is allowed, such as the regular Merge mode.

[0334] i. In this case, signaling notifications for other kinds of mode types (eg, TPM, IBC modes) can be skipped.

[0335] 12. It is proposed that the candidate list construction process is different for regular blocks and for Merge / Motion / IBC Merge shared node blocks.

[0336] a) In one example, more candidates can be put into the Merge list of the shared node block.

[0337] b) In one example, different relative positions of spatially adjacent blocks / temporally adjacent blocks may be accessed to obtain Merge candidates of shared node blocks.

[0338] c) In one example, whether to add a motion candidate and / or how to add a motion candidate from the HMVP table may depend on whether the current block is a shared node block.

[0339] d) In one example, more spatial neighboring blocks may be accessed to obtain Merge candidates for shared node blocks.

[0340] e) In one example, more time-domain neighboring blocks may be accessed to obtain Merge candidates for shared node blocks.

[0341] Fig.242400. The apparatus 2400 may be used to implement one or more methods described herein. The apparatus 2400 may be implemented in a smartphone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. The apparatus 2400 may include one or more processors 2402, one or more memories 2404, and video processing hardware 2406. The processor 2402 may be configured to implement one or more methods described herein. The memory (memories) 2404 may be used to store data and code for implementing the methods and techniques described herein. The video processing hardware 2406 may be used to implement some of the techniques described herein in hardware circuits.

[0342] Fig.26 is a flow chart of a method 2600 for processing a video. The method 2600 includes identifying (2605) sub-video blocks that constitute an intra-block copy (IBC) Merge shared node, and further processing (2610) the sub-video blocks using an IBC Merge candidate list associated with the IBC Merge shared node.

[0343] Fig. 27 2700 is a flow chart of a method 2700 for processing a video. The method 2700 includes determining (2705) to derive a motion candidate from a history-based motion vector prediction (HMVP) table, determining (2710) an operational state of a DMVD process based on the determination to derive the motion candidate from the HMVP table, the operational state being enabled or disabled, and further processing (2715) a video block according to the operational state of the DMVD process.

[0344] Fig.28 28 is a flow chart of a method 2800 for processing a video. The method 2800 includes determining (2805) that a sub-video block is associated with a node, the node being one or more of: a Merge sharing node, a motion sharing node, an intra-block copy (IBC) Merge sharing node, or a parameter sharing node, determining (2810) an operational state of a history-based motion vector prediction (HMVP) table update technique based on the determination that the sub-video block is associated with the node, the operational state being enabled or disabled, and further processing (2815) the video block based on the operational state of the HMVP table update technique.

[0345] Fig.29 2900 is a flow chart of a method 2900 for processing a video. The method 2900 includes: determining (2905) codec mode information about a first video block; determining (2910) characteristics of a history-based motion vector prediction (HMVP) table update technique based on the determination of the codec mode information; and further processing (2915) the first video block according to the HMVP table update technique having the characteristics.

[0346] Fig.30 3000 is a flowchart of a method for processing a video. The method 3000 includes: performing (3005) a first candidate list construction technique on a first video block, and performing (3010) a second candidate list construction technique on a second video block, wherein the first video block and the second video block are different types of video blocks, and the second video block is one of the following: a Merge shared node block, a motion shared node block, or an intra-frame block copy (IBC) Merge shared node block, and the first candidate list construction technique is different from the second candidate list construction technique.

[0347] Fig.31 3100 is a flowchart of a method 3100 for processing a video. The method 3100 includes determining (3105) that a first video unit of a video is an intra-block copy (IBC) Merge shared node, and dividing the first video unit into a plurality of sub-blocks, wherein the sub-blocks encoded and decoded in the IBC Merge mode under the first video unit are processed in parallel; deriving (3110) motion information of the first sub-block under the first video unit; and using the derived motion information to convert (3115) between a bitstream representation of the video and the first sub-block, wherein in the derivation of the motion information of the first sub-block, the motion information of any other sub-block under the first video is omitted.

[0348] Fig.32 3200 is a flowchart of a method 3200 for processing a video. The method 3200 includes determining (3205) that a first video unit of a video is an intra-block copy (IBC) advanced motion vector prediction (AMVP) shared node, and dividing the first video unit into a plurality of sub-blocks, wherein the sub-blocks encoded and decoded in the IBC AMVP mode under the first video unit are processed in parallel; deriving (3210) motion information of the first sub-block under the first video unit; and converting (3215) between a bitstream representation of the video and the first sub-block using the derived motion information, wherein in the derivation of the motion information of the first sub-block, the motion information of any other sub-block under the first video is omitted.

[0349] Fig.33is a flowchart of a method 3300 for processing a video. The method 3300 includes determining (3305) that a first video unit of a video is an inter-frame advanced motion vector prediction (AMVP) motion sharing node, and dividing the first video unit into a plurality of sub-blocks, wherein the sub-blocks encoded and decoded in the inter-frame AMVP mode under the first video unit are processed in parallel; deriving (3310) motion information of a first sub-block among the plurality of sub-blocks encoded and decoded in the inter-frame AMVP mode under the first video unit; and based on the derived motion information, converting (3315) between a first sub-block of the plurality of sub-blocks and a bitstream representation of a video including the first video unit, wherein in the derivation of the motion information of the first sub-block, the motion information of any other sub-blocks under the first video unit is omitted.

[0350] Fig.34 3400 is a flowchart of a method 3400 for processing a video. The method 3400 includes determining (3405) that a first video unit of a video is a parameter sharing node, and the first video unit is divided into a plurality of sub-blocks; during conversion between the first sub-block and a bitstream representation of the video including the first video unit, deriving (3410) parameters of a first sub-block among the plurality of sub-blocks under the first video unit, wherein the first sub-block is encoded and decoded using a codec tool; and performing (3415) conversion based on the parameters, wherein the parameters associated with the codec tool derived during the conversion are independent of information of a second sub-block under the same parameter sharing node.

[0351] Fig.35 is a flowchart of a method 3500 for processing a video. The method 3500 includes determining (3505) that a first video unit of a video is a motion sharing node, and dividing the first video unit into a plurality of sub-blocks as leaf nodes; deriving (3510) a motion candidate list for a first sub-block under the first video unit; and based on the motion candidate list, performing (3515) a conversion between the first sub-block of the plurality of sub-blocks encoded and decoded in a first mode and a bitstream representation of the video including the first video unit, wherein motion information of a second sub-block of the plurality of sub-blocks encoded and decoded in the first mode is omitted in the process of constructing the motion candidate list, and wherein, when encoding or decoding the first sub-block, a motion candidate derived from a history-based motion vector prediction (HMVP) table is not added to the motion candidate list.

[0352] Fig.36is a flowchart of a method 3600 for processing a video. The method 3600 includes predicting (3605) motion information of a first video block by using a motion list constructed by a first candidate list construction technique for the first video block when the first video block is outside a motion sharing node during a conversion between a first video block and a bitstream representation of a video containing the first video block; predicting (3610) motion information of the second video block by using a Merge list constructed by a second candidate list construction technique for the second video block during a conversion between a second video block of a first video unit and a bitstream representation of the video, the second candidate list construction technique being different from the first candidate list construction technique, and the first video unit being a sharing node block; and performing (3615) conversion based on the predicted motion information of the first and second video blocks.

[0353] Fig.37 is a flow chart of a method 3700 for processing a video. The method 3700 includes maintaining (3705) one or more tables, wherein each table includes motion candidates derived from a previously processed video block that was processed before a first video block of a video; deriving (3710) motion information for the first video block; applying (3715) refinement to the motion information when at least one decoder-side motion vector derivation (DMVD) technique is enabled; converting (3720) between a bitstream representation of the first video block and the first video block using the refined motion information; and determining (3725) whether to use the refined motion information or the unrefined motion information for updating the one or more tables.

[0354] Fig.38 is a flow chart of a method 3800 for processing a video. The method 3800 includes maintaining (3805) one or more tables, wherein each table includes motion candidates derived from a previously processed video block that was processed before a first video block of the video; deriving (3810) motion information for the first video block; applying (3815) refinement to the motion information when at least one decoder-side motion vector derivation (DMVD) technique is enabled; converting (3820) between a bitstream representation of the first video block and the first video block using the refined motion information; and updating (3825) the one or more tables with unrefined motion information for the first video block.

[0355] Fig.39is a flow chart of a method 3900 for processing a video. The method 3900 includes maintaining (3905) one or more tables, wherein each table includes motion candidates derived from a previously processed video block that was processed before a first video block in a video; deriving (3910) motion information for the first video block; determining (3915) to enable or disable refinement of the motion information for the first video block using at least one decoder-side motion vector derivation (DMVD) technique based on whether the motion information is derived from one or more tables; and converting (3920) between a bitstream representation of the video and the first video block based on the determination.

[0356] Referring to methods 2600-3900, some examples of motion prediction and its use under a shared Merge list are described in Section 4 of this document.

[0357] Referring to methods 2600-3900, video blocks may be encoded in a video bitstream, wherein bit efficiency may be achieved by using bitstream generation rules related to motion prediction under a shared Merge list.

[0358] The following examples are provided in this disclosure.

[0359] 1. A method of video processing, comprising: maintaining one or more tables, wherein each table includes motion candidates derived from a previously processed video block that was processed before a first video block of a video; deriving motion information for the first video block; applying refinement to the motion information when at least one decoder-side motion vector derivation (DMVD) technique is enabled; converting between a bitstream representation of the first video block and the first video block using the refined motion information; and determining whether to update the one or more tables using the refined motion information or the unrefined motion information.

[0360] 2. The method of example 1, wherein updating the one or more tables with refined motion information is prohibited.

[0361] 3. The method of example 1, wherein updating the one or more tables with unrefined motion information is enabled.

[0362] 4. The method of any of Examples 1-3, wherein updating the one or more tables with refined motion information is enabled.

[0363] 5. A video processing method, comprising: maintaining one or more tables, wherein each table includes motion candidates derived from a previously processed video block that was processed before a first video block of a video; deriving motion information for the first video block; applying refinement to the motion information when at least one decoder-side motion vector derivation (DMVD) technique is enabled; converting between a bitstream representation of the first video block and the first video block using the refined motion information; and updating the one or more tables with unrefined motion information for the first video block.

[0364] 6. The method of example 5, wherein updating the one or more tables with refined motion information is prohibited.

[0365] 7. A method for processing a video, comprising: maintaining one or more tables, wherein each table includes motion candidates derived from a previously processed video block that is processed before a first video block in a video; deriving motion information for the first video block; determining, based on whether the motion information is derived from one or more tables, using at least one decoder-side motion vector derivation (DMVD) technique to enable or disable refinement of the motion information for the first video block; and converting between a bitstream representation of the video and the first video block based on the determination.

[0366] 8. The method of example 7, wherein DMVD processing is disabled when deriving the motion information from the one or more tables.

[0367] 9. The method of example 7, further comprising: skipping the DMVD processing if the motion information is copied directly from a candidate in the one or more tables.

[0368] 10. The method of example 7, further comprising: skipping the DMVD process if the motion information is generated using motion candidates in the one or more tables.

[0369] 11. The method of Example 10, wherein the motion information comes from pairs of bidirectional prediction candidates derived from at least one candidate in the one or more tables.

[0370] 12. The method of any one of Examples 1-11, wherein the DMVD processing comprises at least one of a decoder motion vector refinement (DMVR) and a bidirectional optical flow (BDOF) technique.

[0371] 13. A video processing method, comprising: maintaining one or more tables, each of which includes motion candidates; determining that a first video unit of a video is a shared node, wherein sub-blocks encoded and decoded in a specific mode under the first video unit are processed in parallel; deriving motion information of a first sub-block under the first video unit; using the derived motion information to convert between a bitstream representation of the video and the first sub-block; and determining whether to use the derived motion information of the first sub-block to update the one or more tables.

[0372] 14. A method as described in Example 13, wherein the specific mode is an inter-frame Merge mode, the shared node is a Merge shared node, and one or more table update processes using motion information of the first sub-block under the shared node are disabled, wherein the first sub-block is encoded and decoded in the inter-frame Merge mode.

[0373] 15. A method as described in Example 13, wherein the specific mode is the inter-frame AMVP mode, the shared node is an inter-frame AMVP shared node, and one or more table update processes using the motion information of the first sub-block under the shared node are disabled, wherein the first sub-block is encoded and decoded in the inter-frame AMVP mode.

[0374] 16. A method as described in Example 13, wherein the specific mode is IBC Merge mode, the shared node is an IBC Merge shared node, and one or more table update processes using motion information of the first sub-block under the shared node are disabled, wherein the first sub-block is encoded and decoded in IBC Merge mode.

[0375] 17. A method as described in Example 13, wherein the specific mode is the inter-frame Merge mode, the shared node is an IBC AMVP shared node, and one or more table update processes using the motion information of the first sub-block under the shared node are disabled, wherein the first sub-block is encoded and decoded in the IBC AMVP mode.

[0376] 18. The method of example 13, further comprising: disabling one or more table update processes that use motion information of any sub-block under the shared node.

[0377] 19. The method of example 13, wherein one or more table update processes are enabled only for a specific sub-block and are disabled for all other sub-blocks under the shared node.

[0378] 20. The method of Example 19, wherein the specific sub-block is the last sub-block under the shared node in coding order.

[0379] 21. The method of Example 19, wherein the particular sub-block is the last sub-block under the shared node in decoding order.

[0380] 22. The method as described in Example 13 also includes: storing motion information of all sub-blocks; and updating one or more tables by using one or more of multiple representative motion information in the motion information of all sub-blocks.

[0381] 23. A method as described in any of Examples 1-22, wherein after encoding and decoding the sub-blocks under the shared node, how to update the one or more tables depends on encoding and decoding mode information.

[0382] 24. The method of Example 23, wherein the coding and decoding mode information includes a conventional Merge mode and an IBC mode.

[0383] 25. The method of any of Examples 1-24, wherein the one or more tables are history-based motion vector prediction (HMVP) tables.

[0384] 26. The method of any of Examples 1-24, wherein the updated one or more tables are used to process subsequent video blocks of the video.

[0385] 27. A video decoding device, comprising: a processor configured to implement one or more of the methods described in Examples 1 to 26.

[0386] 28. A video encoding device, comprising: a processor configured to implement one or more of the methods described in Examples 1 to 26.

[0387] 29. A computer program product comprising computer code stored thereon, which, when executed by a processor, causes the processor to implement the method as described in any one of Examples 1 to 26.

[0388] The method may include: wherein the motion information of the first sub-video block is excluded from an IBC Merge candidate list of a second sub-video block added to the same motion sharing node.

[0389] The method may include: wherein performing further processing of the sub-video block includes maintaining a global history-based motion vector prediction (HMVP) table and a temporary HMVP table.

[0390] The method may include: copying content from a global HMVP table to a temporary HMVP table; and after copying the content from the global HMVP table to the temporary HMVP table, decoding the first sub-video block.

[0391] The method may include: wherein the motion information of the first sub-video block is used to update a global HMVP table instead of being used to update a temporary HMVP table.

[0392] The method may include: wherein the motion candidates stored in the temporary HMVP table are used to construct an IBC Merge candidate list.

[0393] The method may include: wherein the sub-video blocks constitute a motion sharing node, and the sub-video blocks share the same motion candidate list.

[0394] The method may include: wherein the same motion candidate list is an advanced motion vector prediction (AMVP) candidate list.

[0395] The method may include: wherein, building the same motion candidate list for the shared node blocks.

[0396] The method may include: wherein the sub-video blocks constitute a parameter sharing node, and the sub-video blocks share the same cross-component linear model (CCLM), local illumination compensation (LIC) parameters, or other parameters derived at the decoder end.

[0397] The method may include: wherein shared parameters are derived for the shared node blocks.

[0398] The method may include: wherein, in a parameter derivation process of a second sub-video block of the sub-video blocks, a reconstructed sample of a first sub-video block of the sub-video blocks is not used.

[0399] The method may include: wherein, in a parameter derivation process of a second sub-video block of the sub-video blocks, a reference sample of a first sub-video block of the sub-video blocks is not used.

[0400] The method may include wherein motion candidates derived from a history based motion vector prediction (HMVP) table are not added to the IBC Merge candidate list.

[0401] The method may include: wherein the motion sharing node or the parameter sharing node is defined to be the same as the Merge sharing node used for the regular Merge mode.

[0402] The method may include: wherein the DMVD processing is a decoder-side motion vector derivation (DMVD) process.

[0403] The method may include wherein disabling an operating state based on a Merge candidate copied from the HMVP table.

[0404] The method may include wherein the operating state is disabled based on a Merge candidate generated with a motion candidate from a HMVP table.

[0405] The method may include updating an HMVP table with refined motion information for the video block, wherein the operating state is enabled.

[0406] The method may include wherein the operating state is enabled for a first one of the sub-video blocks that is a last block in a coding order or a decoding order.

[0407] The method may include: wherein a consistent bitstream is satisfied based on a found Merge shared node or an intra-block copy (IBC) Merge shared node, and each sub-video block that is Merge-encoded shares the same mode.

[0408] The method may include: wherein the same mode is a regular Merge mode.

[0409] The method may include: wherein the same mode is a triangular prediction mode (TPM).

[0410] The method may include: wherein the same mode is an intra block copy (IBC) Merge mode.

[0411] The method may include: finding a Merge sharing node, and the sub-video blocks of the Merge sharing node share a Merge list and share motion candidates in the Merge list.

[0412] The method may include: wherein, a selected motion candidate index is encoded for a sub-video block.

[0413] The method may include: wherein the selected motion candidate index is encoded and decoded based on the sub-video block being the first Merge-encoded sub-video block in the encoding order or the decoding order.

[0414] The method may include: wherein a Merge sharing node is found, and the sub-video blocks share a Merge list and motion candidates in the Merge list of the mode type.

[0415] The method may include, wherein the mode type is a normal Merge mode, an Intra Block Copy (IBC) Merge mode, or a triangular prediction mode.

[0416] It should be understood that the disclosed technology can be embodied in a video encoder or decoder to improve compression efficiency using an enhanced codec tree structure.

[0417] The disclosure and other solutions, examples, embodiments, modules and functional operations described in this document may be implemented in digital electronic circuits, or in computer software, firmware or hardware, including the structures disclosed in this document and their structural equivalents, or in a combination of one or more of them. The disclosure and other embodiments may be implemented as one or more computer program products, i.e., one or more computer program instruction modules encoded on a computer-readable medium, for execution by a data processing device or for controlling the operation of the data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material combination that affects a machine-readable propagation signal, or a combination of one or more of them. The term "data processing device" encompasses all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal, such as a machine-generated electrical signal, an optical signal or an electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0418] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including stand-alone programs or modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). A computer program may be deployed to execute on one computer or on multiple computers located on one site or distributed on multiple sites and interconnected by a communications network.

[0419] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0420] For example, processors suitable for executing computer programs include general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include or be operably coupled to one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, to receive data from the one or more mass storage devices, or to transfer data to the one or more mass storage devices, or to both receive and transfer data. However, the computer does not need to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

[0421] Although this patent document contains many details, these details should not be interpreted as limitations on any subject matter or the scope of protection that can be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. In this patent document, certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments, either individually or in any suitable subcombination. In addition, although the above features may be described as working in certain combinations and even initially claimed as such, in some cases, one or more features from the combination may be removed from the claimed combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0422] Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve the desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0423] Only a few implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and shown in this patent document.

Claims

1. A video processing method, comprising: maintaining one or more tables, wherein each table includes motion candidates derived from a previously processed video block that was processed before a first video block of the video; deriving motion information of the first video block; applying refinement to the motion information when at least one decoder-side motion vector derivation (DMVD) technique is enabled; converting between a bitstream representation of the first video block and the first video block using the refined motion information; and A determination is made whether to use the refined motion information or the unrefined motion information to update the one or more tables.

2. The method of claim 1, wherein: Updating the one or more tables with refined motion information is prohibited.

3. The method of claim 1, wherein: Updating the one or more tables with unrefined motion information is enabled.

4. The method of claim 1, wherein: Updating the one or more tables with refined motion information is enabled.

5. The video processing method according to claim 1, further comprising: determining, based on whether the motion information is derived from one or more tables, to enable or disable refinement of the motion information for the first video block using at least one decoder-side motion vector derivation (DMVD) technique; as well as Based on the determination, a conversion is performed between a bitstream representation of the video and the first video block.

6. The method of claim 5, wherein: When the motion information is derived from the one or more tables, DMVD processing is disabled.

7. The method of claim 5, further comprising: If the motion information is copied directly from the candidate in the one or more tables, the DMVD process is skipped.

8. The method of claim 5, further comprising: If the motion information is generated using motion candidates in the one or more tables, the DMVD process is skipped.

9. The method of claim 8, wherein: The motion information is derived from pairs of bi-directional prediction candidates derived from at least one candidate in the one or more tables.

10. The method according to any one of claims 1 to 9, wherein: The DMVD processing includes at least one of decoder motion vector refinement DMVR and bidirectional optical flow BDOF technology.

11. The video processing method according to claim 1, further comprising: Determining that a first video unit of the video is a shared node, wherein sub-blocks encoded and decoded in a specific mode under the first video unit are processed in parallel; deriving motion information of a first sub-block under the first video unit; Using the derived motion information to convert between a bitstream representation of the video and the first sub-block; as well as A determination is made whether to update the one or more tables using the derived motion information of the first sub-block.

12. The method of claim 11, wherein: The specific mode is the inter-frame Merge mode, the shared node is a Merge shared node, and one or more table update processes using motion information of the first subblock under the shared node are disabled, wherein the first subblock is encoded and decoded in the inter-frame Merge mode.

13. The method of claim 11, wherein: The specific mode is the inter-AMVP mode, the shared node is the inter-AMVP shared node, and one or more table update processes using motion information of the first sub-block under the shared node are disabled, wherein the first sub-block is encoded and decoded in the inter-AMVP mode.

14. The method of claim 11, wherein: The specific mode is an IBC Merge mode, the shared node is an IBC Merge shared node, and one or more table update processes using motion information of the first sub-block under the shared node are disabled, wherein the first sub-block is encoded and decoded in the IBC Merge mode.

15. The method of claim 11, wherein: The specific mode is the inter-frame Merge mode, the shared node is the IBC AMVP shared node, and one or more table update processes using the motion information of the first sub-block under the shared node are disabled, wherein the first sub-block is encoded and decoded in the IBC AMVP mode.

16. The method of claim 11, further comprising: One or more table update processes that use motion information of any sub-block under the shared node are disabled.

17. The method of claim 11, wherein: One or more table update processes are enabled only for a specific sub-block, and are disabled for all other sub-blocks under the shared node.

18. The method of claim 17, wherein: The specific sub-block is the last sub-block under the shared node in the coding order.

19. The method of claim 17, wherein: The specific sub-block is the last sub-block under the shared node in decoding order.

20. The method of claim 11, further comprising: Store the motion information of all sub-blocks; as well as The one or more tables are updated by using one or more of a plurality of representative motion information among the motion information of all sub-blocks.

21. The method according to any one of claims 11 to 20, wherein: After encoding and decoding the sub-blocks under the shared node, how to update the one or more tables depends on the encoding and decoding mode information.

22. The method of claim 21, wherein: The encoding and decoding mode information includes a conventional Merge mode and an IBC mode.

23. A video processing method, comprising: maintaining one or more tables, wherein each table includes motion candidates derived from a previously processed video block that was processed before a first video block of the video; deriving motion information of the first video block; applying refinement to the motion information when at least one decoder-side motion vector derivation (DMVD) technique is enabled; converting between a bitstream representation of the first video block and the first video block using the refined motion information; and The one or more tables are updated with unrefined motion information for the first video block.

24. The method of claim 23, wherein: Updating the one or more tables with refined motion information is prohibited.

25. The method of any one of claims 1 to 24, wherein: The one or more tables are history-based motion vector prediction HMVP tables.

26. The method of any one of claims 1 to 24, wherein: The updated one or more tables are used to process subsequent video blocks of the video.

27. A video decoding device, comprising: A processor configured to implement the method according to any one of claims 1 to 26.

28. A video encoding device, comprising: A processor configured to implement the method according to any one of claims 1 to 26.

29. A computer program product comprising computer code stored thereon, which, when executed by a processor, causes the processor to implement the method of any one of claims 1 to 26.

Images