History-based motion vector prediction with default parameters

Abstract

Video encoding and decoding techniques including history-based motion vector prediction with default parameters are described. An exemplary video processing method includes: for conversion between a video unit of a video and a bitstream representation of the video, resetting a size N HMVP table by adding K history-based motion vector prediction (HMVP) motion candidates to the HMVP table, wherein the K HMVP candidate motion candidates have default values, wherein N and K are positive integers, and wherein K≤N; and performing conversion between the video unit and the bitstream representation using the HMVP table resulting from the reset.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application timely claims the priority and benefit of International Patent Application No. PCT / CN2019 / 090451 filed on June 8, 2019, and International Patent Application No. PCT / CN2019 / 094703 filed on July 4, 2019, in accordance with applicable patent laws and / or the rules applicable to the Paris Convention. For all legal purposes, the entire disclosure of the above-mentioned application is incorporated by reference as part of the disclosure of this application. Technical Field

[0003] This patent document relates to video encoding and decoding technology, equipment and systems. Background Art

[0004] Despite advances in video compression, digital video still accounts for the largest use of bandwidth on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth required for digital video usage is expected to continue to grow. Summary of the invention

[0005] Various embodiments and techniques are described herein for performing video encoding or decoding using history-based motion vector prediction (HMVP) with default parameters.

[0006] In one exemplary aspect, a video processing method is disclosed. The method includes: for conversion between a video unit of a video and a bitstream representation of the video, resetting a size N of a HMVP table by adding K history-based motion vector prediction (HMVP) motion candidates to the HMVP table, wherein the K HMVP candidate motion candidates have default values, wherein N and K are positive integers, and wherein K≤N; and performing conversion between the video unit and the bitstream representation using the HMVP table obtained from the reset.

[0007] In another exemplary aspect, a video processing method is disclosed. The method includes: adding a history-based motion vector prediction (HMVP) table of a video unit from a first set of K default HMVP motion candidates, wherein the first set of K HMVP motion candidates of the video unit are adaptively changed from a second set of K HMVP motion candidates included in the HMVP table for a second video unit of the video, wherein the second video unit is different from the video unit, wherein the first set of K HMVP motion candidates and the second set of K HMVP motion candidates have default values, wherein the size of the HMVP table is N, wherein N and K are positive integers, and wherein K≤N; and performing conversion between the video unit of the video and a bitstream representation of the video, wherein the bitstream representation includes a syntax element indicating the first set of K HMVP motion candidates added to the HMVP table.

[0008] In yet another exemplary aspect, a video processing method is disclosed. The method includes: when determining that a history-based motion vector prediction (HMVP) table of size N is reset, populating the HMVP table with a set of K default HMVP candidates, where N and K are positive integers, and where K≤N; and performing conversion between a current block and a bitstream representation of the current block based on the HMVP table.

[0009] In yet another exemplary aspect, another video processing method is disclosed. The method includes: for a conversion between a current block and a bitstream representation of the current block, determining a set of default HMVPs to be used during the conversion; and performing the conversion using the set of default HMVP candidates, wherein different HMVP candidates are used during the conversion of different video units.

[0010] In yet another exemplary aspect, another video processing method is disclosed. The method includes: determining that a set of N default history-based motion vector prediction (HMVP) candidates has been populated in an HMVP table before converting between a current block and a bitstream representation of the current block; adding motion candidates to the HMVP table; and performing the conversion using the HMVP table containing the motion candidates.

[0011] In another exemplary aspect, another video processing method is disclosed. The method includes: storing information of an interpolation filter for the current video block, or side information about a switchable interpolation filter for the current video block, together with past motion information of the current video block in a table before conversion between a current video block and a bitstream representation of the current video block; performing conversion based on entries of the table; and updating the table after performing the conversion.

[0012] In another exemplary aspect, a video processing device is disclosed. The device includes a processor configured to execute the above disclosed method. The device can also perform video encoding or video decoding.

[0013] In yet another exemplary aspect, a computer-readable medium is disclosed, wherein a processor-implemented code for the above method is stored on the medium.

[0014] These and other aspects are described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 An example derivation process for Merge candidate list construction is shown.

[0016] Figure 2 An example of the location of spatial merge candidates is shown.

[0017] Figure 3 An example of candidate pairs for redundancy checking considering spatial Merge candidates is shown.

[0018] Figure 4A-4B Example locations of the second PU for Nx2N and 2NxN partitions are shown.

[0019] Figure 5 is an illustration of motion vector scaling for temporal merge candidates.

[0020] Figure 6 Examples C0 and C1 of candidate positions of the time-domain Merge candidate are shown.

[0021] Figure 7 An example of combining bi-predictive Merge candidates is shown.

[0022] Figure 8 The derivation process for motion vector prediction candidates is summarized.

[0023] Fig. 9 A diagram of motion vector scaling for spatial motion vector candidates is shown.

[0024] Fig. 10A and 10B An example of a simplified affine motion model is shown.

[0025] Fig.11 An example of the affine MVF for each sub-block is shown.

[0026] Fig.12 Examples of candidate positions for the Affine Merge mode are shown.

[0027] Fig.13 An example of a modified Merge list construction process is shown.

[0028] Fig.14 An example of inter prediction based on triangular partitioning is shown.

[0029] Fig.15 An example is shown in which a CU applies a first weighting factor group.

[0030] Fig.16 An example of a motion vector storage implementation is shown.

[0031] Fig.17 An embodiment of a final motion vector expression (UMVE) search process is shown.

[0032] Fig.18 An example of a UMVE search point is shown.

[0033] Fig.19 An example of MVD (0, 1) mirrored between list 0 and list 1 in DMVR is shown.

[0034] Fig. 20 An example of motion vectors (MVs) that may be checked in one iteration is shown.

[0035] Fig.21 An example of intra block copy (IBC) is shown.

[0036] Fig. 22 is a block diagram of an example of a hardware platform for implementing the methods described in this document.

[0037] Fig.23 is a flow chart of an example method of video processing.

[0038] Fig.24 A table showing extended signaling for AMVR.

[0039] Fig.25 An example of an interpolation filter used in video encoding or decoding is shown.

[0040] Fig.26 is a block diagram illustrating an exemplary video codec system that may utilize the techniques of this disclosure.

[0041] Fig. 27 is a block diagram illustrating an example of a video encoder.

[0042] Fig.28 is a block diagram illustrating an example of a video decoder.

[0043] Fig.29 is a block diagram illustrating an example video processing system in which the various techniques disclosed herein may be implemented.

[0044] Figures 30 to 31A flow chart illustrating an example method of video processing is shown. DETAILED DESCRIPTION

[0045] The use of section headings in this document is for ease of understanding and does not limit the embodiments disclosed in a section to only that section. In addition, although certain embodiments are described with reference to a versatile video codec or other specific video codecs, the disclosed techniques are also applicable to other video codec techniques. In addition, although some embodiments describe video encoding steps in detail, it should be understood that the corresponding steps of undo encoding decoding will be implemented by the decoder. In addition, the term video processing includes video encoding or compression, video decoding or decompression, and video transcoding, in which video pixels are represented from one compression format to another compression format, or at different compression bit rates.

[0046] 1. Overview

[0047] This patent document relates to video coding technology. Specifically, it relates to symmetric motion vector difference coding. It can be applied to existing video coding standards (such as HEVC), or to a pending standard (multi-function video codec). It may also be applicable to future video coding standards or video codecs.

[0048] 2. Initial Discussion

[0049] Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T produced H.261 and H.263, ISO / IEC produced MPEG-1 and MPEG-4 Visual, and the two organizations worked together to produce the H.262 / MPEG-2 Video and H.264 / MPEG-4 Enhanced Video Codec (AVC) and H.265 / HEVC standards. Starting with H.262, video codec standards are based on a hybrid video codec structure that utilizes temporal prediction and transform codecs. In order to explore future video codec technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and introduced them into a reference software called the "Joint Exploration Model" (JEM). In April 2018, a Joint Video Experts Team (JVET) between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) was established to work on the VVC standard with the goal of reducing bitrate by 50% compared to HEVC.

[0050] 2.1 Inter-frame prediction in HEVC / H.265

[0051] For inter-coded codec units (CUs), they can be coded with one prediction unit (PU) or two PUs depending on the partitioning mode. Each inter-predicted PU has motion parameters for one or two reference picture lists. The motion parameters include motion vectors and reference picture indexes. The use of one of the two reference picture lists can also be signaled using inter_pred_idc. Motion vectors can be explicitly coded as deltas relative to the predictor.

[0052] When a CU is coded in skip mode, a PU is associated with the CU and has no significant residual coefficients, no coded motion vector increments or reference picture indices. A Merge mode is specified, by which the motion parameters of the current PU can be obtained from neighboring PUs (including spatial and temporal candidates). Merge mode can be applied to any inter-predicted PU, not just skip mode. An alternative to Merge mode is the explicit transmission of motion parameters, where the motion vector (more precisely, the motion vector difference (MVD) compared to the motion vector predictor), the corresponding reference picture index of each reference picture list, and the use of the reference picture list are explicitly signaled per PU. In the present disclosure, such a mode is named Advanced Motion Vector Prediction (AMVP).

[0053] When signaling indicates that one of the two reference picture lists is to be used, a PU is generated from a block of samples. This is called "unidirectional prediction". Unidirectional prediction is available for both P slices and B slices.

[0054] When signaling indicates that two reference picture lists are to be used, a PU is generated from two sample blocks. This is called "bidirectional prediction". Bidirectional prediction is only available for B slices.

[0055] The following provides details about the inter prediction mode specified in HEVC. The description will start with the Merge mode.

[0056] 2.1.1 Reference Image List

[0057] In HEVC, the term inter prediction is used to refer to predictions derived from data elements (e.g., sample values ​​or motion vectors) of reference pictures other than the currently decoded picture. As in H.264 / AVC, a picture can be predicted from multiple reference pictures. Reference pictures used for inter prediction are organized in one or more reference picture lists. A reference index identifies which reference pictures in the list should be used to create the prediction signal.

[0058] A single reference picture list, List 0, is used for P slices, and two reference picture lists, List 0 and List 1, are used for B slices. It should be noted that the reference pictures contained in Lists 0 / 1 may be from both past and future pictures in terms of capture / display order.

[0059] 2.1.2Merge Mode

[0060] 2.1.2.1 Merge Mode Candidate Derivation

[0061] When predicting a PU using Merge mode, an index pointing to an entry in the Merge candidate list is parsed from the bitstream and used to retrieve motion information. The construction of this list is specified in the HEVC standard and can be summarized according to the following steps in order:

[0062] Step 1: Initial candidate derivation

[0063] oStep 1.1: Spatial Candidate Derivation

[0064] oStep 1.2: Redundancy check of spatial candidates

[0065] oStep 1.3: Time Domain Candidate Derivation

[0066] Step 2: Insert additional candidates

[0067] oStep 2.1: Create bidirectional prediction candidates

[0068] oStep 2.2: Insert zero motion candidates

[0069] These steps are also Figure 1 As schematically shown in . For spatial Merge candidate derivation, up to four Merge candidates are selected from candidates located at five different positions. For temporal Merge candidate derivation, up to one Merge candidate is selected from two candidates. Since a constant number of candidates is assumed at the decoder for each PU, additional candidates are generated when the number of candidates obtained from step 1 does not reach the maximum number of Merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, truncated unary binarization (TU) is used to encode and decode the index of the best Merge candidate. If the size of the CU is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

[0070] Hereinafter, operations associated with the aforementioned steps are described in detail.

[0071] Figure 1 An example derivation process for Merge candidate list construction is shown.

[0072] 2.1.2.2 Derivation of Spatial Candidates

[0073] In the derivation of spatial merge candidates, Figure 2 Up to four Merge candidates are selected from the candidates at the depicted position. The order of derivation is A1, B1, B0, A0 and B2. Position B2 is considered only when any PU at position A1, B1, B0, A0 is unavailable (for example, because it belongs to another stripe or slice) or is intra-coded. After adding the candidate at position A1, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving the coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only the possible candidate pairs are considered. Figure 3 , and adds a candidate to the list only if the corresponding candidate for redundancy check has different motion information. Another source of repeated motion information is a "second PU" associated with a partition different from 2N×2N. As an example, Figure 4A and 4B The second PU is depicted for the N×2N and 2N×N cases, respectively. When the current PU is partitioned as N×2N, the candidate at position A1 is not considered for list construction. In fact, adding this candidate would result in two prediction units with the same motion information, which is redundant for having only one PU in the codec unit. Similarly, when the current PU is partitioned as 2N×N, position B1 is not considered.

[0074] 2.1.2.3 Derivation of time domain candidates

[0075] In this step, only one candidate is added to the list. Specifically, in the derivation of the temporal Merge candidate, the scaled motion vector is derived based on the collocated PU, which belongs to the picture with the smallest POC difference with the current picture in a given reference picture list. The reference picture list to be used for deriving the collocated PU is explicitly signaled in the slice header. Figure 5As shown by the dotted line, a scaled motion vector for the temporal Merge candidate is obtained, which is scaled from the motion vector of the collocated PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal Merge candidate is set equal to zero. The actual implementation of the scaling process is described in the HEVC specification. For B slices, two motion vectors are obtained, one for reference picture list 0 and the other for reference picture list 1, and the two motion vectors are combined to obtain a bidirectional prediction Merge candidate.

[0076] 2.1.2.4 Collocation of pictures and PUs

[0077] When TMVP is enabled (ie, slice_temporal_mvp_enabled_flag is equal to 1), the variable ColPic representing the collocated picture is derived as follows:

[0078] - If the current slice is a B slice and the signaled collocated_from_l0_flag is equal to 0, ColPic is set equal to RefPicList1[collocated_ref_idx].

[0079] - Otherwise (slice_type is equal to B, and collocated_from_l0_flag is equal to 1, or slice_type is equal to P), ColPic is set equal to RefPicList0[collocated_ref_idx].

[0080] Herein, collocated_ref_idx and collocated_from_l0_flag are two syntax elements that may be signaled in a slice header.

[0081] In the collocated PU(Y) belonging to the reference frame, the position of the temporal candidate is selected between candidates C0 and C1, such as Figure 6 As shown. If the PU at position C0 is not available, is intra-coded, or is outside the current codec tree unit (CTU also known as LCU, largest codec unit) row, position C1 is used. Otherwise, position C0 is used for the derivation of the time domain Merge candidate.

[0082] The relevant syntax elements are described as follows:

[0083] 7.3.6.1 Generic Stripe Segment Header Syntax

[0084]

[0085] 2.1.2.5 Derivation of MVs for TMVP Candidates

[0086] More specifically, to derive TMVP candidates, the following steps are performed:

[0087] 1) Set reference picture list X = 0, and the target reference picture is the reference picture with index equal to 0 (ie, curr_ref) in list X. Call the derivation process of the concatenated motion vector to obtain the MV of list X pointing to curr_ref.

[0088] 2) If the current slice is a B slice, set reference picture list X = 1, and the target reference picture is the reference picture with index equal to 0 (ie, curr_ref) in list X. Call the derivation process of the concatenated motion vector to obtain the MV of list X pointing to curr_ref.

[0089] The next subsection 2.1.2.5.1 will describe the derivation process of the collocated motion vector.

[0090] 2.1.2.5.1 Derivation of Collocated Motion Vectors

[0091] For the collocated block, it can be intra-coded or inter-coded using uni-prediction or bi-prediction. If it is intra-coded, the TMVP candidate is set to unavailable.

[0092] If it is uni-directional prediction from list A, the motion vector of list A is scaled to the target reference picture list X.

[0093] If it is bidirectional prediction and the target reference picture list is X, the motion vector of list A is scaled to the target reference picture list X, and A is determined according to the following rules:

[0094] - If no reference picture has a larger POC value than the current picture, set A to X.

[0095] - Otherwise, set A to collocated_from_l0_flag.

[0096] 2.1.2.6 Additional Candidate Insertion

[0097] In addition to spatial and temporal Merge candidates, there are two additional types of Merge candidates: combined bidirectional prediction Merge candidates and zero Merge candidates. Combined bidirectional prediction Merge candidates are generated by utilizing spatial and temporal Merge candidates. Combined bidirectional prediction Merge candidates are only used for B slices. Combined bidirectional prediction candidates are generated by combining the first reference picture list motion parameters of the original candidate with the second reference picture list motion parameters of another candidate. If the two tuples provide different motion hypotheses, they will form a new bidirectional prediction candidate. For example, Figure 7 It shows the case when two candidates in the original list (left) have mvL0 and refIdxL0 or mvL1 and refIdxL1, which are used to create a combined bi-predictive Merge candidate added to the final list (right). There are many rules about combining, which are considered to generate these additional Merge candidates.

[0098] Zero motion candidates are inserted to fill the remaining entries in the Merge candidate list to reach the MaxNumMergeCand capacity. These candidates have zero spatial displacement and a reference picture index that starts at zero and increases each time a new zero motion candidate is added to the list. Finally, no redundancy check is performed on these candidates.

[0099] 2.1.3AMVP

[0100] AMVP exploits the spatiotemporal correlation of motion vectors with neighboring PUs for explicit transmission of motion parameters. For each reference picture list, a motion vector candidate list is constructed by first checking the availability of the PU positions above and to the left in the temporal domain, removing redundant candidates and adding zero vectors to make the candidate list a constant length. The codec can then select the best predictor from the candidate list and send a corresponding index indicating the selected candidate. Similar to Merge index signaling, a truncated unary is used to encode and decode the index of the best motion vector candidate. The maximum value to be encoded in this case is 2 (see Figure 8 ). In the following sections, details on the derivation process of motion vector prediction candidates are provided.

[0101] 2.1.3.1 Derivation of AMVP Candidates

[0102] Figure 8 The process of deriving motion vector prediction candidates is summarized.

[0103] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For spatial motion vector candidate derivation, based on the Figure 2The motion vectors of each PU at the five different positions shown in ultimately derive two motion vector candidates.

[0104] For temporal motion vector candidate derivation, a motion vector candidate is selected from two candidates, which are derived based on two different collocated positions. After generating the first list of spatio-temporal candidates, duplicate motion vector candidates in the list are removed. If the number of potential candidates is greater than 2, motion vector candidates whose reference picture index within the associated reference picture list is greater than 1 are removed from the list. If the number of spatio-temporal motion vector candidates is less than 2, an additional zero motion vector candidate is added to the list.

[0105] 2.1.3.2 Spatial Motion Vector Candidates

[0106] In the derivation of spatial motion vector candidates, at most two candidates are considered among five potential candidates, which are located at Figure 2 The derivation order of the left side of the current PU is defined as A0, A1, and scaled A0, scaled A1. The derivation order of the upper side of the current PU is defined as B0, B1, B2, scaled B0, scaled B1, scaled B2. Therefore, for each side, there are four cases that can be used as motion vector candidates, two of which do not require spatial scaling and two use spatial scaling. The four different cases are summarized as follows.

[0107] No airspace scaling

[0108] -(1) Same reference picture list and same reference picture index (same POC)

[0109] -(2) Different reference picture lists, but same reference pictures (same POC)

[0110] Airspace scaling

[0111] -(3) Same reference picture list, but different reference pictures (different POC)

[0112] -(4) Different reference picture lists, and different reference pictures (different POC)

[0113] First check the case without spatial scaling, then check spatial scaling. Consider spatial scaling when the POC is different between the reference picture of the neighboring PU and the reference picture of the current PU regardless of the reference picture list. If all PUs of the left candidate are unavailable or intra-coded, then scaling of the above motion vectors is allowed to help parallel derivation of the left and above MV candidates. Otherwise, spatial scaling of the above motion vectors is not allowed.

[0114] In the spatial domain scaling process, the motion vectors of neighboring PUs are scaled in a similar way to the temporal domain scaling. Fig. 9 As shown, the main difference is that the reference picture list and the index of the current PU are given as input; the actual scaling process is the same as the time domain scaling process.

[0115] 2.1.3.3 Temporal Motion Vector Candidates

[0116] Except for the reference picture index derivation, all the processes used to derive the temporal merge candidate are the same as those used to derive the spatial motion vector candidate (see Figure 6 ). The reference picture index is signaled to the decoder.

[0117] 2.2 Inter-frame prediction method in VVC

[0118] There are several new codec tools for inter prediction improvements, such as adaptive motion vector difference resolution (AMVR) for signaling MVD, Merge with Motion Vector Difference (MMVD), triangular prediction mode (TPM), combined intra-inter prediction (CIIP), advanced TMVP (ATMVP, also known as SbTMVP), affine prediction mode, generalized bidirectional prediction (GBI), decoder-side motion vector refinement (DMVR) and bidirectional optical flow (BIO, also known as BDOF).

[0119] VVC supports three different Merge list construction processes:

[0120] 1) Sub-block Merge candidate list : It includes ATMVP and affine Merge candidates. Affine mode and ATMVP mode share a Merge list construction process. Here, ATMVP and affine Merge candidates can be added in sequence. The sub-block Merge list size is signaled in the slice header, and the maximum value is 5.

[0121] 2) Regular Merge List : For inter-frame coded blocks, a shared Merge list construction process is used. Here, spatial / temporal Merge candidates, HMVP, paired Merge candidates, and zero motion candidates can be inserted in order. The regular Merge list size is signaled in the slice header, and the maximum value is 6. MMVD, TPM, and CIIP rely on the regular Merge list.

[0122] 3) IBC Merge List : It is implemented similarly to the regular Merge list.

[0123] Similarly, VVC supports three AMVP lists:

[0124] 1) Affine AMVP candidate list,

[0125] 2) Regular AMVP candidate list,

[0126] 3) IBC AMVP candidate list: Due to the adoption of JVET-N0843, the construction process is the same as the IBC Merge list.

[0127] 2.2.1 Codec Block Structure in VVC

[0128] In VVC, a quadtree / binary tree / multitree (QT / BT / TT) structure is used to divide an image into square or rectangular blocks.

[0129] In addition to QT / BT / TT, VVC also uses a split tree (also known as a dual codec tree) for I frames. Using the split tree, the codec block structure is signaled separately for the luminance and chrominance components.

[0130] In addition, except for blocks encoded using some specific coding methods (for example, intra-frame sub-partition prediction where PU is equal to TU but smaller than CU, and sub-block transform of inter-frame coding blocks where PU is equal to CU but TU is smaller than PU), CU is set to PU and TU.

[0131] 2.2.2 Affine Prediction Model

[0132] In HEVC, motion compensated prediction (MCP) only applies the translation motion model. However, in the real world, there may be a variety of motions, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, the simplified affine transformation motion compensated prediction is applied to the 4-parameter affine model and the 6-parameter affine model. Figures 10A-10B As shown, the affine motion field of a block is described by two control point motion vectors (CPMVs) for a 4-parameter affine model and by 3 CPMVs for a 6-parameter affine model.

[0133] The motion vector field (MVF) of a block is described by the following equation with the 4-parameter affine model in equation (1) (where the 4 parameters are defined as variables a, b, e and f) and the 6-parameter affine model in equation (2) (where the 6 parameters are defined as variables a, b, c, d, e and f):

[0134]

[0135]

[0136] Where (mv h 0,mv h 0) is the motion vector of the upper left control point, (mv h 1,mv h 1) is the motion vector of the upper right control point, (mv h2,mv h 2) is the motion vector of the motion vector control point in the lower left corner. All three motion vectors are called control point motion vectors (CPMVs), (x, y) represents the coordinates of the representative point relative to the upper left corner sample point in the current block, and (mv h (x,y),mv v (x,y)) is the motion vector derived for the sample at (x,y). The CP motion vector can be signaled (e.g. in affine AMVP mode) or derived on the fly (e.g. in affine Merge mode). w and h are the width and height of the current block. In practice, division is performed by rounding and right shifting. In VTM, the representative point is defined as the center position of the sub-block, for example, when the coordinates of the upper left corner of the sub-block relative to the upper left sample in the current block are (xs, ys), the coordinates of the representative point are defined as (xs+2, ys+2). For each sub-block (i.e. 4x4 in VTM), the representative point is used to derive the motion vector for the entire sub-block.

[0137] To further simplify motion compensation prediction, sub-block based affine transformation prediction is applied. To derive the motion vector of each M×N (in the current VVC, both M and N are set to 4) sub-block, as Fig.11 As shown, the motion vector of the center sample of each sub-block is calculated according to equations (1) and (2) and rounded to a fractional precision of 1 / 16. Then, a motion compensation interpolation filter suitable for 1 / 16 pixels is applied to generate a prediction for each sub-block with a derived motion vector. The affine mode introduces an interpolation filter of 1 / 16 pixels.

[0138] After MCP, the high-precision motion vector of each sub-block is rounded and saved with the same precision as the regular motion vector.

[0139] 2.2.3 Merge of the entire block

[0140] 2.2.3.1 Translating the Merge List Construction of the Conventional Merge Mode

[0141] 2.2.3.1.1 History-based Motion Vector Prediction (HMVP)

[0142] Different from the Merge list design, in VVC, a history-based motion vector prediction (HMVP) method is adopted.

[0143] In HMVP, the motion information of the previously coded blocks is stored. The motion information of the previously coded blocks is defined as HMVP candidates. Multiple HMVP candidates are stored in a table called HMVP table, and the table is maintained in the real-time encoding / decoding process. When encoding / decoding of a new slice / LCU row / slice starts, the HMVP table is cleared. Whenever there is an inter-coded block, non-sub-block, non-TPM mode, the associated motion information is added to the last item of the table as a new HMVP candidate. The overall encoding process is Fig.12 Shown in.

[0144] 2.2.3.1.2 Conventional Merge List Construction Processing

[0145] The construction of a regular Merge list (for translational motion) can be summarized in the following sequence of steps:

[0146] οStep 1: Derive Spatial Candidates

[0147] οStep 2: Insert HMVP candidates

[0148] οStep 3: Insert pairwise average candidates

[0149] οStep 4: Default Movement

[0150] HMVP candidates can be used in the AMVP and Merge candidate list construction processes. Fig.13 A modified Merge candidate list construction process (grey shaded) is shown. When the Merge candidate list is not full after inserting the TMVP candidate, the HMVP candidates stored in the HMVP table can be used to fill the Merge candidate list. Considering that a block usually has a higher correlation with the nearest neighboring block in terms of motion information, the HMVP candidates in the table are inserted in descending order of index. The last entry in the table is added to the list first and the first entry is added last. Similarly, redundancy removal is applied to the HMVP candidates. Once the total number of available Merge candidates reaches the maximum number of Merge candidates allowed to be signaled, the Merge candidate list construction process terminates.

[0151] It should be noted that all spatial / temporal / HMVP candidates should be coded and decoded in non-IBC mode. Otherwise, they are not allowed to be added to the regular Merge candidate list.

[0152] The HMVP table contains up to 5 regular motion candidates, and each candidate is unique.

[0153] 2.2.3.1.2.1 Pruning

[0154] Only when the corresponding candidates for redundancy check do not have the same motion information, the candidate is added to the list. This comparison process is called pruning process.

[0155] The pruning process between candidate spaces depends on the TPM usage of the current block.

[0156] When the current block is encoded and decoded without using the TPM mode (such as regular Merge, MMVD, CIIP), HEVC pruning processing (ie, 5-pruning) is used for the spatial Merge candidate.

[0157] 2.2.3.1.2.2 Update HMVP Table

[0158] After decoding an inter-coded block using non-TPM, non-sub-block mode, the HMVP table is updated. The motion information of the current block can first be pruned to all available HMVP candidates in the HMVP table. If an identical one is found, the redundant one is removed from the table and all the remaining ones are pushed forward by decreasing the index by 1. The motion information of the current block is then added to the HMVP table.

[0159] If the counter denoted by K (recording the number of available HMVP candidates in the table) is less than 5, K is increased by 1. Otherwise, K remains unchanged.

[0160] The relevant specifications are defined as follows:

[0161] 8.5.2.16 Update of the History-Based Motion Vector Predictor Candidate List

[0162] The input to this process is:

[0163] -1 / 16 fractional sample accuracy of luminance motion vectors mvL0 and mvL1,

[0164] - reference indices refIdxL0 and refIdxL1,

[0165] - The prediction list uses flags predFlagL0 and PredFlag1,

[0166] - Bidirectional prediction weight index bcwIdx.

[0167] The MVP candidate hMvpCand consists of luma motion vectors mvL0 and mvL1, reference indexes refIdxL0 and refIdxL1, prediction list usage flags predFlagL0 and predflag1, and bidirectional prediction weight index bcwIdx.

[0168] Use the candidate hMvpCand to modify the candidate list HmvpCandList in the following order:

[0169] 1. Set the variable identicalCandExist equal to FALSE and the variable removeIdx equal to 0.

[0170] 2. When NumHmvpCand is greater than 0, for each index hMvpIdx where hMvpIdx=0..NumHmvpCand-1, apply the following steps until identicalCandExist equals TRUE:

[0171] -When hMvpCand is equal to HmvpCandList[hMvpIdx], set identicalCandExist equal to TRUE and set removeIdx equal to hMvpIdx.

[0172] 3. The candidate list HmvpCandList is updated as follows:

[0173] - If identicalCandExist is equal to TRUE or NumHmvpCand is equal to MaxNumMergeCand-1, the following applies:

[0174] - For each index i where i = (removeIdx+1) ... (NumHmvpCand-1), set HmvpCandList[i-1] equal to HmvpCandList[i].

[0175] -Set HmvpCandList[NumHmvpCand-1] equal to mvCand.

[0176] Otherwise (identicalCandExist is equal to FALSE and NumHmvpCand is less than MaxNumMergeCand-1), the following applies:

[0177] -Set HmvpCandList[NumHmvpCand++] equal to mvCand.

[0178] Here, MaxNumMergeCand is set to 5.

[0179] 2.2.4 Triangular Prediction Model (TPM)

[0180] In VVC, triangular partitioning mode is supported for inter prediction. Triangular partitioning mode is only applicable to CUs that are 8x8 or larger and are encoded and decoded in Merge mode but not in MMVD or CIIP mode. For CUs that meet these conditions, a CU level flag is signaled to indicate whether triangular partitioning mode is applied.

[0181] When using this mode, Fig.14 As shown, the CU is evenly divided into two triangular partitions using diagonal partitioning or anti-diagonal partitioning. Each triangular partition in the CU uses its own motion for inter-frame prediction; each partition is only allowed to perform unidirectional prediction, that is, each partition has one motion vector and one reference index. Unidirectional prediction motion constraints are applied to ensure that only two motion compensated predictions are required for each CU, just like traditional bidirectional prediction.

[0182] If the CU level flag indicates that the current CU is encoded and decoded using the triangle partitioning mode, a flag indicating the direction of the triangle partitioning (diagonal or anti-diagonal) and two Merge indices (one for each partition) will be further signaled. After predicting each triangle partition, a hybrid process with adaptive weights is used to adjust the sample values ​​along the diagonal or anti-diagonal edges. This is the prediction signal for the entire CU, and like other prediction modes, the transform and quantization process will be applied to the entire CU. Finally, the motion field of the CU predicted using the triangle partitioning mode is stored in 4x4 units.

[0183] The regular Merge candidate list is reused for triangular partition Merge prediction without additional motion vector pruning. For each Merge candidate in the regular Merge candidate list, one (and only one) of its L0 or L1 motion vectors is used for triangular prediction. In addition, the selection order of L0 and L1 motion vectors is based on their Merge index parity. With this scheme, the regular Merge list can be used directly.

[0184] 2.2.4.1 TPM Merge List Construction Processing

[0185] Basically, the conventional Merge list construction process is applied as proposed in JVET-N0340. However, some modifications are added.

[0186] Specifically, the following applies:

[0187] 1) How the pruning process is performed depends on the current block's use of the TPM.

[0188] - If the current block does not use TPM codec, call HEVC5 pruning applied to the spatial Merge candidate;

[0189] - Otherwise (if the current block uses TPM codec), full pruning is applied when adding new spatial merge candidates. That is, B1 is compared with A1; B0 is compared with A1 and B1; A0 is compared with A1, B1, and B0; and B2 is compared with A1, B1, A0, and B0.

[0190] 2) The condition of whether to check the motion information from B2 depends on the current block's use of TPM.

[0191] -If the current block does not use TPM encoding and decoding, B2 is accessed and checked only when there are less than 4 spatial merge candidates before checking B2.

[0192] - Otherwise (if the current block uses TPM encoding and decoding), B2 will be accessed and checked before adding B2 regardless of how many available spatial merge candidates there are.

[0193] 2.2.4.2 Adaptive weighted processing

[0194] After predicting each triangular prediction unit, an adaptive weighting process is applied to the diagonal edge between two triangular prediction units to derive the final prediction of the entire CU. Two weighting factor groups are defined as follows:

[0195] o a first set of weighting factors: {7 / 8, 6 / 8, 4 / 8, 2 / 8, 1 / 8} and {7 / 8, 4 / 8, 1 / 8} for luma and chroma samples, respectively;

[0196] o A second set of weighting factors: {7 / 8, 6 / 8, 5 / 8, 4 / 8, 3 / 8, 2 / 8, 1 / 8} and {6 / 8, 4 / 8, 2 / 8} are used for luma and chroma samples respectively.

[0197] The weighting factor set is selected based on the comparison of the motion vectors of the two triangular prediction units. The second weighting factor set is used when any of the following conditions is met:

[0198] - The reference pictures of the two triangular prediction units are different from each other

[0199] -The absolute value of the difference between the horizontal values ​​of two motion vectors is greater than 16 pixels.

[0200] -The absolute value of the difference between the vertical values ​​of two motion vectors is greater than 16 pixels.

[0201] Otherwise, the first set of weighting factors is used. Fig.15 shown.

[0202] 2.2.4.3 Motion Vector Storage

[0203] The motion vector of the triangle prediction unit ( Fig.16 Mv1 and Mv2 in CU are stored in 4×4 grids. For each 4×4 grid, a unidirectional prediction or bidirectional prediction motion vector is stored according to the position of the 4×4 grid in the CU. Fig.16As shown, a unidirectional prediction motion vector Mv1 or Mv2 is stored for a 4×4 grid located in a non-weighted area (i.e. not at a diagonal edge). On the other hand, a bidirectional prediction motion vector is stored for a 4×4 grid located in a weighted area. The bidirectional prediction motion vector is derived from Mv1 and Mv2 according to the following rule:

[0204] 1) In the case where Mv1 and Mv2 have motion vectors from different directions (L0 or L1), Mv1 and Mv2 are simply combined to form a bidirectional prediction motion vector.

[0205] 2) When both Mv1 and Mv2 come from the same L0 (or L1) direction,

[0206] - If the reference picture of Mv2 is the same as a picture in the L1 (or L0) reference picture list, Mv2 is scaled to that picture. Mv1 and the scaled Mv2 are combined to form a bidirectional prediction motion vector.

[0207] - If the reference picture of Mv1 is the same as a picture in the L1 (or L0) reference picture list, then Mv1 is scaled to that picture. The scaled Mv1 and Mv2 are combined to form a bidirectional prediction motion vector.

[0208] - Otherwise, only Mv1 is stored for the weighted region.

[0209] 2.2.4.4Merge mode syntax, semantics, and decoding processing

[0210] 7.3.5.1 Generic Strip Header Syntax

[0211]

[0212]

[0213] 7.3.7.5 Codec unit syntax

[0214]

[0215]

[0216] 7.3.7.7 Merge Data Syntax

[0217]

[0218]

[0219] 7.4.6.1 Generic Strip Header Semantics

[0220] six_minus_max_num_merge_cand specifies the maximum number of Merge motion vector prediction (MVP) candidates supported in a slice minus 6. The maximum number of Merge MVP candidates MaxNumMergeCand is derived as follows:

[0221] MaxNumMergeCand=6-six_minus_max_num_merge_cand (7-57)

[0222] The value of MaxNumMergeCand should be between 1 and 6, inclusive.

[0223] five_minus_max_num_subblock_merge_cand specifies the maximum number of sub-block based Merge motion vector prediction (MVP) candidates supported in a slice minus 5. When five_minus_max_num_subblock_merge_cand is not present, it is inferred to be equal to 5-sps_sbtmvp_enabled_flag. The maximum number of sub-block based Merge MVP candidates MaxNumSubblockMergeCand is derived as follows:

[0224] MaxNumSubblockMergeCand=5-five_minus_max_num_subblock_merge_cand(7-58)

[0225] The value of MaxNumSubblockMergeCand should be between 0 and 5 (inclusive).

[0226] 7.4.8.5 Codec unit semantics

[0227] pred_mode_flag equal to 0 specifies that the current codec unit is coded in inter-frame prediction mode. pred_mode_flag equal to 1 specifies that the current codec unit is coded in intra-frame prediction mode.

[0228] When pred_mode_flag is not present, it is inferred as follows:

[0229] - If cbWidth is equal to 4 and cbHeight is equal to 4, pred_mode_flag will be inferred to be equal to 1.

[0230] - Otherwise, pred_mode_flag is inferred to be equal to 1 when decoding an I slice and equal to 0 when decoding a P or B slice, respectively.

[0231] For x = x0..x0+cbWidth-1 and y = y0..y0+cbHeight-1, the variable CuPredMode[x][y] is derived as follows:

[0232] - If pred_mode_flag is equal to 0, set CuPredMode[x][y] equal to mode_INTER.

[0233] - Otherwise (pred_mode_flag is equal to 1), set CuPredMode[x][y] equal to mode_INTRA.

[0234] pred_mode_ibc_flag equal to 1 specifies that the current codec unit is coded in IBC prediction mode. pred_mode_ibc_flag equal to 0 specifies that the current codec unit is not coded in IBC prediction mode.

[0235] When pred_mode_ibc_flag is not present, it is inferred as follows:

[0236] - If cu_skip_flag[x0][y0] is equal to 1, and cbWidth is equal to 4, and cbHeight is equal to 4, then pred_mode_ibc_flag is inferred to be equal to 1.

[0237] - Otherwise, if cbWidth and cbHeight are both equal to 128, pred_mode_ibc_flag is inferred to be equal to 0.

[0238] - Otherwise, pred_mode_ibc_flag is inferred to be equal to the value of sps_ibc_enabled_flag when decoding an I slice and to be equal to 0 when decoding a P or B slice, respectively.

[0239] When pred_mode_ibc_flag is equal to 1, the variable CuPredMode[x][y] is set equal to MODE_IBC with x=x0..x0+cbWidth-1 and y=y0..y0+cbHeight-1.

[0240] general_merge_flag[x0][y0] specifies whether the inter prediction parameters of the current codec unit are inferred from neighboring inter prediction partitions. The array index x0, y0 specifies the position (x0, y0) of the top left luma sample of the considered codec block relative to the top left luma sample of the picture.

[0241] When general_merge_flag[x0][y0] is not present, it is inferred as follows:

[0242] - If cu_skip_flag[x0][y0] is equal to 1, general_merge_flag[x0][y0] is inferred to be equal to 1.

[0243] Otherwise, general_merge_flag[x0][y0] is inferred to be equal to 0.

[0244] mvp_l0_flag[x0][y0] specifies the motion vector predictor index for list 0, where x0, y0 specify the position (x0, y0) of the top-left luma sample of the considered codec block relative to the top-left luma sample of the picture.

[0245] When mvp_l0_flag[x0][y0] is not present, it is inferred to be equal to 0.

[0246] The semantics of mvp_l1_flag[x0][y0] are the same as mvp_l0_flag, with l0 and list 0 replaced by l1 and list 1 respectively.

[0247] According to Tables 7-10, inter_pred_idc[x0][y0] specifies whether the current codec uses list 0, list 1, or bidirectional prediction. The array index x0, y0 specifies the position (x0, y0) of the top left luma sample of the considered codec block relative to the top left luma sample of the picture.

[0248] Table 7-10 Association between names and inter-frame prediction modes

[0249]

[0250] When inter_pred_idc[x0][y0] does not exist, it is inferred to be equal to PRED_L0.

[0251] 7.4.8.7Merge Data Semantics

[0252] regular_merge_flag[x0][y0] equal to 1 specifies that the regular Merge mode is used to generate the inter prediction parameters of the current codec unit. The array index x0, y0 specifies the position (x0, y0) of the upper left luma sample of the considered codec block relative to the upper left luma sample of the picture.

[0253] When the rule regular_merge_flag[x0][y0] does not exist, it is inferred as follows:

[0254] - regular_merge_flag[x0][y0] is inferred to be equal to 1 if all of the following are true:

[0255] -sps_mmvd_enabled_flag is equal to 0.

[0256] -general_merge_flag[x0][y0] is equal to 1.

[0257] -cbWidth*cbHeight is equal to 32.

[0258] Otherwise, regular_merge_flag[x0][y0] is inferred to be equal to 0.

[0259] mmvd_merge_flag[x0][y0] equal to 1 specifies the use of motion vector difference Merge mode to generate inter prediction parameters for the current codec unit. Array index x0, y0 specifies the position (x0, y0) of the top left luma sample of the considered codec block relative to the top left luma sample of the picture.

[0260] When mmvd_merge_flag[x0][y0] is not present, it is inferred as follows:

[0261] - mmvd_merge_flag[x0][y0] is inferred to be equal to 1 if all of the following are true:

[0262] -sps_mmvd_enabled_flag is equal to 1.

[0263] -general_merge_flag[x0][y0] is equal to 1.

[0264] -cbWidth*cbHeight is equal to 32.

[0265] -regular_merge_flag[x0][y0] is equal to 0.

[0266] Otherwise, mmvd_merge_flag[x0][y0] is inferred to be equal to 0.

[0267] mmvd_cand_flag[x0][y0] specifies whether the first (0) or second (1) candidate in the Merge candidate list is used with the motion vector difference derived from mmvd_distance_idx[x0][y0] and mmvd_direction_idx[x0][y0]. The array index x0, y0 specifies the position (x0, y0) of the top-left luma sample of the considered codec block relative to the top-left luma sample of the picture.

[0268] When mmvd_cand_flag[x0][y0] is not present, it is inferred to be equal to 0.

[0269] mmvd_distance_idx[x0][y0] specifies the index used to derive MmvdDistance[x0][y0] as specified in Table 7-12. The array index x0, y0 specifies the position (x0, y0) of the top-left luma sample of the considered codec block relative to the top-left luma sample of the picture.

[0270] Table 7-12 Specification of MmvdDistance[x0][y0] based on mmvd_distance_idx[x0][y0]

[0271]

[0272] mmvd_direction_idx[x0][y0] specifies the index used to derive MmvdSign[x0][y0] as specified in Table 7-13. The array index x0, y0 specifies the position (x0, y0) of the top-left luma sample of the considered codec block relative to the top-left luma sample of the picture.

[0273] Table 7-13 Specification of MmvdSign[x0][y0] based on mmvd_direction_idx[x0][y0]

[0274] mmvd_direction_idx[x0][y0] MmvdSign[x0][y0][0] MmvdSign[x0][y0][1] 0 +1 0 1 -1 0 2 0 +1 3 0 -1

[0275] The two components of Merge plus MVD offset MmvdOffset[x0][y0] are derived as follows:

[0276] MmvdOffset[x0][y0][0]=(MmvdDistance[x0][y0]<<2)*MmvdSign[x0][y0

[0277] ][0](7-124)

[0278] MmvdOffset[x0][y0][1]=(MmvdDistance[x0][y0]<<2)*MmvdSign[x0][y0

[0279] ][1](7-125)

[0280] merge_subblock_flag[x0][y0] specifies whether to infer subblock-based inter prediction parameters for the current codec unit from neighboring blocks. Array index x0, y0 specifies the position (x0, y0) of the top-left luma sample of the considered codec block relative to the top-left luma sample of the picture. When merge_subblock_flag[x0][y0] is not present, it is inferred to be equal to 0.

[0281] merge_subblock_idx[x0][y0] specifies the merge candidate index of the sub-block based merge candidate list, where x0, y0 specify the position (x0, y0) of the upper left luma sample of the considered codec block relative to the upper left luma sample of the picture.

[0282] When merge_subblock_idx[x0][y0] does not exist, it is inferred to be equal to 0.

[0283] ciip_flag[x0][y0] specifies whether combined inter-frame Merge and intra-frame prediction is applied to the current codec unit. The array index x0, y0 specifies the position (x0, y0) of the upper left luma sample of the considered codec block relative to the upper left luma sample of the picture.

[0284] When ciip_flag[x0][y0] is not present, it is inferred to be equal to 0.

[0285] When ciip_flag[x0][y0] is equal to 1, the variable IntraPredModeY[x][y] (x=xCb..xCb+cbWidth-1 and y=yCb..yCb+cbHeight-1) is set equal to INTRA_PLANAR.

[0286] The variable MergeTriangleFlag[x0][y0] specifies whether to use triangle-based motion compensation to generate prediction samples for the current codec unit when decoding B slices. It is derived as follows:

[0287] - Set MergeTriangleFlag[x0][y0] equal to 1 if all of the following are true:

[0288] -sps_triangle_enabled_flag is equal to 1.

[0289] -slice_type is equal to B.

[0290] -general_merge_flag[x0][y0] is equal to 1.

[0291] -MaxNumTriangleMergeCand is greater than or equal to 2.

[0292] -cbWidth*cbHeight is greater than or equal to 64.

[0293] -regular_merge_flag[x0][y0] is equal to 0.

[0294] -mmvd_merge_flag[x0][y0] is equal to 0.

[0295] -merge_sub block_flag[x0][y0] is equal to 0.

[0296] -ciip_flag[x0][y0] is equal to 0.

[0297] Otherwise, set MergeTriangleFlag[x0][y0] equal to 0.

[0298] merge_triangle_split_dir[x0][y0] specifies the split direction of the Merge triangle mode. The array index x0, y0 specifies the position (x0, y0) of the upper left luma sample of the considered codec block relative to the upper left luma sample of the picture.

[0299] When merge_triangle_split_dir[x0][y0] does not exist, it is inferred to be equal to 0.

[0300] merge_triangle_idx0[x0][y0] specifies the first Merge candidate index of the triangle-based motion compensation candidate list, where x0, y0 specify the position (x0, y0) of the upper left luma sample of the considered codec block relative to the upper left luma sample of the picture.

[0301] When merge_triangle_idx0[x0][y0] does not exist, it is inferred to be equal to 0.

[0302] merge_triangle_idx1[x0][y0] specifies the second Merge candidate index of the triangle-based motion compensation candidate list, where x0, y0 specify the position (x0, y0) of the upper left luma sample of the considered codec block relative to the upper left luma sample of the picture.

[0303] When merge_triangle_idx1[x0][y0] does not exist, it is inferred to be equal to 0.

[0304] merge_idx[x0][y0] specifies the merge candidate index of the merge candidate list, where x0, y0 specify the position (x0, y0) of the upper left luma sample of the considered codec block relative to the upper left luma sample of the picture.

[0305] When merge_idx[x0][y0] does not exist, it is inferred as follows:

[0306] - If mmvd_merge_flag[x0][y0] is equal to 1, then merge_idx[x0][y0] is inferred to be equal to mmvd_cand_flag[x0][y0].

[0307] Otherwise (mmvd_merge_flag[x0][y0] is equal to 0), merge_idx[x0][y0] is inferred to be equal to 0.

[0308] 2.2.4.4.1 Decoding Process

[0309] The decoding process provided in JVET-N0340 is defined as follows:

[0310] 8.5.2.2 Derivation of Luminance Motion Vectors in Merge Mode

[0311] This process is called only when general_merge_flag[xCb][yCb] is equal to 1, where (xCb, yCb) specifies the upper left sample of the current luma codec block relative to the upper left luma sample of the current picture.

[0312] The input to this process is:

[0313] - The luminance position (xCb, yCb) of the upper left sample of the current luminance codec block relative to the upper left luminance sample of the current picture,

[0314] - variable cbWidth, specifies the width of the current codec block in luminance samples,

[0315] - The variable cbHeight specifies the height of the current codec block in luma samples.

[0316] The output of this processing is:

[0317] -1 / 16 fractional sample precision luminance motion vectors mvL0[0][0] and mvL1[0][0],

[0318] - reference indices refIdxL0 and refIdxL1,

[0319] - The prediction list uses flags predFlagL0[0][0] and PredFlagL1[0][0],

[0320] - bidirectional prediction weight index bcwIdx,

[0321] -Merge candidate list mergeCandList.

[0322] Set the bidirectional prediction weight index bcwIdx to 0.

[0323] The motion vectors mvL0[0][0] and mvL1[0][0], the reference indices refIdxL0 and refIdxL1, and the prediction use flags predFlagL0[0][0] and predFlagL1[0][0] are derived by the following sequential steps:

[0324] 1. With the luma codec block position (xCb, yCb), luma codec block width cbWidth and luma codec block height cbHeight as input, call the process of deriving spatial Merge candidates from adjacent codec units specified in Section 8.5.2.3, and output the availability flags availableFlagA0, availableFlagA1, availableFlagB0, availableFlagB1 and availableFlagB2, reference index refIdxLXA0, refIdxLXA1 ,refIdxLXB0,refIdxLXB1, and refIdxLXB2, prediction lists use flags predFlagLXA0, predFlagLXA1, predFlagLXB0, predFlagLXB1, and predFlagLXB2, and motion examples mvLXA0, mvLXA1, mvLXB0, mvLXB1, and mvLXB2 (where X is 0 or 1), and bidirectional prediction weight indices bcwIdxA0, bcwIdxA1, bcwIdxB0, bcwIdxB1, bcwIdxB2.

[0325] 2. Set the reference index refIdxLXCol (X is 0 or 1) and the bidirectional prediction weight index bcwIdxCol of the time domain Merge candidate Col to be equal to 0.

[0326] 3. With the luma position (xCb, yCb), luma codec block width cbWidth, luma codec block height cbHeight and variable refIdxL0Col as input, call the derivation process of temporal luma motion vector prediction specified in clause 8.5.2.11, and output the availability flag availableFlagL0Col and the temporal motion vector mvL0Col. The variables availableFlagCol, predFlagL0Col and PredFlag1Col are derived as follows:

[0327] availableFlagCol=availableFlagL0Col (8-263)

[0328] predFlagL0Col=availableFlagL0Col (8-264)

[0329] predFlagL1Col=0 (8-265)

[0330] 4. When slice_type is equal to B, the derivation process of temporal luma motion vector prediction specified in clause 8.5.2.11 is called with luma position (xCb, yCb), luma codec block width cbWidth, luma codec block height cbHeight and variable refIdxL1Col as input, and the output is the availability flag availableFlagL1Col and the temporal motion vector mvL1Col. The derivation of variables availableFlagCol and predFlagL1Col is as follows:

[0331] availableFlagCol=availableFlagL0Col||availableFlagL1Col (8-266)

[0332] predFlagL1Col=availableFlagL1Col (8-267)

[0333] 5. The Merge candidate list mergeCandList is constructed as follows:

[0334]

[0335]

[0336] 6. Set the variables numCurrMergeCand and numOrigMergeCand equal to the number of Merge candidates in mergeCandList.

[0337] 7. When numCurrMergeCand is less than (MaxNumMergeCand-1) and NumHmvpCand is greater than 0, the following applies:

[0338] - With mergeCandList and numCurrMergeCand as input, invoke the history-based derivation process for Merge candidates specified in 8.5.2.6 and produce the modified mergeCandList and numCurrMergeCand as output.

[0339] - Set numOrigMergeCand equal to numCurrMergeCand.

[0340] 8. When numCurrMergeCand is less than MaxNumMergeCand and greater than 1, the following applies:

[0341] - Invoke the derivation of pairwise average Merge candidates specified in clause 8.5.2.4 with as input mergeCandList, reference indices refIdxL0N and refIdxL1N, prediction list usage flags predFlagL0N and predFlagL1N, motion vectors mvL0N and mvL1N for each candidate N in mergeCandList, and numCurrMergeCand, and assign the output to mergeCandList, numCurrMergeCand, reference indices refIdxL0avgCand and refIdxL1avgCand, prediction list usage flags predFlagL0avgCand and predFlagL1avgCand, and motion vectors mvL0avgCand and mvL1avgCand of the candidate avgCand added to mergeCandList. Set the bidirectional prediction weight index bcwIdx of the candidate avgCand added to mergeCandList to be equal to 0.

[0342] - Set numOrigMergeCand equal to numCurrMergeCand.

[0343] 9. Invoke the derivation process for zero motion vector Merge candidates specified in clause 8.5.2.5 with mergeCandList, reference indices refIdxL0N and refIdxL1N, prediction list usage flags predFlagL0N and predFlagL1N, motion vectors mvL0N and mvL1N for each candidate N in mergeCandList, and numCurrMergeCand as input, and assign the output to mergeCandList, numCurrMergeCand, and each new candidate zeroCand added to mergeCandList. m The reference index refIdxL0zeroCand m and refIdxL1zeroCand m , prediction list uses flag predFlagL0zeroCand m and predFlagL1zeroCand m and the motion vector mvL0zeroCand m and mvL1zeroCand m Each new candidate zeroCand added to mergeCandList m The bidirectional prediction weight index bcwIdx is set equal to 0. The number of added candidates numZeroMergeCand is set equal to (numCurrMergeCand-numOrigMergeCand). When numZeroMergeCand is greater than 0, m ranges from 0 to numZeroMergeCand-1, including 0 and numZeroMergeCand-1.

[0344] 10. Perform the following assignment, where N is the candidate at position merge_idx[xCb][yCb] in the Merge candidate list mergeCandList (N=mergeCandList[merge_idx[xCb][yCb]]), and X is replaced by 0 or 1:

[0345] refIdxLX=refIdxLXN (8-269)

[0346] predFlagLX[0][0]=predFlagLXN (8-270)

[0347] mvLX[0][0][0]=mvLXN[0] (8-271)

[0348] mvLX[0][0][1]=mvLXN[1] (8-272)

[0349] bcwIdx=bcwIdxN (8-273)

[0350] 11. When mmvd_merge_flag[xCb][yCb] is equal to 1, the following applies:

[0351] -With the luma position (xCb, yCb), reference indices refIdxL0, refIdxL1, and prediction list usage flags predFlagL0[0][0] and predFlagL1[0][0] as input, call the Merge motion vector difference derivation process specified in 8.5.2.7, and output the motion vector mMvdL0 and mMvdL1.

[0352] - Add the motion vector difference mMvdLX to the Merge motion vector mvLX (X is 0 and 1) as follows:

[0353] mvLX[0][0][0]+=mMvdLX[0] (8-274)

[0354] mvLX[0][0][1]+=mMvdLX[1] (8-275)

[0355] mvLX[0][0][0]=Clip3(-2 17 ,2 17 -1,mvLX[0][0][0]) (8-276)

[0356] mvLX[0][0]1]=Clip3(-2 17 ,2 17 -1,mvLX[0][0][1]) (8-277)

[0357] 2.2.5 MMVD

[0358] In JVET-L0054, the final motion vector representation (UMVE, also called MMVD) is proposed. UMVE and the proposed motion vector representation method are used in Skip or Merge mode.

[0359] UMVE reuses the same Merge candidates as those contained in the regular Merge candidate list in VVC. Among the Merge candidates, a base candidate can be selected and further extended by the proposed motion vector expression method.

[0360] UMVE provides a new method to represent motion vector difference (MVD), in which the starting point, motion amplitude and motion direction are used to represent the MVD.

[0361] Fig.17 An example of a UMVE search process is shown.

[0362] Fig.18 An example of a UMVE search point is shown.

[0363] The proposed technique uses the Merge candidate list as is. However, only candidates of the default Merge type (MRG_TYPE_DEFAULT_N) are considered for UMVE extension.

[0364] The base candidate index defines the starting point. The base candidate index indicates the best candidate among the candidates in the list, as shown below.

[0365] Table 1 Basic candidate IDX

[0366]

[0367] If the number of base candidates is equal to 1, the base candidate IDX is not signaled.

[0368] The distance index is the motion amplitude information. The distance index indicates the predefined distance from the start point information. The predefined distances are as follows:

[0369] Table 2 Distance IDX

[0370]

[0371] The direction index indicates the direction of the MVD relative to the starting point. The direction index can represent four directions as shown below.

[0372] Table 3 Direction IDX

[0373]

[0374]

[0375] The UMVE flag is signaled immediately after the Skip flag and the Merge flag are sent. If the Skip and Merge flags are true, then the UMVE flag is parsed. If the UMVE flag is equal to 1, then the UMVE syntax is parsed. However, if not 1, then the Affine flag is parsed. If the Affine flag is equal to 1, then it is Affine mode, however, if not 1, then the Skip / Merge index of the Skip / Merge mode of the VTM is parsed.

[0376] Additional line buffers due to UMVE candidates are not needed, since software's Skip / Merge candidates are used directly as base candidates. By using the input UMVE index, the complement of the MV is determined before motion compensation. No long line buffers need to be kept for this.

[0377] Under the current common test conditions, the first or second Merge candidate in the Merge candidate list can be selected as the basic candidate.

[0378] UMVE is also called Merge with MV Difference (MMVD).

[0379] 2.2.6 Combined Intra-Inter Prediction (CIIP)

[0380] In JVET-L0100, multi-hypothesis prediction is proposed, in which combined intra-frame and inter-frame prediction is a method to generate multiple hypotheses.

[0381] When multi-hypothesis prediction is applied to improved intra mode, multi-hypothesis prediction combines an intra prediction and a merge index prediction. In the Merge CU, a flag is signaled for the Merge mode, and when the flag is true, the intra mode is selected from the intra candidate list. For the luma component, the intra candidate list is derived from only one intra prediction mode (i.e., planar mode). The weights applied to the prediction block from intra and inter prediction are determined by the codec mode (intra or non-intra) of the two neighboring blocks (A1 and B1).

[0382] 2.2.7 Merge based on sub-block technology

[0383] It is proposed that in addition to the regular Merge list of non-sub-block Merge candidates, all candidate motions related to sub-blocks are put into a separate Merge list.

[0384] The candidate motions associated with the sub-blocks are put into a separate Merge list, called the sub-block Merge candidate list.

[0385] In one example, the sub-block Merge candidate list includes ATMVP candidates and affine Merge candidates.

[0386] The sub-block Merge candidate list is filled with candidates in the following order:

[0387] a) ATMVP candidates (may be available or not);

[0388] b) Affine Merge list (including inherited affine candidates and constructed affine candidates)

[0389] c) 4-parameter affine model with zero MV filled

[0390] 2.2.7.1 ATMVP (also known as sub-block temporal motion vector predictor, SbTMVP)

[0391] The basic idea of ​​ATMVP is to derive multiple sets of temporal motion vector predictors for a block. Each sub-block is assigned a set of motion information. When generating ATMVP Merge candidates, motion compensation is performed at the 8x8 level rather than the entire block level.

[0392] In the current design, ATMVP predicts the motion vector of a sub-CU within a CU in two steps, which are described in the following two subsections respectively.

[0393] 2.2.7.1.1 Derivation of Initialized Motion Vector

[0394] Denote the initialization motion vector by tempMv. When block A1 is available and non-intra coded (ie coded in inter or IBC mode), the following steps are applied to derive the initialization motion vector.

[0395] - Set tempMv equal to the motion vector of block A1 from List 1, denoted by mvL1A1, if all of the following conditions are met:

[0396] - the reference picture index of list 1 is available (not equal to -1) and has the same POC value as the collocated picture (i.e. DiffPicOrderCnt(ColPic, RefPicList[1][refIdxL1A1]) is equal to 0),

[0397] - all reference pictures have no greater POC than the current picture (i.e., DiffPicOrderCnt(aPic, currPic) for each picture aPic in each reference picture list of the current slice is less than or equal to 0),

[0398] - Current stripe is equal to B stripe,

[0399] -collocated_from_l0_flag is equal to 0.

[0400] - Otherwise, if all of the following are true, then tempMv is set equal to the motion vector of block A1 from list 0, denoted by mvL0A1:

[0401] - the reference picture index of list 0 is available (not equal to -1),

[0402] - It has the same POC value as the collocated picture (ie DiffPicOrderCnt(ColPic, RefPicList[0][refIdxL0A1]) is equal to 0).

[0403] - Otherwise, use zero motion vector as initialization MV.

[0404] The initialization motion vector is used to identify the corresponding block in the collocated picture signaled in the slice header (the center position of the current block plus the rounded MV, clipped to be within a certain range if necessary).

[0405] If the block is inter-coded, go to step 2. Otherwise, the ATMVP candidate is set to unavailable.

[0406] 2.2.7.1.2 Sub-CU motion derivation

[0407] The second step is to divide the current CU into sub-CUs and obtain the motion information of each sub-CU from the block corresponding to each sub-CU in the collocated picture.

[0408] If the corresponding block of the sub-CU is coded in inter-frame mode, the motion information is used to derive the final motion information of the current sub-CU by calling the derivation process of the collocated MV that is the same as the traditional TMVP process. Basically, if the corresponding block is predicted from the target list X for unidirectional prediction or bidirectional prediction, the motion vector is used; otherwise, if it is predicted from the list Y (Y=1-X) for unidirectional prediction or bidirectional prediction and NoBackwardPredFlag is equal to 1, the MV of list Y is used. Otherwise, no motion candidate is found.

[0409] If the block in the collocated picture identified by the initialization MV and the position of the current sub-CU is intra-IBC coded, or no motion candidate is found as described above, the following further applies:

[0410] Denote the motion vector used to obtain the motion field in the collocated picture Rcol as MVcol. To minimize the impact of MV scaling, the MVs in the spatial candidate list for deriving MVcol are selected as follows: If the reference picture of the candidate MV is the collocated picture, then this MV is selected and used as MVcol without any scaling. Otherwise, the MV with the closest reference picture to the collocated picture is selected to derive MVcol using scaling.

[0411] 2.2.8 Refinement of motion information

[0412] 2.2.8.1 Decoder-side Motion Vector Refinement (DMVR)

[0413] In the bidirectional prediction operation, for the prediction of a block area, two prediction slices formed using the motion vectors (MVs) of list 0 and the MVs of list 1 are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of the bidirectional prediction are further refined.

[0414] For DMVR in VVC, assume that the MVD mirroring between list 0 and list 1 is as follows Fig.19 As shown, bilateral matching is performed to refine the MV, that is, to find the best MVD among multiple MVD candidates. The MVs of the two reference picture lists are represented by MVL0 (L0X, L0Y) and MVL1 (L1X, L1Y). The MVD represented by (MvdX, MvdY) of list 0 that can minimize the cost function (such as SAD) is defined as the best MVD. For the SAD function, it is defined as the SAD between the reference block of list 0 derived by the motion vector (L0X+MvdX, L0Y+MvdY) in the list 0 reference picture and the reference block of list 1 derived by the motion vector (L1X-MvdX, L1Y-MvdY) in the list 1 reference picture.

[0415] The motion vector refinement process can be iterated twice. In each iteration, up to 6 MVDs (with integer pel precision) can be checked in two steps, such as Fig. 20 As shown. In the first step, MVD (0, 0), (-1, 0), (1, 0), (0, -1), (0, 1) are checked. In the second step, one of MVD (-1, -1), (-1, 1), (1, -1) or (1, 1) can be selected and further checked. Assume that the function Sad(x, y) returns the SAD (sum of absolute differences) value of MVD (x, y). The MVD denoted (MvdX, MvdY) checked in the second step is determined as follows:

[0416] MvdX = -1;

[0417] MvdY=-1;

[0418] If (Sad(1,0) <Sad(-1,0))

[0419] MvdX = 1;

[0420] If (Sad(0,1) <Sad(0,-1))

[0421] MvdY=1;

[0422] In the first iteration, the starting point is the signaled MV, and in the second iteration, the starting point is the signaled MV plus the best MVD selected in the first iteration. DMVR is applicable only when one reference picture is the previous picture and the other reference picture is the subsequent picture, and both reference pictures are at the same picture order count distance from the current picture.

[0423] In order to further simplify the DMVR processing, JVET-M0147 proposed several modifications to the JEM design. More specifically, the DMVR design adopted by VTM-4.0 (to be released) has the following main features:

[0424] ο Terminate early when the SAD at position (0,0) between list 0 and list 1 is less than a threshold.

[0425] o Terminate early when the SAD is zero somewhere between list 0 and list 1.

[0426] o Block size of DMVR: W*H>=64&&H>=8, where W and H are the width and height of the block.

[0427] For DMVR with CU size > 16*16, the CU is divided into multiple 16x16 sub-blocks. If only the width or height of the CU is greater than 16, it is only divided in the vertical or horizontal direction.

[0428] o Reference block size (W+7)*(H+7) (for luma).

[0429] ο25-point SAD-based integer pixel search (i.e. (+-)2 refinement search range, single level)

[0430] ο DMVR based on bilinear interpolation.

[0431] o Sub-pixel refinement based on the “parametric error surface equation”. This process is performed only when the minimum SAD cost is not equal to zero and the best MVD in the previous MV refinement iteration is (0, 0).

[0432] o Luma / chroma MC filled with reference blocks (if needed).

[0433] ο Thinned MV is only used for MC and TMVP.

[0434] 2.2.8.1.1 Use of DMVR

[0435] DMVR can be enabled when all of the following conditions are true:

[0436] - The DMVR enabled flag in SPS (i.e. sps_dmvr_enabled_flag) is equal to 1,

[0437] -TPM flag, inter-frame affine flag, sub-block Merge flag (ATMVP or affine Merge), and MMVD flag are all equal to 0,

[0438] -Merge flag equals 1,

[0439] - the current block is bi-predicted and the POC distance between the current picture and the reference picture in list 1 is equal to the POC distance between the reference picture and the current picture in list 0,

[0440] -The current CU height is greater than or equal to 8,

[0441] - The number of luma samples (CU width * height) is greater than or equal to 64.

[0442] 2.2.8.1.2 Sub-pixel refinement based on “parametric error surface equation”

[0443] The method is summarized as follows:

[0444] 1) Parameter error surface fits are only calculated when the center position is the best cost position for a given iteration.

[0445] 2) Fit a 2D parabolic error surface equation of this form using the center position cost and the costs at the positions from the center (-1,0), (0,-1), (1,0), and (0,1):

[0446] E(x,y)=A(x-x0) 2 +B(y-y0) 2 +C

[0447] Where (x0, y0) corresponds to the location with the lowest cost, and C corresponds to the minimum cost value. By solving the 5 equations for the 5 unknowns, (x0, y0) is calculated as follows:

[0448] x0=(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0)))

[0449] y0=(E(0,-1)-E(0,1)) / (2((E(0,-1)+E(0,1)-2E(0,0)))

[0450] (x0, y0) can be computed to any desired sub-pixel precision by adjusting the precision with which the division is performed (i.e., the number of bits to compute the quotient). For 1 / 16 pixel precision, only 4 bits of the absolute value of the quotient need to be computed, which facilitates the implementation of the 2 required divisions per CU based on fast shift-subtract.

[0451] 3) The calculated (x0, y0) is added to the integer distance refinement MV to obtain the sub-pixel accurate refinement increment MV.

[0452] 2.3 Intra-block copy (IBC)

[0453] Intra-frame block copy (IBC), also known as current picture reference, has been adopted by HEVC Screen Content Codec Extension (HEVC-SCC) and the current VVC Test Model (VTM-4.0). IBC extends the concept of motion compensation from inter-frame codecs to intra-frame codecs. Fig.21 As shown, when IBC is applied, the current block is predicted by the reference block in the same picture. Before the current block is encoded or decoded, the samples in the reference block must have been reconstructed. Although IBC is not very effective for most camera-captured sequences, it shows significant codec gains for screen content. The reason is that there are many repeated patterns in screen content pictures, such as icons and text characters. IBC can effectively eliminate the redundancy between these repeated patterns. In HEVC-SCC, IBC can be applied if the inter-frame codec unit (CU) selects the current picture as a reference picture. In this case, MV is renamed block vector (BV), and BV always has integer pixel precision. In order to be compatible with the main configuration HEVC, the current picture is marked as a "long-term" reference picture in the decoded picture buffer (DPB). It should be noted that similarly, in the multi-view / 3D video codec standard, the inter-view reference picture is also marked as a "long-term" reference picture.

[0454] After the BV finds its reference block, a prediction can be generated by copying the reference block. The residual can be obtained by subtracting the reference pixels from the original signal. Then, transforms and quantization can be applied as in other codec modes.

[0455] However, when the reference block is outside the picture, or overlaps with the current block, or is outside the reconstructed area, or is outside the valid area subject to certain constraints, some or all pixel values ​​are undefined. Basically, there are two solutions to deal with such problems. One is to not allow this situation, such as in bitstream consistency. The other is to apply padding to those undefined pixel values. The following subsections will introduce the solutions in detail.

[0456] 2.3.1 IBC in VVC Test Model (VTM4.0)

[0457] In the current VVC test model, that is, in the VTM-4.0 design, the entire reference block should be together with the current codec tree unit (CTU) and not overlap with the current block. Therefore, there is no need to fill the reference block or prediction block. The IBC flag is encoded as the prediction mode of the current CU. Therefore, there are three prediction modes for each CU, MODE_INTRA, MODE_INTER, and MODE_IBC.

[0458] 2.3.1.1 IBC Merge Mode

[0459] In IBC merge mode, the index pointing to the entry in the IBC Merge candidate list is parsed from the bitstream. The construction of the IBC Merge list can be summarized in the following steps:

[0460] οStep 1: Derive Spatial Candidates

[0461] οStep 2: Insert HMVP candidates

[0462] οStep 3: Insert pairwise average candidates

[0463] In the derivation of spatial domain Merge candidates, Figure 2 As shown, up to four Merge candidates are selected from the candidates at the positions shown in A1, B1, B0, A0, and B2. The derivation order is A1, B1, B0, A0, and B2. Position B2 is considered only when any PU at position A1, B1, B0, A0 is unavailable (for example, because it belongs to another slice or piece) or when IBC mode encoding and decoding is not used. After adding the candidate at position A1, the insertion of the remaining candidates is checked for redundancy to ensure that candidates with the same motion information are excluded from the list, thereby improving encoding and decoding efficiency.

[0464] After inserting the spatial domain candidates, if the IBC Merge list size is still less than the maximum size of the IBC Merge list, the IBC candidates from the HMVP table can be inserted. Redundancy checks are performed when inserting HMVP candidates.

[0465] Finally, the pairwise average candidates are inserted into the IBC Merge list.

[0466] When the reference block identified by a Merge candidate is outside the picture, or overlaps with the current block, or is outside the reconstruction area, or is outside the valid area subject to certain constraints, the Merge candidate is called an invalid Merge candidate.

[0467] Note that invalid merge candidates may be inserted into the IBC Merge list.

[0468] 2.3.1.2 IBC AMVP Model

[0469] In IBC AMVP mode, the AMVP index pointing to the entry in the IBC AMVP list is parsed from the bitstream. The construction of IBCAMVP can be summarized in the following steps:

[0470] οStep 1: Derive Spatial Candidates

[0471] Check A0, A1 until a usable candidate is found.

[0472] Check B0, B1, B2 until a usable candidate is found.

[0473] οStep 2: Insert HMVP candidates

[0474] οStep 3: Insert zero candidates

[0475] After inserting the spatial candidates, if the IBC AMVP list size is still smaller than the maximum IBC AMVP list size, the IBC candidates in the HMVP table can be inserted.

[0476] Finally, insert 0 candidates into the IBC AMVP list.

[0477] 2.3.1.3 Chroma IBC Mode

[0478] In the current VVC, motion compensation in chroma IBC mode is performed at the sub-block level. The chroma block will be split into several sub-blocks. Each sub-block determines whether the corresponding luminance block has a block vector, and if so, determines its validity. There is a codec constraint in the current VTM, and if all sub-blocks in the current chroma CU have valid luminance block vectors, the chroma IBC mode will be tested. For example, in YUV 420 video, the chroma block is NxM, and then the concatenated luminance area is 2Nx2M. The sub-block size of the chroma block is 2x2. There are several steps to perform the chroma mv derivation, followed by a block copy process.

[0479] 1) First, split the chrominance block into (N>>1)*(M>>1) sub-blocks.

[0480] 2) Each sub-block with the upper left sample point coordinate located at (x, y) obtains the corresponding luminance block covering the same upper left sample point with the coordinate located at (2x, 2y).

[0481] 3) The codec checks the block vector (bv) of the acquired luma block. If one of the following conditions is met, bv will be considered invalid.

[0482] a. The bv of the corresponding luminance block does not exist.

[0483] b. The prediction block identified by bv has not been reconstructed.

[0484] c. The prediction block identified by bv partially or completely overlaps with the current block.

[0485] 4) Set the chrominance motion vector of the sub-block to the motion vector of the corresponding luminance sub-block.

[0486] When valid bv is found for all sub-blocks, IBC mode is allowed at the codec.

[0487] 2.3.2 Latest Development of IBC (in VTM5.0)

[0488] 2.3.2.1 Single BV List

[0489] VVC adopts JVET-N0843. In JVET-N0843, the BV predictors of Merge mode and AMVP mode in IBC will share a common predictor list, which consists of the following elements:

[0490] ο2 adjacent airspace locations (A1, B1, such as Figure 2 (shown)

[0491] ο5 HMVP entries

[0492] οDefault zero vector

[0493] The number of candidates in the list is controlled by a variable derived from the stripe header. For Merge mode, at most the first 6 entries of this list will be used; for AMVP mode, the first 2 entries of this list will be used. And this list meets the shared Merge list region requirement (sharing the same list in SMR).

[0494] In addition to the above BV predictor candidate list, JVET-N0843 also proposes to simplify the pruning operations between HMVP candidates and existing Merge candidates (A1, B1). In the simplification, since only the first HMVP candidate and the spatial Merge candidate are compared, there will be at most 2 pruning operations.

[0495] 2.3.2.2 Update of IBC HMVP table

[0496] Similar to the design of the HMVP table update process for inter-mode coding, pruning is applied and a counter is maintained to record the number of available IBC candidates in the IBC HMVP table.

[0497] The relevant specifications are defined as follows:

[0498] 8.6.2.6 Update of the History-Based Motion Vector Predictor Candidate List

[0499] The input to this process is:

[0500] - Luma motion vector mvL with 1 / 16 fractional sample accuracy.

[0501] The candidate list HmvpIbcCandList is modified in the following order of steps:

[0502] 1. Set the variable identicalCandExist equal to FALSE and the variable removeIdx equal to 0.

[0503] 2. When NumHmvpIbcCand is greater than 0, for each index hMvpIdx of hMvpIdx=0..NumHmvpIbcCand-1, the following steps apply until identicalCandExist equals TRUE:

[0504] -When hMvpCand is equal to HmvpIbcCandList[hMvpIdx], set identicalCandExist equal to TRUE and set removeIdx equal to hMvpIdx.

[0505] 3. The candidate list HmvpIbcCandList is updated as follows:

[0506] - If identicalCandExist equals TRUE or NumHmvpIbcCand equals MaxNumMergeCand-1, the following applies:

[0507] - For each index i where i = (removeIdx+1)..(NumHmvpIbcCand-1), set HmvpIbcCandList[i-1] equal to HmvpIbcCandList[i].

[0508] -Set -HmvpIbcCandList[NumHmvpIbcCand-1] equal to mvCand.

[0509] Otherwise (identicalCandExist is equal to FALSE and less than MaxNumMergeCand-1), the following applies:

[0510] -Set HmvpIbcCandList[NumHmvpIbcCand++] equal to mvCand.

[0511] Here, MaxNumMergeCand is set to 6.

[0512] 2.4 Switchable interpolation filter

[0513] 2.4.1 Half-Pixel AMVR Mode

[0514] An additional AMVR mode for non-affine non-Merge inter-coded CUs is proposed, which allows motion vector differences to be signaled with half-pixel precision. The existing AMVR scheme in the current VVC draft is directly extended in the following way: immediately following the syntax element amvr_flag, if amvr_flag==1, there is a new context modeling binary syntax element hpel_amvr_flag, if hpel_amvr_flag==1, it indicates the use of the new half-pixel AMVR mode. Otherwise, that is, if hpel_amvr_flag==0, as in the current VVC draft, the selection between full-pixel and 4-pixel AMVR modes is indicated by the syntax element amvr_precision_flag.

[0515] AMVR signaling is extended, such as Fig.24 As shown in the table.

[0516] In JVET-N0309, an optional brightness half-pixel interpolation filter was proposed, in which two 6-tap filters with smoothing characteristics were tested in CE4-1, denoted as FlatTop and Gauss. Fig.25 .

[0517] 2.4.2 Explicit Signaling

[0518] For non-affine and non-Merge inter-frame coded CUs using half-pixel motion vector precision (i.e., half-pixel AMVR mode), an optional luma half-pixel interpolation filter is used. For the test described in 2.4.1, a Gaussian luma half-pixel interpolation filter is used. For this test, switching between two optional half-pixel interpolation filters is performed based on the value of the new syntax element hpel_if_idx. The syntax element hpel_if_idx is signaled only in half-pixel AMVR mode, as shown below:

[0519]

[0520] 2.4.3 Implicit Signaling

[0521] In Skip / Merge mode using spatial merge candidates, the information of applying interpolation filters to half-pixel positions is inherited from neighboring blocks.

[0522] 3. Examples of Problems Solved by Disclosed Embodiments

[0523] The current HMVP design must keep track of the number of available HMVP candidates in the HMVP table. Two different counters are maintained for the HMVP table for inter-codec blocks and the IBC HMVP table for IBC codec blocks. Whenever the HMVP table needs to be updated, the counter needs to be checked and modified accordingly.

[0524] 4. Example Embodiments

[0525] The following list should be considered as examples to explain the general concept. These embodiments should not be interpreted narrowly. In addition, these inventions can be combined in any way.

[0526] The examples described below for various implementations illustrate LUT-based motion vector prediction based on the disclosed technology that can enhance existing and future video codec standards. In the LUT-based motion vector prediction method, during the encoding / decoding process, one or more tables are maintained that have motion information from previously encoded blocks. During the encoding / decoding of a block, the associated motion information in the LUT can be added to the motion candidate list, and after encoding / decoding a block, the LUT can be updated. The LUT-based method is also called a history-based motion vector prediction (HMVP) method. The motion information includes at least one of the following: prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference accuracy, or motion vector difference value. Candidate motions may also include other kinds of codec information, such as intra-frame prediction modes for intra-frame mode encoding and decoding; illumination compensation parameters for IC parameter encoding and decoding; and filter parameters.

[0527] Before encoding and decoding a video unit (e.g., a slice / tile / picture / CTU row / CTU), the HMVP table can be reset by filling one or more default candidate motions instead of resetting the counter of the available HMVP candidates in the table to 0. N represents the size of the HMVP table, that is, at most N HMVP candidates can be stored.

[0528] In the following discussion, the term "slice" may also be referred to as "slice group" or "slice / brick / CTU row" or other video data units.

[0529] 1. Whenever the HMVP table is reset, K (K<=N) default HMVP candidates may be used to populate the HMVP table.

[0530] a. In one example, the K default HMVP candidates are the same.

[0531] i. Optionally, at least two of the K default HMVP candidates are different.

[0532] ii. Optionally, a default HMVP candidate can be added to the HMVP table multiple times.

[0533] 1) Optionally, the first m (m < K) default HMVP candidates can be different from each other. However, the remaining last (K - m) candidates are the same as one of the first m candidates.

[0534] 2) Optionally, the last m (m < K) default HMVP candidates can be different from each other. However, the remaining first (K - m) candidates are the same as one of the first m candidates.

[0535] iii. Optionally, the first m (m < K) default HMVP candidates can be different from each other. However, the remaining last (K - m) candidates are filled by repeatedly copying the first m candidates.

[0536] b. In one example, the default HMVP candidate can be set as a unidirectional prediction motion candidate.

[0537] i. In one example, for P slices / pictures, the prediction direction can be set to list 0.

[0538] ii. In one example, for B slices / pictures, the prediction direction can be set to list 0 or list 1.

[0539] iii. Optionally, in addition, the motion vector (MV) of the default HMVP candidate can be set to (0, 0).

[0540] iv. Optionally, in addition, the MV of the default HMVP candidate can be set to (-m * W, 0), where m and W are two integers. For example, W is one of the allowed widths of the coding / decoding unit / prediction unit / transformation unit.

[0541] v. Optionally, in addition, the MV of the default HMVP candidate can be set to (0, -q * H), where q and H are two integers. For example, H is one of the allowed heights of the coding / decoding unit / prediction unit / transformation unit.

[0542] vi. Optionally, in addition, the MV of the default HMVP candidate can be set to (-m * W, -q * H), where m, W, q, and H are integers. For example, W and H are respectively one of the allowed width and height of the coding / decoding unit / prediction unit / transformation unit.

[0543] vii. Optionally, in addition, the reference index of the default HMVP candidate can be set to any value less than the number of reference pictures in the prediction direction (i.e., list 0 or 1).

[0544] viii. Optionally, in addition, the reference picture can be set to the current picture.

[0545] ix. Optionally, in addition, the IBC flag can be set to 1.

[0546] c. In one example, the default HMVP candidate may be set as the bi-directional prediction motion candidate.

[0547] i. In one example, the items under 1.ii may be used to set the motion information associated with each prediction direction.

[0548] ii. Optionally, in addition, for two MVs, the weighting factors of the two predictions from the two directions may be set equal.

[0549] 1) In one example, the generalized bidirectional prediction (GBI) index is set to 0.

[0550] d. The default HMVP candidate may include motion information from another video unit, such as another CTU row / another picture.

[0551] e. In one example, for a P slice / picture, the default HMVP candidate may include (Mv(0,0), i), where i is a reference picture index in the range of [0, the number of reference pictures in list 0 minus 1].

[0552] i. Optionally, further, candidates may be added to the HMVP table in ascending / descending order of i.

[0553] ii. Optionally, furthermore, if the number of reference pictures in list 0 is less than K, one of the default motion candidates may be repeatedly added until K candidates are available.

[0554] f. Optionally, in addition, if the number of reference pictures in list 0 is less than K, the available default HMVP candidates may be repeatedly added until K candidates are available. In one example, for a B slice / picture, the default HMVP candidate may include a K0 unidirectional prediction HMVP candidate from list X, a K1 unidirectional prediction HMVP candidate from list Y (Y equals (1-X)), and a K2 bidirectional prediction HMVP candidate, where (K0+K1+K2=K, and K0 / K1 / K2>=0).

[0555] i. In one example, for a B slice / picture, the default HMVP candidate may include only bi-predictive HMVP candidates.

[0556] ii. In one example, for a B slice / picture, the default HMVP candidate may include uni-directional prediction HMVP candidates and bi-directional prediction HMVP candidates in an interlaced manner.

[0557] iii. In one example, for a B slice / picture, the default HMVP candidate may include a uni-directional prediction HMVP candidate followed by a bi-directional prediction HMVP candidate.

[0558] g. Optionally, in addition, when K is equal to N, the counter maintained for recording the number of available HMVP candidates in the table is removed.

[0559] h. In one example, a default HMVP candidate can only reference reference pictures whose reference index is equal to a specific value (eg, 0).

[0560] 2. The default HMVP candidate may change adaptively from one video unit to another.

[0561] a. The default HMVP candidate can be adaptively changed from one CTU row to another.

[0562] b. The default HMVP candidate may change adaptively from one slice to another.

[0563] 3. For different HMVP tables, the default HMVP candidates may be different.

[0564] a. In one example, the selection of the default HMVP candidate may depend on the HMVP table for the codec mode.

[0565] i. In one example, the default HMVP candidates used to populate the HMVP table for inter-codec blocks and IBC-codec blocks may be different.

[0566] 4. When N default HMVP candidates are populated in the HMVP table before encoding / decoding a video unit, the following further applies when a new motion candidate is added.

[0567] a. In one example, the new motion candidate is compared with the available HMVP candidates according to the ascending or descending order of the index of the HMVP candidates in the HMVP table.

[0568] i. In one example, when the same HMVP candidate is found, the comparison process is terminated. Optionally, furthermore, the same HMVP candidate is removed from the table and the new motion candidate is added to the last entry of the table.

[0569] ii. In one example, after checking all HMVP candidates in the table, the comparison process terminates. Optionally, in addition, all identical M HMVP candidates are removed from the table, and new motion candidates are repeatedly added until the table is full.

[0570] 5. The default HMVP candidate may be signaled in the VPS / DPS / SPS / APS / PPS / picture header / slice header / slice group header.

[0571] a. Optionally, multiple sets of default HMVP candidates may be predefined, and one or more indexes may be signaled to indicate which set of HMVP candidates a picture / slice / slice group / slice / CTU row uses.

[0572] b. Optionally, multiple groups of default HMVP candidates may be predefined, and an index of a coding mode (eg, IBC or non-IBC) may be signaled to indicate which group of HMVP candidates a picture / slice / slice group / slice / CTU row uses.

[0573] 6. When updating the HMVP table / HMVP list, the new motion candidate to be added may be added before all existing motion candidates in the HMVP table / HMVP list. In this case, the entry index of the table with the smallest value may indicate the latest motion candidate in decoding order.

[0574] a. In one example, if a new motion candidate is the same or similar to an existing candidate in the table, then the motion candidate may not be added.

[0575] i. Optionally, if the new motion candidate is the same or similar to an existing candidate in the table (eg, the MV difference is less than a threshold), the new motion candidate may not be added, and motion candidates that are the same or similar to the existing candidates may be removed from the table.

[0576] 7. The above method is applicable to all HMVP tables (eg, including those to be used by inter-frame or IBC codec blocks).

[0577] a. Optionally, which HMVP table the above method should apply depends on the usage of the table, such as whether it is used for an IBC codec block.

[0578] 8. The proposed method can be applied to other codecs with history information, such as history-based intra-mode codecs with default intra-prediction modes filling the LUT.

[0579] Storage of motion information in HMVP table / HMVP list

[0580] 9. Indications of interpolation filters for blocks of switchable interpolation filters and / or side information may be stored in motion candidates stored in the HMVP table / HMVP list.

[0581] a. In one example, the motion candidate includes an indication of the interpolation filter used for the block stored along with other motion information, such as MV, reference picture index, GBi (generalized bi-directional prediction) / BCW (also known as bi-directional prediction with CU weight) index, etc.

[0582] b. In one example, two motion candidates with different interpolation filters may be considered as different motion candidates.

[0583] c. In one example, two motion candidates with different interpolation filters may be considered the same motion candidate. For example, for motion candidate X in the HMVP table, if motion candidate Y has the same information as X except for the interpolation filter, X may be removed from the HMVP table and Y may be inserted into the HMVP table.

[0584] d. In one example, whether to store an indication of the interpolation filter of a motion candidate in the HMVP table may depend on the MV resolution of the motion candidate.

[0585] i. For example, if the candidate MV has half-pixel resolution, an indication of the interpolation filter used for the motion candidate may be stored in the HMVP table.

[0586] ii. In one example, interpolation filters may be stored for each prediction direction.

[0587] Use of motion information in HMVP tables / HMVP lists

[0588] 10. During the pruning process of new candidates against existing candidates in the candidate list (eg, comparing and / or avoiding adding new candidates), the interpolation filter information should also be compared.

[0589] e. In one example, assuming that there are two candidates with the same motion information except for the interpolation filters associated with the two candidates, they are considered different. In this case, a new candidate can also be added to the candidate list.

[0590] 11. When a motion (HMVP) candidate in an HMVP table / HMVP list is added as a new (eg AMVR or merge) candidate to a list (eg AMVP or merge list), the interpolation filter associated with the motion candidate may also be inherited by the new candidate.

[0591] f. In one example, in the pruning process, interpolation filter information may also be considered.

[0592] 12. Multiple new candidates can be derived into lists (eg AMVP or Merge list) based on one motion (HMVP) candidate in the HMVP table / HMVP list, and each can be associated with a different interpolation filter.

[0593] Fig. 22 2200. The apparatus 2200 may be used to implement one or more of the methods described herein. The apparatus 2200 may be implemented in a smartphone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. The apparatus 2200 may include one or more processors 2202, one or more memories 2204, and video processing hardware 2206. The processor 2202 may be configured to implement one or more of the methods described in this document. The memory (memories) 2204 may be used to store data and code for implementing the methods and techniques described herein. The video processing hardware 2206 may be used to implement some of the techniques described in this document in hardware circuits.

[0594] Fig.23 2300 is a flow chart of an example method 2300 for video processing. The method 2300 includes: when determining that a history-based motion vector prediction (HMVP) table of size N is reset, using (2302) a set of K default HMVP candidates to populate the HMVP table, where N and K are positive integers and K≤N; and performing (2304) a conversion between a current block and a bitstream representation of the current block based on the HMVP table.

[0595] Some embodiments may preferably include the following technical examples.

[0596] 1. A method for video processing, comprising: when determining that a history-based motion vector prediction (HMVP) table of size N is reset, filling the HMVP table with a set of K default HMVP candidates, wherein N and K are positive integers and K≤N; and performing conversion between a current block and a bitstream representation of the current block based on the HMVP table.

[0597] 2. The method of example 1, wherein each of the sets of K default HMVP candidates is the same.

[0598] 3. The method of example 1, wherein at least two of the set of K default HMVP candidates are different.

[0599] 4. The method of example 1, wherein at least one of the set of K default HMVP candidates is added to the HMVP table multiple times.

[0600] 5. The method of Example 4, wherein each of the first m candidates in the HMVP table is different, and wherein each of the last (Km) candidates in the HMVP table is the same as one of the first m candidates, where m is a positive integer.

[0601] 6. The method of Example 4, wherein each of the last (Km) candidates in the HMVP table is different, and wherein each of the first m candidates in the HMVP table is the same as one of the last (Km) candidates, where m is a positive integer.

[0602] 7. The method of example 4, wherein each of the first m candidates in the HMVP table is different, and wherein the last (Km) candidates in the HMVP table include one or more copies of the first m candidates.

[0603] 8. The method of example 1, wherein at least one of the set of K default HMVP candidates is a uni-directionally predicted motion candidate.

[0604] 9. The method according to example 8, wherein the slice or picture including the current block is a P slice or picture, respectively, and wherein the prediction direction of the uni-directional prediction motion candidate is set to list 0 (L0).

[0605] 10. The method according to example 8, wherein the slice or picture including the current block is a B slice or picture, respectively, and wherein the prediction direction of the uni-directional prediction motion candidate is set to list 0 (L0) or list 1 (L1).

[0606] 11. The method of example 8, wherein the motion vector of the uni-directional prediction motion candidate is set to (0, 0).

[0607] 12. The method of example 8, wherein the motion vector of the uni-directional prediction motion candidate is set to (-m×W, 0), where m and W are integers.

[0608] 13. The method of example 8, wherein the motion vector of the uni-directional prediction motion candidate is set to (0, -q×H), where q and H are integers.

[0609] 14. The method of example 8, wherein the motion vector of the uni-directional prediction motion candidate is set to (-m×W, -q×H), where m, q, H and W are integers.

[0610] 15. A method according to example 12 or 14, wherein W is the allowed width of a prediction unit (PU), a coding unit (CU), or a transform unit (TU) associated with the current block.

[0611] 16. The method of example 13 or 14, wherein H is an allowed height of a prediction unit (PU), a coding unit (CU), or a transform unit (TU) associated with the current block.

[0612] 17. The method of example 1, wherein at least one of the set of K default HMVP candidates is a bi-predictive motion candidate.

[0613] 18. The method of example 1, wherein at least one of the set of K default HMVP candidates includes motion information from a different video unit than the video unit containing the current block.

[0614] 19. The method of example 18, wherein the video unit comprises a codec tree unit (CTU) row or a picture.

[0615] 20. A method for video processing, comprising: for a conversion between a current block and a bitstream representation of the current block, determining a set of default history-based motion vector prediction (HMVP) candidates to be used during the conversion; and performing the conversion using the set of default HMVP candidates, wherein different HMVP candidates are used during the conversion of different video units.

[0616] 21. The method of example 20, wherein the different video units comprise different codec tree unit (CTU) rows or different blocks.

[0617] 22. A method for video processing, comprising: prior to conversion between a current block and a bitstream representation of the current block, determining that a set of N default history-based motion vector prediction (HMVP) candidates has been populated in an HMVP table; adding motion candidates to the HMVP table; and performing conversion using the HMVP table containing the motion candidates.

[0618] 23. The method of example 22, further comprising comparing the motion candidate with each candidate in the HMVP table based on an ascending or descending order of an index of the set of N default HMVP candidates.

[0619] 24. The method of example 23, wherein the comparison is terminated when the motion candidate matches a particular candidate in the HMVP table.

[0620] 25. The method of example 24, wherein the particular candidate is removed from the HMVP table, and wherein the motion candidate is added to the last entry of the HMVP table.

[0621] 26. A method according to any one of Examples 1 to 25, wherein the set of default HMVP candidates is signaled in a video parameter set (VPS), a decoder parameter set (DPS), a sequence parameter set (SPS), an adaptation parameter set (APS), a picture parameter set (PPS), a picture header, a slice header, or a slice group header.

[0622] 27. The method of any one of Examples 1 to 25, wherein a plurality of sets of default HMVP candidates are predefined, and wherein an index of the set of default HMVP candidates in the plurality of sets of default HMVP candidates is signaled.

[0623] 28. A method according to any one of Examples 1 to 27, wherein the current block is encoded and decoded using an inter-frame coding mode or an intra-frame block copy (IBC) mode.

[0624] 29. A video processing method, comprising: before conversion between a current video block and a bitstream representation of the current video block, storing information of an interpolation filter for the current video block, or side information about a switchable interpolation filter for the current video block, together with past motion information of the current video block in a table; performing conversion based on entries in the table; and updating the table after performing the conversion.

[0625] 30. A method according to Example 29, wherein the past motion information includes motion candidates, the motion candidates including information or side information for an interpolation filter, and at least one of a motion vector, a reference picture index, generalized bidirectional prediction information, and a bidirectional prediction weight index.

[0626] 31. A method according to any of Examples 29-30, wherein the updating includes an update based on information for an interpolation filter.

[0627] 32. A method according to any of Examples 29-31, wherein the update includes an update based on the side information.

[0628] 33. A method according to any of Examples 31-32, wherein information used for the interpolation filter or side information is used to determine the uniqueness of the corresponding motion candidate.

[0629] 34. A method according to any one of Examples 29-33, wherein the updating is based on a pruning process in which new motion candidates are compared with information stored in a table and new motion candidates are selectively added based on unique information contained in the new motion candidates.

[0630] 35. A method according to Example 34, wherein the pruning process determines uniqueness based on information used for the interpolation filter.

[0631] 36. A method according to any of Examples 29-35, wherein updating includes updating a table using motion candidates from an advanced motion vector resolution list or a Merge candidate, and corresponding interpolation filter information or side information.

[0632] 37. A method according to any of Examples 29-35, wherein updating includes generating new candidates by deriving from other candidates with different interpolation information or different side information.

[0633] 38. A method according to any of Examples 29-37, wherein the conversion includes an encoding process that generates a bitstream representation from the current video block.

[0634] 39. A method according to any of Examples 29-37, wherein the conversion includes generating a decoding process of the current video block from a bitstream representation.

[0635] Items 9-12 in the previous section provide additional aspects and variations related to Examples 29-39.

[0636] 40. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein when the processor executes the instructions, the instructions cause the processor to implement the method of any one of Examples 1-39.

[0637] 41. A computer program product stored on a non-transitory computer readable medium, the computer program product comprising program code for implementing the method of any of Examples 1-39.

[0638] In the example listings in this document, the term conversion may refer to generating a bitstream representation for or from a current video block. The bitstream representation need not represent a contiguous group of bits and may be divided into bits contained in a header field or in a codeword representing coded pixel value information.

[0639] It should be understood that the disclosed techniques can be implemented in a video encoder or decoder to improve compression efficiency using techniques including using various implementation rules that take into account the use of history-based motion vector prediction (HMVP) with default parameters as described herein.

[0640] Some embodiments of the disclosed technology include making a decision or determination to enable a video processing tool or mode. In one example, when a video processing tool or mode is enabled, the encoder will use or implement the tool or mode in the processing of a video block, but will not necessarily modify the generated bitstream based on the use of the tool or mode. That is, when a video processing tool or mode is enabled based on a decision or determination, the transformation from a video block to a bitstream representation of a video will use the video processing tool or mode. In another example, when a video processing tool or mode is enabled, the decoder will process the bitstream knowing that the bitstream has been modified based on the video processing tool or mode. That is, the transformation from a bitstream representation of a video to a video block will be performed using a video processing tool or mode enabled based on a decision or determination.

[0641] Some embodiments of the disclosed technology include making a decision or determination to disable a video processing tool or mode. In one example, when the video processing tool or mode is disabled, the encoder will not use the tool or mode in transforming video blocks to a bitstream representation of the video. In another example, when the video processing tool or mode is disabled, the decoder will process the bitstream knowing that the bitstream has not been modified using the video processing tool or mode enabled based on the decision or determination.

[0642] Fig.26 1 is a block diagram illustrating an exemplary video coding system 100 that may utilize the techniques of the present invention. Fig.26 As shown, the video codec system 100 may include a source device 110 and a destination device 120. The source device 110 generates encoded video data, which may be referred to as a video encoding device. The destination device 120 may decode the encoded video data generated by the source device 110, which may be referred to as a video decoding device. The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0643] The video source 112 may include sources such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of these sources. The video data may include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bit stream. The bit stream may include a bit sequence that forms a codec representation of the video data. The bit stream may include a codec picture and related data. The codec picture is a codec representation of the picture. Related data may include sequence parameter sets, picture parameter sets and other syntax structures. The I / O interface 116 may include a modulator / demodulator (modem) and / or a transmitter. The encoded video data may be directly signaled to the destination device 120 via the network 130a via the I / O interface 116. The encoded video data may also be stored on a storage medium / server 130b for access by the destination device 120.

[0644] Destination device 120 may include an I / O interface 126 , a video decoder 124 , and a display device 122 .

[0645] The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain the encoded video data from the source device 110 or the storage medium / server 130b. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to the user. The display device 122 may be integrated with the destination device 120, or may be external to the destination device 120 configured to interface with an external display device.

[0646] The video encoder 114 and the video decoder 124 may operate according to a video compression standard, such as the High Efficiency Video Coding (HEVC) standard, the Versatile Video Codec (VVM) standard, and other current and / or future standards.

[0647] Fig. 27 is a block diagram showing an example of a video encoder 200, which may be Fig.26 The video encoder 114 in the system 100 is shown.

[0648] Video encoder 200 may be configured to perform any or all of the techniques of this disclosure. Fig. 27 In the example of , video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video encoder 200. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0649] The functional components of the video encoder 200 may include a segmentation unit 201, a prediction unit 202 which may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205 and an intra-frame prediction unit 206, a residual generation unit 207, a transform unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transform unit 211, a reconstruction unit 212, a buffer 213 and an entropy coding unit 214.

[0650] In other examples, the video encoder 200 may include more, fewer, or different functional components. In an example, the prediction unit 202 may include an intra block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, where at least one reference picture is a picture where the current video block is located.

[0651] Furthermore, some components (such as the motion estimation unit 204 and the motion compensation unit 205) may be highly integrated, but for the purpose of explanation, they are described in detail in the following sections. Fig. 27 In the example, they are represented separately.

[0652] The partitioning unit 201 may partition a picture into one or more video blocks. The video encoder 200 and the video decoder 300 may support various video block sizes.

[0653] The mode selection unit 203 may select one of the coding modes (intra or inter), for example, based on the error result, and provide the resulting intra or inter coded block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 203 may select a combined intra-frame inter-frame (CIIP) mode in which prediction is based on an inter-frame prediction signal and an intra-frame prediction signal. In the case of inter-frame prediction, the mode selection unit 203 may also select a resolution for the motion vector of the block (e.g., sub-pixel or integer pixel precision).

[0654] To perform inter-frame prediction on the current video block, the motion estimation unit 204 may generate motion information of the current video block by comparing the current video block with one or more reference frames in the buffer 213. The motion compensation unit 205 may determine a predicted video block for the current video block based on the motion information of a picture (different from the picture associated with the current video block) from the buffer 213 and decoded samples.

[0655] For example, motion estimation unit 204 and motion compensation unit 205 may perform different operations on the current video block depending on whether the current video block is in an I slice, a P slice, or a B slice.

[0656] In some examples, the motion estimation unit 204 may perform unidirectional prediction on the current video block, and the motion estimation unit 204 may search for a reference video block of the current video block in the reference pictures of list 0 or list 1. Then, the motion estimation unit 204 may generate a reference index indicating a reference picture in list 0 or list 1 that includes the reference video block and a motion vector indicating a spatial displacement between the current video block and the reference video block. The motion estimation unit 204 may output the reference index, the prediction direction indicator, and the motion vector as motion information of the current video block. The motion compensation unit 205 may generate a predicted video block of the current block based on the reference video block indicated by the motion information of the current video block.

[0657] In other examples, the motion estimation unit 204 may perform bidirectional prediction on the current video block, and the motion estimation unit 204 may search for a reference video block of the current video block in the reference pictures of list 0, and may also search for another reference video block of the current video in the reference pictures of list 1. Then, the motion estimation unit 204 may generate a reference index indicating a reference picture in list 0 or list 1 that includes a reference video block and a motion vector indicating a spatial displacement between the reference video block and the current video block. The motion estimation unit 204 may output the reference index and the motion vector of the current video block as the motion information of the current video block. The motion compensation unit 205 may generate a predicted video block of the current video block based on the reference video block indicated by the motion information of the current video block.

[0658] In some examples, motion estimation unit 204 may output a full set of motion information for use in the decoding process of a decoder.

[0659] In some examples, motion estimation unit 204 may not output a full set of motion information for the current video. Instead, motion estimation unit 204 may reference motion information of another video block to signal motion information of the current video block. For example, motion estimation unit 204 may determine that the motion information of the current video block is sufficiently similar to the motion information of an adjacent video block.

[0660] In one example, motion estimation unit 204 may indicate a value in a syntax structure associated with the current video block that indicates to video decoder 300 that the current video block has the same motion information as another video block.

[0661] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0662] As described above, the video encoder 200 may predictively signal motion vectors. Two examples of predictive signaling techniques that may be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and Merge mode signaling.

[0663] The intra prediction unit 206 may perform intra prediction on the current video block. When the intra prediction unit 206 performs intra prediction on the current video block, the intra prediction unit 206 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include the predicted video block and various syntax elements.

[0664] The residual generation unit 207 may generate residual data for the current video block by subtracting (eg, indicated by a minus sign) the predicted video block of the current video block from the current video block. The residual data for the current video block may include residual video blocks corresponding to different sample-by-sample components of the samples in the current video block.

[0665] In other examples, such as in skip mode, the current video block of the current video block may have no residual data, and the residual generation unit 207 may not perform a subtraction operation.

[0666] Transform processing unit 208 may generate one or more transform coefficient video blocks for a current video block by applying one or more transforms to the residual video block associated with the current video block.

[0667] After transform processing unit 208 generates a transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0668] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to the corresponding samples of one or more prediction video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current block to be stored in the buffer 213.

[0669] After the reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce video block artifacts in the video block.

[0670] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the entropy coding unit 214 receives the data, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy-coded data and output a bitstream including the entropy-coded data.

[0671] Fig.28 It shows that it can be Fig.26 A block diagram of an example of a video decoder 300 of the video decoder 114 in the system 100 is shown.

[0672] Video decoder 300 may be configured to perform any or all of the techniques of this disclosure. Fig.28 In the example of , video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video decoder 300. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0673] exist Fig.28 In the example of FIG. 3 , the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra-frame prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, and a reconstruction unit 306 and a buffer 307. In some examples, the video decoder 300 may perform an encoding process generally similar to that described with respect to the video encoder 200 ( Fig. 27 )The opposite decoding process.

[0674] The entropy decoding unit 301 may retrieve an encoded bitstream. The encoded bitstream may include entropy encoded video data (e.g., encoded blocks of video data). The entropy decoding unit 301 may decode the entropy encoded video data, and the motion compensation unit 302 may determine motion information including motion vectors, motion vector precision, reference picture list index, and other motion information from the entropy decoded video data. For example, the motion compensation unit 302 may determine this information by performing AMVP and Merge modes.

[0675] The motion compensation unit 302 may generate a motion compensation block by interpolating based on an interpolation filter. An identifier of an interpolation filter used with sub-pixel precision may be included in the syntax element.

[0676] The motion compensation unit 302 may calculate interpolation of sub-integer pixels of the reference block using the interpolation filter used by the video encoder 20 during encoding of the video block. The motion compensation unit 302 may determine the interpolation filter used by the video encoder 200 according to the received syntax information, and use the interpolation filter to generate a prediction block.

[0677] The motion compensation unit 302 may use some syntax information to determine the size of blocks used to encode frames and / or slices of the encoded video sequence, partitioning information describing how each macroblock of the pictures of the encoded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-frame coded block, and other information used to decode the encoded video sequence.

[0678] The intra prediction unit 303 may form a prediction block from spatially neighboring blocks using, for example, an intra prediction mode received in the bitstream. The inverse quantization unit 303 inversely quantizes (ie, dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 303 applies an inverse transform.

[0679] The reconstruction unit 306 may add the residual block to the corresponding prediction block generated by the motion compensation unit 202 or the intra prediction unit 303 to form a decoded block. If necessary, a deblocking filter may also be applied to filter the decoded block to remove blockiness artifacts. The decoded video block is then stored in a buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction and also produces decoded video for presentation on a display device.

[0680] In this document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during conversion from a pixel representation of a video to a corresponding bitstream representation, or vice versa. For example, a bitstream representation or a codec representation of a current video block may correspond to bits juxtaposed or propagated at different locations within the bitstream, as defined by the syntax. For example, a video block may be encoded based on a transformed and encoded error residual, and bits in headers and other fields in the bitstream may also be used. In addition, during conversion, the decoder may parse the bitstream based on a determination knowing that certain fields may or may not exist, as described in the solution above. Similarly, the codec may determine whether to include certain syntax fields, and generate a codec representation accordingly by including or excluding syntax fields from the codec representation.

[0681] Fig.29 2900 is a block diagram illustrating an example video processing system 2900 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of system 2900. System 2900 may include an input 2902 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8 or 10-bit multi-component pixel values), or may be received in a compressed or codec format. Input 2902 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, a passive optical network (PON), and wireless interfaces such as Wi-Fi or a cellular interface.

[0682] System 2900 may include a coding component 2904, which may implement various encodings or coding methods described herein. Coding component 2904 may reduce the average bit rate of the video from input 2902 to the output of coding component 2904 to generate a coded representation of the video. Therefore, coding techniques are sometimes referred to as video compression or video transcoding techniques. The output of coding component 2904 may be stored, or signaled by the connected communication represented by component 2906. Component 2908 may use the storage or communication bitstream (or coded) representation of the video received at input 2902 to generate pixel values ​​or displayable video that are signaled to display interface 2910. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "coding" operations or tools, it should be understood that coding tools or operations are used at the encoder, and the corresponding decoding tools or operations of the inverted coding results will be performed by the decoder.

[0683] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB) or High Definition Multimedia Interface (HDMI) or Display Port, etc. Examples of storage interfaces include SATA (Serial Advanced Technology Attachment), PCI, IDE interfaces, etc. The techniques described herein may be implemented in various electronic devices, such as mobile phones, laptop computers, smartphones, or other devices capable of performing digital data processing and / or video display.

[0684] Fig.30 is a flow chart of an example method 3000 for video processing. The method 3000 includes: for conversion between a video unit of a video and a bitstream representation of the video, resetting 3002 a size N N history-based motion vector prediction (HMVP) table by adding K HMVP motion candidates to the HMVP table, wherein the K HMVP candidate motion candidates have default values, wherein N and K are positive integers, and wherein K≤N; and performing 3004 conversion between the video unit and the bitstream representation using the HMVP table resulting from the reset.

[0685] In some embodiments of method 3000, each of the K HMVP candidate motions is the same. In some embodiments of method 3000, at least two of the K HMVP candidate motions are different. In some embodiments of method 3000, at least one motion candidate with a default value is added to the HMVP table multiple times. In some embodiments of method 3000, the first m candidates in the HMVP table are different from each other, each of the last (K - m) candidates in the HMVP table is the same as one of the first m candidates, and m < K. In some embodiments of method 3000, the last (K - m) candidates in the HMVP table are different from each other, each of the first m candidates in the HMVP table is the same as one of the last (K - m) candidates, and m < K. In some embodiments of method 3000, the first m candidates in the HMVP table are different from each other, the last (K - m) candidates in the HMVP table include one or more copies of the first m candidates, and m < K.

[0686] In some embodiments of method 3000, at least one of the K HMVP motion candidates is a uni - directional prediction motion candidate. In some embodiments of method 3000, the video unit is a P - slice or a P - picture, and wherein the prediction direction of the uni - directional prediction motion candidate is set to list 0 (L0). In some embodiments of method 3000, the video unit is a B - slice or a B - picture, and wherein the prediction direction of the uni - directional prediction motion candidate is set to list 0 (L0) or list 1 (L1). In some embodiments of method 3000, the motion vector of the uni - directional prediction motion candidate is set to (0, 0). In some embodiments of method 3000, the motion vector of the uni - directional prediction motion candidate is set to (-m × W, 0), where m and W are integers. In some embodiments of method 3000, the motion vector of the uni - directional prediction motion candidate is set to (0, -q × H), where q and H are integers. In some embodiments of method 3000, the motion vector of the uni - directional prediction motion candidate is set to (-m × W, -q × H), where m, q, H, and W are integers.

[0687] In some embodiments of the method 3000, W is the allowed width of a prediction unit (PU), a codec unit (CU), or a transform unit (TU) of a video block of the video unit. In some embodiments of the method 3000, H is the allowed height of a prediction unit (PU), a codec unit (CU), or a transform unit (TU) of a video block of the video unit. In some embodiments of the method 3000, a reference index of a unidirectional prediction motion candidate is set to a value less than the number of reference pictures in the prediction direction. In some embodiments of the method 3000, a prediction direction of a unidirectional prediction motion candidate is set to list 0 (L0) or list 1 (L1). In some embodiments of the method 3000, a reference picture is set to a current picture associated with the video unit. In some embodiments of the method 3000, a syntax element indicating that intra block copy (IBC) mode is enabled for a video block of the video unit is set to 1.

[0688] In some embodiments of method 3000, at least one of the K HMVP motion candidates is a bidirectional prediction motion candidate. In some embodiments of method 3000, the video unit is a P slice or a picture, and wherein each prediction direction of the bidirectional prediction motion candidate is set to list 0 (L0). In some embodiments of method 3000, the video unit is a B slice or a B picture, and wherein at least one prediction direction of the bidirectional prediction candidate motion is set to list 0 (L0) or list 1 (L1). In some embodiments of method 3000, the motion vector of the bidirectional prediction motion candidate is set to (0, 0). In some embodiments of method 3000, the motion vector of the bidirectional prediction motion candidate is set to (-m×W, 0), where m and W are integers.

[0689] In some embodiments of the method 3000, the motion vector of the bi-directional prediction motion candidate is set to (0, -q×H), where q and H are integers. In some embodiments of the method 3000, the motion vector of the bi-directional prediction motion candidate is set to (-m×W, -q×H), where m, q, H and W are integers. In some embodiments of the method 3000, W is the allowed width of a prediction unit (PU), a codec unit (CU), or a transform unit (TU) associated with a video block of the video unit. In some embodiments of the method 3000, H is the allowed height of a prediction unit (PU), a codec unit (CU), or a transform unit (TU) associated with a video block of the video unit. In some embodiments of the method 3000, the reference index of the bi-directional prediction motion candidate is set to a value less than the number of reference pictures in at least one prediction direction. In some embodiments of the method 3000, at least one prediction direction of the bi-directional prediction motion candidate is set to list 0 (L0) or list 1 (L1).

[0690] In some embodiments of the method 3000, the same weighting factor is used for two motion vectors associated with the bi-directional prediction motion candidate. In some embodiments of the method 3000, a generalized bi-directional prediction (GBi) index of the video unit is set to 0. In some embodiments of the method 3000, at least one of the K HMVP candidate motions includes motion information from a second video unit different from the video unit. In some embodiments of the method 3000, the video unit includes a first codec tree unit (CTU) row or a first picture, and wherein the second video unit includes a second CTU or a second picture.

[0691] In some embodiments of method 3000, for multiple P slices or multiple P pictures, at least one of the K HMVP candidate motions includes a motion vector set to (Mv(0,0), i), where i is a reference picture index in a range between 0 and the number of reference pictures in list 0 minus 1, and the video unit is a P slice from multiple P slices, or the video unit is a P picture from multiple P pictures. In some embodiments of method 3000, the K HMVP motion candidates are added to the HMVP table in ascending order up to i. In some embodiments of method 3000, the K HMVP motion candidates are added to the HMVP table in descending order starting from i. In some embodiments of method 3000, in response to the number of reference pictures in list 0 (L0) being less than K, one or more HMVP motion candidates are repeatedly added to the HMVP table until the HMVP table includes K candidates.

[0692] In some embodiments of the method 3000, in response to the number of reference pictures in list 0 (L0) being less than K, one or more available HMVP motion candidates are repeatedly added to the HMVP table until the HMVP table includes K candidates. In some embodiments of the method 3000, for multiple B slices or multiple B pictures, the K HMVP candidate motions include: K0 unidirectional prediction motion candidates from list X, K1 unidirectional prediction motion candidates from list Y (where Y is equal to (1-X)), and K2 bidirectional prediction motion candidates, where K0+K1+K2=K, and K0 / K1 / K2>=0, and where the video unit is a B slice from multiple B slices, or the video unit is a B picture from multiple B pictures.

[0693] In some embodiments of the method 3000, for multiple B slices or multiple B pictures, the K HMVP motion candidates include only bidirectional prediction motion candidates, and wherein the video unit is a B slice from multiple B slices, or the video unit is a B picture from multiple B pictures. In some embodiments of the method 3000, for multiple B slices or multiple B pictures, the K HMVP motion candidates include unidirectional prediction motion candidates interlaced with bidirectional prediction candidates, and wherein the video unit is a B slice from multiple B slices, or the video unit is a B picture from multiple B pictures. In some embodiments of the method 3000, for multiple B slices or multiple B pictures, the K HMVP motion candidates include unidirectional prediction motion candidates followed by bidirectional prediction motion candidates, and wherein the video unit is a B slice from multiple B slices, or the video unit is a B picture from multiple B pictures.

[0694] In some embodiments of the method 3000, in response to K being equal to N, a counter of available HMVP motion candidates in the HMVP table is excluded from the bitstream representation. In some embodiments of the method 3000, at least one HMVP motion candidate from the K HMVP candidate motions references a reference picture having a reference index equal to a particular value. In some embodiments of the method 3000, the particular value is zero. In some embodiments of the method 3000, the K HMVP candidate motions for the video unit are adaptively changed to obtain a second set of K HMVP motion candidates for a second video unit of the video, wherein the second video unit is different from the video unit.

[0695] Fig.31 is a flowchart of an example method 3100 for video processing. The method 3100 includes: adding 3102 a history-based motion vector prediction (HMVP) table of a video unit from a first set of K default HMVP motion candidates, wherein the first set of K HMVP motion candidates for the video unit is adaptively changed from a second set of K HMVP motion candidates included in the HMVP table for a second video unit of the video, wherein the second video unit is different from the video unit, wherein the first set of K HMVP motion candidates and the second set of K HMVP motion candidates have default values, wherein the size of the HMVP table is N, wherein N and K are positive integers, and wherein K≤N; and performing 3104 conversion between a video unit of the video and a bitstream representation of the video, wherein the bitstream representation includes a syntax element indicating the first set of K HMVP motion candidates added to the HMVP table.

[0696] In some embodiments of method 3000 and / or 3100, the video unit comprises a first codec tree unit (CTU) row, and wherein the second video unit comprises a second CTU row. In some embodiments of method 3000 and / or 3100, the video unit comprises a first slice, and wherein the second video unit comprises a second slice.

[0697] In some embodiments of the method 3000, the K HMVP motion candidates for the HMVP table are different from the K HMVP motion candidates for another HMVP table. In some embodiments of the method 3000, the selection of the K HMVP motion candidates is based on the HMVP table for the codec mode associated with the video block of the video unit. In some embodiments of the method 3000, the K HMVP motion candidates for the video block encoded using the inter-frame codec mode are different from the K HMVP motion candidates for the video block encoded using the intra-frame block copy (IBC) mode, and the video block belongs to the video unit. In some embodiments of the method 3000, the method also includes: before performing the conversion, determining that N HMVP motion candidates have been filled in the HMVP table; and based on the determination, adding another motion candidate to the HMVP table according to a rule. In some embodiments of the method 3000, the rule specifies that another motion candidate is added to the HMVP table by performing a comparison operation, in which the other motion candidate is compared with each candidate in the HMVP table in ascending or descending order of the index of the N HMVP motion candidates. In some embodiments of method 3000, when the other motion candidate matches the motion candidate in the HMVP table, performing the comparison operation is terminated. In some embodiments of method 3000, performing the comparison operation includes upon determining that the other motion candidate is the same as the motion candidate in the HMVP table: removing the motion candidate from the HMVP table and adding the other motion candidate to the last entry in the HMVP table.

[0698] In some embodiments of method 3000, for all K HMVP motion candidates in the HMVP table, after determining whether another motion candidate matches any motion candidate in the HMVP table, performing the comparison operation is terminated. In some embodiments of method 3000, performing the comparison operation includes, when determining that another motion candidate is the same as all M motion candidates in the HMVP table: removing M motion candidates from the HMVP table, and repeatedly adding another motion candidate to the HMVP table until the HMVP table is full. In some embodiments of method 3000 and / or 3100, the K HMVP motion candidates added to the HMVP table are signaled in a video parameter set (VPS), a decoder parameter set (DPS), a sequence parameter set (SPS), an adaptation parameter set (APS), a picture parameter set (PPS), a picture header, a slice header, or a slice group header. In some embodiments of method 3000 and / or 3100, multiple groups of K HMVP motion candidates are predefined, and the bitstream representation includes indices of K HMVP motion candidates in multiple groups of default K HMVP motion candidates for video units.

[0699] In some embodiments of the method 3000 and / or 3100, multiple groups of K HMVP motion candidates are predefined, the bitstream representation includes an index of a codec mode for a video block of a video unit, and the index of the codec mode indicates that K HMVP motion candidates in multiple groups of default K HMVP motion candidates are used for the video unit. In some embodiments of the method 3000 and / or 3100, the video unit includes a picture, a slice, a slice group, a slice, a codec tree unit (CTU) row.

[0700] In some embodiments of method 3000, the HMVP table is updated by adding an additional HMVP motion candidate to the HMVP table before all existing HMVP motion candidates in the HMVP table. In some embodiments of method 3000, the entry index of the HMVP table with the smallest value indicates the latest HMVP motion candidate in the HMVP table, and the additional HMVP motion candidate is added before the latest HMVP motion candidate added to the HMVP table. In some embodiments of method 3000, the method further includes determining not to add the additional HMVP motion candidate to the HMVP table when determining that the additional HMVP motion candidate is the same as or similar to at least one HMVP motion candidate in the HMVP table.

[0701] In some embodiments of method 3000, upon determining that the additional HMVP motion candidate is identical or similar to at least one HMVP motion candidate in the HMVP table, at least one HMVP motion candidate is removed from the HMVP table. In some embodiments of method 3000, determining that the additional HMVP motion candidate is identical or similar to the at least one HMVP motion candidate is determined by determining that a difference between motion vectors corresponding to the additional HMVP motion candidate and the at least one HMVP motion candidate is less than a threshold.

[0702] In some embodiments of the method 3000 and / or 3100, the video block of the video unit is encoded and decoded using an inter-frame coding mode or an intra-frame block copy (IBC) mode. In some embodiments of the method 3000 and / or 3100, K HMVP motion candidates of the HMVP table are selected from multiple groups of K HMVP motion candidates, and the K HMVP motion candidates of the HMVP table are selected based on the coding mode of the video block of the video unit. In some embodiments of the method 3000 and / or 3100, the coding mode includes an inter-frame coding mode or an intra-frame block copy (IBC) mode. In some embodiments of the method 3000 and / or 3100, the default value also includes a default value for an intra-frame prediction mode. In some embodiments of the method 3000, the reset also includes marking an existing entry in the HMVP table as unavailable or deleting an existing entry before adding the K HMVP motion candidates. In some embodiments of the method 3000 and / or 3100, the execution of the conversion includes an encoding operation that generates a bitstream representation from the video block of the video unit. In some embodiments of methods 3000 and / or 3100, the performance of the conversion includes a decoding operation to generate a video block of the video unit from the bitstream representation.

[0703] In summary, for the purpose of illustration, specific embodiments of the disclosed technology are described herein, but various modifications may be made without departing from the scope of the present invention. Therefore, the disclosed technology is not limited except as set forth in the appended claims.

[0704] The disclosed and other solutions, examples, embodiments, modules and functional operations described in this document can be implemented in digital electronic circuits, or computer software, firmware or hardware, including the structures disclosed in this document and their structural equivalents, or a combination of one or more thereof. The implementation of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-volatile computer-readable medium for execution by a data processing device or to control the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a storage device, a material composition that affects a machine-readable propagation signal, or a combination of one or more thereof. The term "data processing unit" or "data processing device" includes all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or a multiprocessor or a computer group. In addition to hardware, the device may also include code that creates an execution environment for a computer program, for example, code constituting a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof. A propagated signal is an artificially generated signal, for example, a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0705] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language (including compiled or interpreted languages) and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program may be deployed for execution on one or more computers that are located at one site or distributed across multiple sites and interconnected by a communications network.

[0706] The processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuits, and the apparatus may also be implemented as special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits).

[0707] For example, processors suitable for executing computer programs include general and special purpose microprocessors, and any one or more of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor that executes instructions and one or more storage devices that store instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to one or more mass storage devices to receive data from them or transfer data to one or more mass storage devices, or both. However, a computer does not necessarily have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable hard disks; magneto-optical disks; and CD ROM and DVD ROM disks. The processor and memory may be supplemented by, or incorporated into, dedicated logic circuits.

[0708] Although this patent document contains many details, they should not be interpreted as limitations on any subject matter or the scope of the claims, but rather as descriptions of features of specific embodiments of specific technologies. Certain features described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various functions described in the context of a single embodiment can also be implemented separately in multiple embodiments, or in any suitable subcombination. In addition, although the above-mentioned features may be described as working in certain combinations, or even initially required to be so, in some cases, one or more features in the claim combination can be removed from the combination, and the combination of claims can be directed to subcombinations or variations of subcombinations.

[0709] Likewise, although operations are described in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order or order shown, or that all illustrated operations be performed, in order to achieve the desired results. In addition, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0710] Only a few implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A video processing method, comprising: for conversion between a video unit of a video and a bitstream representation of the video, resetting a history-based motion vector prediction (HMVP) table of size N by adding K HMVP motion candidates to the HMVP table, The K HMVP candidate motion candidates have default values, Where N and K are positive integers, and where K ≤ N; and performing conversion between said video unit and said bitstream representation using said HMVP table resulting from said resetting, adding at least one motion candidate having a default value to the HMVP table a plurality of times, wherein the first m candidates in the HMVP table are different from each other, and each of the last (Km) candidates in the HMVP table is the same as one of the first m candidates; or The last (Km) candidates in the HMVP table are different from each other, wherein each of the first m candidates in the HMVP table is the same as one of the last (Km) candidates; or The first m candidates in the HMVP table are different from each other, wherein the last (Km) candidates in the HMVP table include one or more copies of the first m candidates, Where m <K。 2 . The method of claim 1 , wherein each of the K HMVP candidate motions is the same. The method of claim 1 , wherein at least two of the K HMVP candidate motions are different. The method of claim 1 , wherein at least one of the K HMVP motion candidates is a uni-directional prediction motion candidate.

5. The method of claim 4, wherein the video unit is a P slice or a P picture, and wherein: The prediction direction of the unidirectional prediction motion candidate is set to list 0 (L0).

6. The method of claim 4, wherein the video unit is a B slice or a B picture, and wherein: The prediction direction of the unidirectional prediction motion candidate is set to list 0 (L0) or list 1 (L1). The method of claim 4 , wherein the motion vector of the uni-directional prediction motion candidate is set to (0, 0). 8 . The method of claim 4 , wherein the motion vector of the uni-directional prediction motion candidate is set to (−m×W, 0), where m and W are integers. 9 . The method of claim 4 , wherein the motion vector of the uni-directional prediction motion candidate is set to (0, −q×H), where q and H are integers.

10. The method of claim 4, wherein the motion vector of the uni-directional prediction motion candidate is set to (-m×W, -q×H), where m, q, H, and W are integers.

11. The method of claim 8 or 10, wherein W is an allowed width of a prediction unit (PU), a coding unit (CU), or a transform unit (TU) of a video block of the video unit.

12. The method of claim 9 or 10, wherein H is an allowed height of a prediction unit (PU), a coding unit (CU), or a transform unit (TU) of a video block of the video unit. 13 . The method of claim 4 , wherein the reference index of the uni-directional prediction motion candidate is set to a value smaller than the number of reference pictures in a prediction direction. 14 . The method of claim 13 , wherein the prediction direction of the uni-directional prediction motion candidate is set to list 0 ( L0 ) or list 1 ( L1 ).

15. The method of claim 4, wherein a reference picture is set to a current picture associated with the video unit.

16. The method of claim 4, wherein a syntax element indicating that an intra block copy (IBC) mode is enabled for a video block of the video unit is set to 1. 17 . The method of claim 1 , wherein at least one of the K HMVP motion candidates is a bi-predictive motion candidate.

18. The method of claim 17, wherein the video unit is a P slice or a picture, and wherein: Each prediction direction of the bi-predictive motion candidate is set to list 0 (L0).

19. The method of claim 17, wherein the video unit is a B slice or a B picture, and wherein: At least one prediction direction of the bi-prediction candidate motion is set to list 0 (L0) or list 1 (L1).

20. The method of claim 17, wherein the motion vector of the bi-predictive motion candidate is set to (0, 0).

21. The method of claim 17, wherein the motion vector of the bi-predictive motion candidate is set to (-m×W, 0), where m and W are integers.

22. The method of claim 17, wherein the motion vector of the bi-predictive motion candidate is set to (0, -q x H), where q and H are integers.

23. The method of claim 17, wherein the motion vector of the bi-directional prediction motion candidate is set to (-m×W, -q×H), where m, q, H, and W are integers.

24. The method of claim 21 or 23, wherein W is an allowed width of a prediction unit (PU), a coding unit (CU), or a transform unit (TU) associated with a video block of the video unit.

25. The method of claim 22 or 23, wherein H is an allowed height of a prediction unit (PU), a coding unit (CU), or a transform unit (TU) associated with a video block of the video unit.

26. The method of claim 17, wherein the reference index of the bi-predictive motion candidate is set to a value smaller than the number of reference pictures in at least one prediction direction. 27 . The method of claim 26 , wherein the at least one prediction direction of the bi-predictive motion candidate is set to list 0 ( L0 ) or list 1 ( L1 ).

28. The method of claim 17, wherein the same weighting factor is used for two motion vectors associated with the bi-predictive motion candidate.

29. The method of claim 17, wherein a generalized bi-directional prediction (GBi) index of the video unit is set to 0.

30. The method of claim 1, wherein at least one of the K HMVP candidate motions comprises motion information from a second video unit different from the video unit.

31. The method of claim 30, wherein the video unit comprises a first codec tree unit (CTU) row or a first picture, and wherein, The second video unit includes a second CTU or a second picture.

32. The method according to claim 1, in, For multiple P slices or multiple P pictures, at least one of the K HMVP candidate motions includes a motion vector set to (Mv(0,0), i), where i is a reference picture index in the range between 0 and the number of reference pictures in list 0 minus 1, and The video unit is a P slice from the plurality of P slices, or the video unit is a P picture from the plurality of P pictures.

33. The method of claim 32, wherein the K HMVP motion candidates are added to the HMVP table in ascending order up to i.

34. The method of claim 32, wherein the K HMVP motion candidates are added to the HMVP table in descending order starting from i.

35. The method of claim 32, wherein: In response to the number of reference pictures in list 0 (L0) being less than K, one or more HMVP motion candidates are repeatedly added to the HMVP table until the HMVP table includes K candidates.

36. The method of claim 1, wherein: In response to the number of reference pictures in list 0 (L0) being less than K, one or more available HMVP motion candidates are repeatedly added to the HMVP table until the HMVP table includes K candidates.

37. The method of claim 36, wherein: For multiple B slices or multiple B pictures, the K HMVP candidate motions include: K0 unidirectional prediction motion candidates from list X, K1 unidirectional prediction motion candidates from list Y, where Y is equal to (1-X), and K2 bi-directional prediction motion candidates, where K0+K1+K2=K, and K0 / K1 / K2>=0, and The video unit is a B slice from the multiple B slices, or the video unit is a B picture from the multiple B pictures.

38. The method according to claim 36, in, For multiple B slices or multiple B pictures, the K HMVP motion candidates include only bidirectional prediction motion candidates, and The video unit is a B slice from the multiple B slices, or the video unit is a B picture from the multiple B pictures.

39. The method according to claim 36, in, For multiple B slices or multiple B pictures, the K HMVP motion candidates include unidirectional prediction motion candidates interlaced with bidirectional prediction motion candidates, and The video unit is a B slice from the multiple B slices, or the video unit is a B picture from the multiple B pictures.

40. The method according to claim 36, in, For multiple B slices or multiple B pictures, the K HMVP motion candidates include a unidirectional prediction motion candidate followed by a bidirectional prediction motion candidate, and The video unit is a B slice from the multiple B slices, or the video unit is a B picture from the multiple B pictures.

41. The method of claim 1, wherein in response to K being equal to N, a counter of available HMVP motion candidates in the HMVP table is excluded from the bitstream representation.

42. The method of claim 1, wherein: At least one HMVP motion candidate from among the K HMVP motion candidates references a reference picture having a reference index equal to a specific value. The method of claim 42 , wherein the specific value is zero.

44. The method according to claim 1, wherein the K HMVP motion candidates for the video unit are adaptively changed to obtain a second set of K HMVP motion candidates for a second video unit of the video, The second video unit is different from the video unit.

45. The method of claim 44, wherein the video unit comprises a first codec tree unit (CTU) row, and wherein, The second video unit includes a second CTU row.

46. ​​The method of claim 44, wherein the video unit comprises a first slice, and wherein: The second video unit includes a second slice.

47. The method of claim 1, wherein the K HMVP motion candidates for the HMVP table are different from K HMVP motion candidates for another HMVP table.

48. The method of claim 47, wherein selection of the K HMVP motion candidates is based on the HMVP table for a codec mode associated with a video block of the video unit.

49. The method according to claim 47, in, The K HMVP motion candidates for a video block encoded using an inter-coding mode are different from the K HMVP motion candidates for a video block encoded using an intra-block copy (IBC) mode, and Wherein the video block belongs to the video unit.

50. The method of claim 1, further comprising: Before performing the converting, determining that N HMVP motion candidates have been filled in the HMVP table; as well as Based on the determination, another motion candidate is added to the HMVP table according to a rule.

51. The method of claim 50, wherein the rule specifies that the another motion candidate is added to the HMVP table by performing a comparison operation in which the another motion candidate is compared with each candidate in the HMVP table in ascending or descending order of the indices of the N HMVP motion candidates.

52. The method of claim 51, wherein: When the another motion candidate matches the motion candidate in the HMVP table, the performing of the comparison operation is terminated.

53. The method of claim 51, wherein: The performing the comparison operation includes, when it is determined that the other motion candidate is the same as the motion candidate in the HMVP table: remove the motion candidate from the HMVP table, and The further motion candidate is added to the last entry in the HMVP table.

54. The method of claim 51, wherein: For all of the K HMVP motion candidates in the HMVP table, after determining whether the another motion candidate matches any motion candidate in the HMVP table, the performing of the comparison operation is terminated.

55. The method of claim 54, wherein: The performing the comparison operation includes, when it is determined that the another motion candidate is the same as all the M motion candidates in the HMVP table: remove the M motion candidates from the HMVP table, and The another motion candidate is repeatedly added to the HMVP table until the HMVP table is full.

56. The method of claim 1, wherein: The K HMVP motion candidates added to the HMVP table are signaled in a video parameter set (VPS), a decoder parameter set (DPS), a sequence parameter set (SPS), an adaptation parameter set (APS), a picture parameter set (PPS), a picture header, a slice header, or a slice group header.

57. The method according to claim 1, where multiple groups of K HMVP motion candidates are predefined, and Wherein the bitstream representation comprises indices of the K HMVP motion candidates in the plurality of sets of K HMVP motion candidates for the video unit.

58. The method according to claim 1, There are multiple groups of K HMVP motion candidates predefined, wherein the bitstream representation comprises an index of a codec mode for a video block of the video unit, and The index of the coding mode indicates that the K HMVP motion candidates in the multiple groups of K HMVP motion candidates are to be used for the video unit.

59. The method of claim 57 or 58, wherein the video unit comprises a picture, a slice, a slice group, a slice, a codec tree unit (CTU) row.

60. The method of claim 1, wherein the HMVP table is updated by: Additional HMVP motion candidates are added to the HMVP table ahead of all existing HMVP motion candidates in the HMVP table.

61. The method according to claim 60, in, The entry index of the HMVP table with the smallest value indicates the latest HMVP motion candidate in the HMVP table, and Wherein, the additional HMVP motion candidate is added before the latest HMVP motion candidate added to the HMVP table.

62. The method of claim 1, further comprising: When it is determined that the additional HMVP motion candidate is the same as or similar to at least one HMVP motion candidate in the HMVP table, it is determined not to add the additional HMVP motion candidate to the HMVP table.

63. The method of claim 62, wherein: When it is determined that the additional HMVP motion candidate is the same as or similar to at least one HMVP motion candidate in the HMVP table, the at least one HMVP motion candidate is removed from the HMVP table.

64. The method according to any one of claims 62 to 63, wherein: The additional HMVP motion candidate is determined to be the same as or similar to the at least one HMVP motion candidate by determining that a difference between motion vectors corresponding to the additional HMVP motion candidate and the at least one HMVP motion candidate is less than a threshold value.

65. The method of claim 1, wherein video blocks of the video unit are encoded using an inter-coding mode or an intra-block copy (IBC) mode.

66. The method according to claim 1, in, selecting the K HMVP motion candidates for the HMVP table from a plurality of groups of K HMVP motion candidates, and The K HMVP motion candidates for the HMVP table are selected based on a coding mode of a video block of the video unit.

67. The method of claim 66, wherein the codec mode comprises an inter-codec mode or an intra-block copy (IBC) mode.

68. The method of claim 1, wherein the default value further comprises a default value for an intra-prediction mode.

69. The method of claim 1, wherein the resetting further comprises marking an existing entry in the HMVP table as unavailable or deleting the existing entry before adding the K HMVP motion candidates.

70. The method of claim 1, wherein performance of the conversion comprises an encoding operation that generates the bitstream representation from video blocks of the video unit.

71. The method of claim 1, wherein performance of the conversion comprises a decoding operation that generates a video block of the video unit from the bitstream representation.

72. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein: When the processor executes the instructions, the instructions cause the processor to implement the method of any one of claims 1 to 71.

73. A computer program product stored on a non-transitory computer readable medium, the computer program product comprising program code executed by a processor to implement the method of any one of claims 1 to 71.

Images