Partial / Full Pruning When Adding HMVP Candidates to Merge / AMVP

Abstract

A video processing method, apparatus and program with higher coding efficiency are provided that implement a motion vector prediction technique based on a merged list of stored motion vectors, where at least one motion candidate is stored. A video processing method includes maintaining a set of tables, each table including a motion candidate, each motion candidate associated with corresponding motion information. The method further includes updating a motion candidate list based on the motion candidates in the one or more tables using a motion candidate pruning process, and converting between a first video block and a bitstream representation of a video including the first video block using the constructed candidate list.

Description

Technical Field

[0001] Cross - reference to Related Applications Book This application timely claims the priority and benefits of International Patent Application No. PCT / CN2018 / 093663 filed on June 29, 2018, International Patent Application No. PCT / CN2018 / 105193 filed on September 12, 2018, International Patent Application No. PCT / CN2018 / 117627 filed on November 27, 2018, and International Patent Application No. PCT / CN2019 / 071214 filed on January 10, 2019. This is a divisional application of Japanese Patent Application No. 2021-523110, based on the international patent application PCT / IB2019 / 055582 filed on July 1, 2019. 。 All of the above patent applications They are incorporated herein by reference as part of the disclosure of this application.

[0002] This patent specification relates to video encoding and decoding technologies, devices, and systems.

Background Art

[0003] Despite the progress of video compression, digital video still occupies the largest amount of bandwidth usage in the Internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demand for the use of digital video is expected to continue to grow.

Summary of the Invention

[0004] This specification discloses a method, system, and device for encoding and decoding digital video using a merge list of motion vectors.

[0005] In one exemplary embodiment, a video processing method includes maintaining a set of tables, each table including motion candidates, each motion candidate being associated with corresponding motion information; using a pruning process on the motion candidates to update a candidate list based on the motion candidates in one or more tables; and performing a conversion between a first video block and a bit - stream representation of a video including the first video block using the constructed candidate list.

[0006] In another embodiment, a video processing method is provided that includes receiving a bitstream representation of video including a first video block; applying a pruning process to motion candidates in one or more tables in order to update a candidate list, wherein each table includes motion candidates and each motion candidate is associated with corresponding motion information; and performing a conversion between the bitstream representation and the first video block using the configured candidate list.

[0007] In yet another representative embodiment, the various techniques described herein may be implemented as a computer program product stored on a non-temporary computer-readable medium. This computer program product includes program code for performing the methods described herein.

[0008] In yet another representative embodiment, a video encoding or decoding device may implement the method described herein.

[0009] Details of one or more implementation configurations are described in the attached files, drawings, and the following description. Other features will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0010] [Figure 1] This block diagram shows examples of video encoder implementations. [Figure 2] This shows the division of macroblocks in the H.264 video coding standard. [Figure 3] This example shows how to split a coding block (CB) into prediction blocks (PU). [Figure 4] This diagram illustrates an exemplary implementation for subdividing a CTB into CBs and transformation blocks (TBs). Solid lines indicate CB boundaries, dotted lines indicate TB boundaries, and the diagram includes an example of a CTB with its subdivision, as well as the corresponding quadtree. [Figure 5] This shows an example of a Quad Tree Binary Tree (QTBT) structure for partitioning video data. [Figure 6] An example of dividing video blocks is shown. [Figure 7] An example of a quadtree partition is shown. [Figure 8] An example of a tree-type signaling notification is shown. [Figure 9] This shows an example of the derivation process for constructing a merge candidate list. [Figure 10] Examples of spatial merge candidate locations are shown. [Figure 11] Examples of candidate pairs considered for redundancy checks of spatial merge candidates are shown. [Figure 12] Examples of the second PU location for N×2N and 2N×N partitions are shown. [Figure 13] This shows the scaling of motion vectors for temporal merge candidates. [Figure 14] This shows the candidate positions for temporal merge candidates and the corresponding pictures at those positions. [Figure 15] An example of a bidirectional predictive merge candidate is shown. [Figure 16] This shows an example of the process for deriving candidate motion vector predictions. [Figure 17] An example of scaling motion vectors for spatial motion vector candidates is shown. [Figure 18] An example of ATMVP (Alternative Temporal Motion Vector Prediction) for predicting the motion of a CU is shown. [Figure 19] An example of identifying source blocks and source pictures is shown in the illustration. [Figure 20] An example of a CU with four subblocks and neighboring blocks is shown. [Figure 21] Here is an example of bilateral matching. [Figure 22] An example of template matching is shown. [Figure 23]An example of one motion estimation (ME: Motion Estimation) in FRUC (Frame Rate Up Conversion) is shown. [Figure 24] An example of DMVR based on bilateral template matching is shown. [Figure 25] An example of spatially neighboring blocks used to derive spatial merge candidates is shown. [Figure 26] An example of a method for selecting representative positions for look-up table update is shown. [Figure 27A] An example of updating a look-up table with a new set of motion information is shown. [Figure 27B] An example of updating a look-up table with a new set of motion information is shown. [Figure 28] It is a block diagram showing an example of a hardware platform for implementing the decoding of visual media or the encoding technology of visual media described in this specification. [Figure 29] It is a flowchart showing an example of a video processing method. [Figure 30] It is a flowchart showing another exemplary video processing method. [Figure 31] An example of a decoding flowchart by the proposed HMVP method is shown. [Figure 32] An example of table update using the proposed HMVP method is shown. [Figure 33A] An example of a LUT update method based on redundancy removal (removing one redundant motion candidate) is shown. [Figure 33B] An example of a LUT update method based on redundancy removal (removing one redundant motion candidate) is shown. [Figure 34A] An example of a LUT update method based on redundancy removal (removing multiple redundant motion candidates) is shown. [Figure 34B] An example of a LUT update method based on redundancy removal (removing multiple redundant motion candidates) is shown.

Mode for Carrying Out the Invention

[0011] To improve video compression rates, researchers are constantly seeking new technologies for encoding video.

[0012] 1. Introduction

[0013] This specification relates to video coding techniques, specifically to the coding of motion information in video coding (e.g., merge mode, AMVP mode). It may be applied to existing video coding standards such as HEVC, or to the development of a new standard (Versatile Video Coding). The present invention is also applicable to future video coding standards or video codecs.

[0014] Simple explanation

[0015] Video coding standards have primarily evolved through the development of well-known ITU-T and ISO / IEC standards. ITU-T created H.261 and H.263, ISO / IEC created MPEG-1 and MPEG-4 Visual, and the two organizations jointly developed H.262 / MPEG-2 Video, H.264 / MPEG-4 AVC (Advanced Video Coding), and H.265 / HEVC standards. Since H.262, video coding standards have been based on a hybrid video coding structure that utilizes time prediction and transformation coding. An example of a typical HEVC encoder framework is shown in Figure 1.

[0016] 2.1 Partition Structure

[0017] 2.1.1 Partition Tree Structure in H.264 / AVC

[0018] In previous standards, the core of the coding layer consisted of macroblocks containing 16x16 blocks of luminance samples, and in the case of typical 4:2:0 color sampling, two corresponding 8x8 blocks of saturation samples.

[0019] The intra-encoded blocks use spatial prediction to leverage spatial correlations between pixels. Two partitions are defined: 16x16 and 4x4.

[0020] Intercoded blocks use temporal prediction instead of spatial prediction by estimating motion between pictures. Motion can be estimated independently for any 16x16 macroblock or its sub-macroblock partitions. 16x8, 8x16, 8x8, 8x4, 4x8, 4x4 (see Figure 2). Only one motion vector (MV) is allowed per sub-macroblock partition.

[0021] 2.1.2 Partition Tree Structure in HEVC

[0022] In HEVC, a CTU is partitioned into CUs using a quadtree structure called a coding tree to adapt to various local features. The decision of whether to code a picture region, using interpicture (temporal) or intrapicture (spatial) prediction, is made at the CU level. Each CU can be further partitioned into one, two, or four PUs depending on the PU partitioning type. Within a single PU, the same prediction process is applied, and relevant information is sent to the decoder on a PU-by-PU basis. After applying the prediction process based on the PU partitioning type to obtain residual blocks, the CU can be partitioned into Transform Units (TUs) based on another quadtree structure similar to the coding tree for the CU. One of the key features of the HEVC structure is that it has multiple partition concepts, including CUs, PUs, and TUs.

[0023] The following focuses on various features related to hybrid video coding using HEVC.

[0024] 1) Coding Tree Unit and Coding Tree Block (CTB) Structure. A similar structure in HEVC is the Coding Tree Unit (CTU), which has a size selected by the encoder and may be larger than a conventional macroblock. A CTU consists of a luminance CTB and a corresponding chrominance CTB and syntactic elements. The size L×L of the luminance CTB can be selected as L=16, 32, or 64 samples, with larger sizes generally enabling better compression. HEVC then supports dividing the CTB into smaller blocks using a tree structure and quadtree-like signaling.

[0025] 2) Coding Units (CUs) and Coding Blocks (CBs): The syntax of the quadtree of a CTU specifies the size and position of its luminance and chrominance CBs. The root of the quadtree is associated with the CTU. Thus, the size of the luminance CTB is the maximum size supported for the luminance CBs. Dividing a CTU into luminance CBs and chrominance CBs is signaled together. One luminance CB and usually two chrominance CBs, along with their associated syntax, form one coding unit (CU). A CTB may contain only one CU, or it may be divided to form multiple CUs, each CU having a division into its associated prediction units (PUs) and a tree of one transformation unit (TU).

[0026] 3) Prediction Units and Prediction Blocks (PB): The decision of whether to code the picture region using interpicture or intrapicture prediction is made at the CU level. The partitioning structure of the PU has its root at the CU level. Based on the basic prediction type determination, the sizes of the luminance and saturation CBs can then be further partitioned and predicted from the luminance and saturation prediction blocks (PBs). HEVC supports samples with variable PB sizes from 64×64 to 4×4. Figure 3 shows an example of an allowed PB for an M×M CU.

[0027] 4) TU and Transformation Blocks: Predicted residuals are coded using block transformations. The TU tree structure has its root at the CU level. This luminance CB residual may be identical to the luminance transformation block (TB), or it may be further divided into smaller luminance TBs. The same applies to saturation TBs. For square TB sizes of 4x4, 8x8, 16x16, and 32x32, integer basis functions similar to the integer basis functions of the discrete cosine transform (DCT) are specified. For the 4x4 transformation of the luminance intrapicture predictive residual, an integer transformation derived from the form of the discrete sine transform (DST) is specified as an alternative.

[0028] Figure 4 shows an example of subdividing a CTB into CBs [and transformation blocks (TBs)]. Solid lines indicate CB boundaries, and dotted lines indicate TB boundaries. (a) CTB and its subdivision (b) corresponding quadtree.

[0029] 2.1.2.1 Splitting the tree structure into transformation blocks and units

[0030] In residual coding, a CB can be recursively partitioned into transformation blocks (TBs). This partitioning is signaled by the residual quadtree. As shown in Figure 4, only partitioning of square CBs and TBs is specified so that one block can be recursively partitioned into quadrants. For a given luminance CB of size M × M, a flag signals whether it can be partitioned into four blocks of size M / 2 × M / 2. If further partitioning is possible, each quadrant is assigned a flag indicating whether it can be partitioned into four quadrants, as signaled by the maximum depth of the residual quadtree shown in the SPS. The resulting leaf node blocks of the residual quadtree are transformation blocks that are further processed by transformation coding. The encoder indicates the maximum and minimum luminance TB sizes it will use. If the CB size is greater than the maximum TB size, partitioning is implicitly performed. If partitioning results in a luminance TB size smaller than the indicated minimum, partitioning is not implicitly performed. Except when the luminance TB size is 4x4, the saturation TB size is half the luminance TB size in each dimension, in which case one 4x4 saturation TB is used for a region covered by four 4x4 luminance TBs. For intrapicture prediction CUs, decoded samples of the nearest neighbor TB (inside or outside the CB) are used as reference data for intrapicture prediction.

[0031] In contrast to conventional standards, the HEVC design allows a single TB to span multiple PBs for interpicture prediction CUs, maximizing the potential coding efficiency benefits of partitioning TBs in a quadtree structure.

[0032] 2.1.2.2 Parent-child nodes

[0033] The CTB is divided based on a quadtree structure, and its nodes are coding units. Multiple nodes in the quadtree structure include leaf nodes and non-leaf nodes. Leaf nodes have no child nodes in the tree structure (i.e., leaf nodes are not further divided). Non-leaf nodes include the root node of the tree structure. The root node corresponds to the first video block of the video data (e.g., the CTB). In each non-root node of the multiple nodes, each non-root node corresponds to a video block that is a subblock of the video block corresponding to the parent node in the tree structure of each non-root node. Each non-leaf node of the multiple non-leaf nodes has one or more child nodes in the tree structure.

[0034] 2.1.3 Quantree + Binary Tree Block Structure with Larger CTU in JEM

[0035] In 2015, VCEG and MPEG jointly established JVET (Joint Video Exploration Team) to explore future video coding technologies beyond HEVC. Since then, many new methods have been adopted by JVET and incorporated into reference software called JEM (Joint Exploration Mode).

[0036] 2.1.3.1 Divided structure of QTBT block

[0037] Unlike HEVC, the QTBT structure eliminates the concept of multiple partition types. That is, it removes the separation of the CU, PU, ​​and TU concepts and improves the flexibility of the CU partition shape. In the QTBT block structure, the CU can be either square or rectangular. As shown in Figure 5, first, the coding tree unit (CTU) is divided into a quadtree structure. The leaf nodes of the quadtree are further divided into a binary tree structure. There are two types of binary tree divisions: symmetric horizontal division and symmetric vertical division. The leaf nodes of the binary tree are called coding units (CU), and this segmentation is used for prediction and transformation processing without further division. This means that in the QTBT coding block structure, the CU, PU, ​​and TU have the same block size. In JEM, a CU often consists of coding blocks (CBs) of different color components; for example, in the case of P and B slices in a 4:2:0 saturation format, one CU contains one luminance CB and two saturation CBs. Alternatively, a CU often consists of a single component CB; for example, in the case of an I slice, one CU contains either only one luminance CB or only two saturation CBs.

[0038] The following parameters are defined for the QTBT partitioning scheme. -CTU size: The size of the root node of a single quad tree, the same concept as HEVC. -MinQTSize: Minimum allowable leaf node size for a quad tree -MaxBTSize: Maximum allowable root node size of a binary tree -MaxBTDepth: Maximum allowed depth of the binary tree -MinBTSize: Minimum allowable size of a leaf node in a binary tree

[0039] In one example of a QTBT partitioning structure, the size of the CTU is set to a 128x128 luminance sample with two corresponding 64x64 block chroma samples, MinQTSize is set to 16x16, MaxBTSize is set to 64x64, MinBTSize (for both width and height) is set to 4x4, and MaxBTDepth is set to 4. The quadtree partitioning is first applied to the CTU, generating leaf nodes of the quadtree. Leaf nodes of the quadtree can have sizes ranging from 16x16 (i.e., MinQTSize) to 128x128 (i.e., CTU size). If a leaf quadtree node is 128x128, it is not further partitioned by a binary tree because its size exceeds MaxBTSize (i.e., 64x64). Otherwise, the leaf quadtree node may be further partitioned by a binary tree. Therefore, the leaf node of this quadtree is also the root node of the binary tree, and the depth of that binary tree is 0. If the depth of the binary tree reaches MaxBTDepth (i.e., 4), no further partitioning is considered. If the width of a node in the binary tree is equal to MinBTSize (i.e., 4), no further horizontal partitioning is considered. Similarly, if the height of a node in the binary tree is equal to MinBTSize, no further vertical partitioning is considered. The leaf nodes of the binary tree are further processed by prediction and transformation processes without further partitioning. In JEM, the maximum CTU size is 256 × 256 luminance samples.

[0040] Figure 5 (left) shows an example of block partitioning using QTBT, and Figure 5 (right) shows the corresponding tree representation. Solid lines represent partitioning of a quadtree, and dotted lines represent partitioning of a binary tree. At each partition (i.e., non-leaf) node of a binary tree, one flag is signaled to indicate which partition type (i.e., horizontal or vertical) is used. Here, 0 represents a horizontal partition, and 1 represents a vertical partition. In the case of partitioning a quadtree, there is no need to indicate the partition type because the partition always divides the block horizontally and vertically, producing four subblocks of equal size.

[0041] Furthermore, the QTBT method supports the ability for luminance and chroma to have separate QTBT structures. Currently, for P and B slices, the luminance and chroma CTBs in one CTU share the same QTBT structure. However, for I slices, the luminance CTB is divided into CUs by the QTBT structure, and the chroma CTB is divided into chroma CUs by a separate QTBT structure. This means that one CU in one I slice consists of one coding block for one luminance component or one coding block for two chroma components, and one CU in one P or B slice consists of coding blocks for all three color components.

[0042] In HEVC, interpretation for small blocks is limited to reduce memory access for motion compensation, resulting in bidirectional prediction not being supported for 4x8 and 8x4 blocks, and interpretation not being supported for 4x4 blocks. In JEM's QTBT, these limitations are removed.

[0043] 2.1.4 VVC tertiary tree

[0044] In some implementations, tree types other than quad and binary trees are supported. In this implementation, as shown in parts (d) and (e) of Figure 6, two or more ternary (TT) partitions are introduced, i.e., horizontal and vertical central ternary trees.

[0045] Figure 6 shows (a) a quad tree partition, (b) a vertical binary tree partition, (c) a horizontal binary tree partition, (d) a vertical center-side trinity tree partition, and (e) a horizontal center-side trinity tree partition.

[0046] In some implementations, there are two levels of trees: a region tree (quadruple tree) and a prediction tree (binary or ternary tree). The CTU is first partitioned by the region tree (RT). The RT leaves may be further partitioned by prediction trees (PTs). The PT leaves may also be further partitioned by PTs until the maximum PT depth is reached. The PT leaves are the basic coding units. For convenience, we will also refer to them as CUs here. A single CU cannot be further partitioned. Both prediction and transformation are applied to CUs, as in JEM. The entire partition structure is called a "multitype tree".

[0047] 2.1.5 Partition structure of [8]

[0048] The tree structure used in this response is called the Multi-Tree Type (MTT), and it is a generalization of the QTBT. In the QTBT, as shown in Figure 5, the coding tree unit (CTU) is first divided into a quadtree structure. The leaf nodes of the quadtree are further divided into a binary tree structure.

[0049] The basic structure of MTT consists of two types of tree nodes. As shown in Figure 7, the region tree (RT) and prediction tree (PT) support nine types of partitions.

[0050] Figure 7 shows (a) quadrutree partitioning, (b) vertical binary partitioning, (c) horizontal binary partitioning, (d) vertical trinity partitioning, (e) horizontal trinity partitioning, (f) horizontal upward asymmetric binary partitioning, (g) horizontal downward asymmetric binary partitioning, (h) vertical left asymmetric binary partitioning, and (i) vertical right asymmetric binary partitioning.

[0051] A region tree can be recursively divided into square blocks so that one CTU becomes a leaf node of a 4x4 region tree. At each node in the region tree, the prediction tree can be formed from one of three tree types: binary tree (BT), ternary tree (TT), and asymmetric binary tree (ABT). In PT partitioning, it is forbidden for a branch of the prediction tree to have a quaternary tree partition. As in JEM, the luminance tree and chrominance tree are divided into I slices. The signaling methods for RT and PT are shown in Figure 8.

[0052] 2.2 Interpretation in HEVC / H.265

[0053] Each interpreted PU has motion parameters for one or two reference picture lists. The motion parameters include motion vectors and reference picture indices. The use of one of the two reference picture lists may be signaled using inter_pred_idc. The motion vectors may be explicitly coded as differences related to the predictors, and such a coding mode is called the AMVP mode.

[0054] If one CU is coded in skip mode, one PU is associated with this CU, with no significant residual coefficients, coded motion vector differences, or reference picture indices. Specify merge mode, which retrieves motion parameters for the current PU from neighboring PUs, including spatial and temporal candidates. Merge mode can be applied not only for skip mode but also to any interpredicted PU. An alternative to merge mode is explicit transmission of motion parameters, explicitly signaling each PU with motion vectors corresponding to each reference picture list and the reference picture indices for the use of the reference picture lists.

[0055] If the signaling indicates that one of the two reference picture lists should be used, a PU is generated from one block of samples. This is called a "single prediction." Single predictions are available for both P-slice and B-slice.

[0056] If the signaling indicates that both reference picture lists should be used, a PU is generated from two blocks of the samples. This is called "bidirectional prediction." Bidirectional prediction is only available for B-slice.

[0057] The following provides a detailed explanation of the interpretation modes specified in HEVC. First, we will explain the merge mode.

[0058] 2.2.1 Merge Mode

[0059] 2.2.1.1 Derivation of Candidate Merge Modes

[0060] When predicting a PU using merge mode, an index pointing to an entry in the merge candidate list is parsed from the bitstream and used to retrieve motion information. The structure of this list is defined in the HEVC standard and can be assembled based on the following sequence of steps. Step 1: Derivation of initial candidates Step 1.1: Derivation of spatial candidates Step 1.2: Check for spatial redundancy of candidates Step 1.3: Derivation of temporal candidates Step 2: Insert additional candidates Step 2.1: Creating bidirectional prediction candidates Step 2.2: Insertion of zero motion candidates

[0061] These steps are schematically shown in Figure 9. For spatial merge candidate derivation, up to four merge candidates are selected from candidates located at five different positions. For temporal merge candidate derivation, up to one merge candidate is selected from two candidates. The decoder assumes a certain number of candidates per PU, so if the number of candidates does not reach the maximum number of merge candidates (MaxNumMergeCand) signaled in the slice header, additional candidates are generated. Since the number of candidates is constant, the index of the best merge candidate is encoded using truncated unary binarization (TU). If the size of the CU is equal to 8, all PUs of the current CU share the same single merge candidate list as the merge candidate list for 2N × 2N prediction units.

[0062] The following describes in detail the actions associated with the steps described above.

[0063] 2.2.1.2 Derivation of Spatial Candidates

[0064] In deriving spatial merge candidates, up to four merge candidates are selected from the candidates located at the positions shown in Figure 10. The derivation order is A1, B1, B0, A0, B2. Position B2 is considered only if any of the PUs at positions A1, B1, B0, or A0 are unavailable (e.g., belonging to a different slice or tile) or are intra-encoded. After adding the candidate for position A1, the remaining candidates are added and subjected to a redundancy check, which ensures that candidates with the same motion information are removed from the list, thereby improving coding efficiency. To reduce computational complexity, the aforementioned redundancy check does not consider all possible candidate pairs. Instead, only pairs linked by arrows in Figure 11 are considered, and a candidate is added to the list only if the corresponding candidate used in the redundancy check does not have the same motion information. Another source of duplicate motion information is a "second PU" associated with a different partition than 2N×2N. As an example, Figure 12 shows the second PU for the N×2N and 2N×N cases, respectively. When the current PU is divided into N×2N, the candidate for position A1 is not considered in the list configuration. In fact, adding this candidate would lead to two prediction units with the same motion information, which is redundant if we want to have only one PU per coding unit. Similarly, when the current PU is divided into 2N×N, position B1 is not considered.

[0065] 2.2.1.3 Derivation of Temporal Candidates

[0066] In this step, only one candidate is added to the list. Specifically, in deriving this temporal merge candidate, a scaled motion vector is derived based on the co-located PU (PU) belonging to the picture that has the smallest POC difference between it and the current picture in a given list of reference pictures. In the slice header, the list of reference pictures used to derive the co-located PU is explicitly signaled. As shown by the dotted line in Figure 13, the scaled motion vector of the temporal merge candidate is obtained. This is scaled from the motion vector of the co-located PU using the POC distances tb and td. tb is defined as the POC difference between the current picture and its reference picture, and td is defined as the POC difference between the reference picture and the co-located picture of the co-located PU. The reference picture index of the temporal merge candidate is set to equal to zero. The practical implementation of this scaling process is described in the HEVC specification [1]. In the case of a B-slice, two motion vectors are obtained: one for reference picture list 0 and one for reference picture list 1. By combining these, a bidirectional predictive merge candidate is formed. Explanation of scaling of motion vectors for temporal merge candidates.

[0067] For PU(Y) at the same location belonging to the reference frame, the position of the temporal candidate is selected between candidate C0 and candidate C1, as shown in Figure 14. If the PU at position C0 is unavailable, intra-encoded, or outside the current CTU, position C1 is used. Otherwise, position C0 is used to derive the temporal merge candidate.

[0068] 2.2.1.4 Insertion of additional candidates

[0069] In addition to spatial-temporal merge candidates, there are two additional types of merge candidates: joined bidirectional predictive merge candidates and zero merge candidates. Joined bidirectional predictive merge candidates are generated using spatial-temporal merge candidates. Joined bidirectional predictive merge candidates are used only for B slices. Joined bidirectional predictive candidates are generated by combining the first reference picturelist motion parameter of the first candidate with the second reference picturelist motion parameter of another candidate. If these two tuples provide different motion hypotheses, these tuples form a new bidirectional predictive candidate. As an example, Figure 15 shows how to generate a joined bidirectional predictive merge candidate to be added to the final list (right) using two candidates in the original list (left) that have mvL0 and refIdxL0, or mvL1 and refIdxL1. Various rules exist for the combinations considered to generate these additional merge candidates.

[0070] The MaxNumMergeCand capacity is hit by inserting zero-movement candidates and filling in the remaining entries in the merge candidate list. These candidates have zero spatial displacement and a reference picture index that starts at zero and increments each time a new zero-movement candidate is added to the list. The number of reference frames used by these candidates is one for one-way predictions and two for two-way predictions, respectively. Finally, no redundancy checks are performed on these candidates.

[0071] 2.2.1.5 Motion estimation area for parallel processing

[0072] To speed up the coding process, motion estimation can be performed in parallel, thereby simultaneously deriving motion vectors for all prediction units within a given region. Since a single prediction unit cannot derive motion parameters from adjacent PUs until its associated motion estimation is complete, deriving merge candidates from spatial neighbors can interfere with parallel processing. To mitigate the trade-off between coding efficiency and processing latency, HEVC defines a Motion Estimation Region (MER), whose size is signaled in the picture parameter set using the "log2_parallel_merge_level_minus2" syntactic element. When a single MER is defined, merge candidates in the same region are marked as unavailable and therefore not considered in list construction. 7.3.2.3 Picture Parameter Set RBSP Syntax 7.3.2.3.1 General Picture Parameter Set RBSP Syntax

[0073] [Table 1]

[0074] log2_parallel_merge_level_minus2+2 specifies the value of the variable Log2ParMrgLevel, which is used in the derivation process of the luminance motion vector for the merge mode specified in Section 8.5.3.2.2.2 and the derivation process of the spatial merge candidate specified in Section 8.5.3.2.3. The value of log2_parallel_merge_level_minus2 must be within the range of 0 to CtbLog2SizeY-2. The variable Log2ParMrgLevel is derived as follows: Log2ParMrgLevel=log2_parallel_merge_level_minus2+2 (7-37) Note 3: The Log2ParMrgLevel value indicates the built-in ability to derive merge candidate lists in parallel. For example, if Log2ParMrgLevel is equal to 6, merge candidate lists for all prediction units (PUs) and coding units (CUs) contained in a 64x64 block can be derived in parallel.

[0075] 2.2.2 Motion Vector Prediction in AMVP Mode

[0076] Motion vector prediction utilizes the spatial-temporal correlation between motion vectors and neighboring PUs, using this for clear transmission of motion parameters. First, the availability of temporally neighboring PU locations on the left and upper sides is checked, redundant candidates are removed, and zero vectors are added to construct a motion vector candidate list with a constant length. The encoder can then select the best predictor from the candidate list and transmit the corresponding index indicating the selected candidate. Similar to signaling merge indices, the index of the best motion vector candidate is encoded using a shortened unary. In this case, the maximum value to be encoded is 2 (e.g., Figures 2-8). The following chapters describe the derivation process of motion vector prediction candidates in detail.

[0077] 2.2.2.1 Derivation of candidate motion vector predictions

[0078] Figure 16 summarizes the process for deriving candidate motion vectors.

[0079] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. To derive the spatial motion vector candidates, two motion vector candidates are ultimately derived based on the motion vectors of each PU at five different locations, as shown in Figure 11.

[0080] To derive a temporal motion vector candidate, select one motion vector candidate from two candidates derived based on two different identical positional arrangements. After creating the initial list of spatial-temporal candidates, remove any duplicate motion vector candidates from the list. If the number of possible candidates is greater than two, remove motion vector candidates from the list whose reference picture index in the associated reference picture list is greater than 1. If the number of spatial-temporal motion vector candidates is less than two, add an additional zero motion vector candidate to the list.

[0081] 2.2.2.2 Candidate spatial motion vectors

[0082] In deriving candidate spatial motion vectors, we consider up to two candidates that are at the same position as the motion merge from the five possible candidates derived from the PU at the position shown in Figure 11. The derivation order for the left side of the current PU is defined as A0, A1, scaled A0, scaled A1. The derivation order for the upper side of the current PU is defined as B0, B1, B2, scaled B0, scaled B1, scaled B2. Therefore, for each edge, there are four cases that can be used as candidate motion vectors: two cases where spatial scaling is not necessary and two cases where spatial scaling is necessary. The four different cases can be summarized as follows: • No spatial scaling -(1) Same reference picture list and same reference picture index (same POC) -(2) Different reference picture lists, and the same reference picture (same POC) • Spatial scaling -(3) Same reference picture list, but different reference pictures (different POCs) -(4) Different reference picture lists and different reference pictures (different POCs)

[0083] First, check for non-spatial scaling, then perform spatial scaling. Regardless of the reference picture list, consider spatial scaling if the POC differs between the reference picture of a neighboring PU and the reference picture of the current PU. If all PUs of the left-side candidate are unavailable or intra-encoded, scaling the upper motion vector helps in the parallel derivation of the left-side and upper MV candidates. Otherwise, spatial scaling is not allowed for the upper motion vector.

[0084] In spatial scaling, as shown in Figure 17, the motion vectors of neighboring PUs are scaled in the same way as in temporal scaling. The main difference is that the current PU's reference picture list and index are given as input, and the actual scaling process is the same as in temporal scaling.

[0085] 2.2.2.3 Candidates for temporal motion vectors

[0086] Except for deriving the reference picture index, the process for deriving the temporal merge candidate is the same as the process for deriving the spatial motion vector candidate (see Figure 6). The reference picture index is signaled to the decoder.

[0087] 2.2.2.4 AMVP Information Signaling

[0088] In AMVP mode, the bitstream is signaled with four parts: the prediction direction, the reference index, the MVD, and the candidate mv predictor index. Syntax table:

[0089] [Table 2]

[0090] 7.3.8.9 Motion Vector Difference Syntax

[0091] [Table 3]

[0092] 2.3 A Novel Interpretation Method in JEM (Joint Exploration Model)

[0093] 2.3.1 Motion vector prediction based on subCU

[0094] In a JEM with QTBT, each CU can have at most one set of motion parameters for each prediction direction. In the encoder, two sub-CU level motion vector prediction methods are considered by dividing a large CU into sub-CUs and deriving motion information for all sub-CUs of the large CU. The ATMVP (Alternative Temporal Motion Vector Prediction) method allows each CU to fetch multiple sets of motion information from multiple blocks smaller than the current CU in the arranged reference picture. In the STMVP (Spatial-Temporal Motion Vector Prediction) method, the motion vectors of sub-CUs are recursively derived using a temporal motion vector predictor and spatial neighbor motion vectors.

[0095] To maintain a more accurate motion field for subCU motion prediction, motion compression of the reference frame is currently disabled.

[0096] 2.3.1.1 Alternative Temporal Motion Vector Prediction

[0097] In ATMVP (Alternative Temporal Motion Vector Prediction), the TMVP (Temporal Motion Vector Prediction) method is modified by fetching multiple sets of motion information (including motion vectors and reference indices) from a smaller block than the current CU. As shown in Figure 18, a subCU is an N×N square block (by default, N is set to 4).

[0098] ATMVP predicts the motion vectors of subCUs within a CU in two steps. The first step is to identify the corresponding block in the reference picture using a so-called temporal vector. This reference picture is called the motion source picture. The second step is to divide the current CU into subCUs, as shown in Figure 18, and obtain the motion vector and reference index of each subCU from the block corresponding to each subCU.

[0099] In the first step, the reference picture and corresponding block are determined by the motion information of the spatially neighboring blocks of the current CU. To avoid repeatedly scanning neighboring blocks, the first merge candidate in the current CU's merge candidate list is used. The first available motion vector and its associated reference index are set to the index of the temporal vector and the motion source picture. In this way, ATMVP can identify the corresponding block more accurately than TMVP, and the corresponding block (sometimes called an arrayed block) is always located to the lower right or center relative to the current CU. In one example, if the first merge candidate is from a neighboring block on the left (i.e., A1 in Figure 19), the source block and source picture are identified using the associated MV and reference picture.

[0100] Figure 19 shows specific examples of source blocks and source pictures.

[0101] In the second step, the time vector is added to the coordinates of the current CU to identify the corresponding block of the subCU by the temporal vector in the motion source picture. For each subCU, the motion information of the subCU is derived using the motion information of its corresponding block (the smallest motion grid covering the central sample). After identifying the motion information of the corresponding N×N blocks, it is converted to the motion vector and reference index of the current subCU, similar to HEVC's TMVP, and motion scaling and other procedures are applied. For example, the decoder checks whether the low-latency condition is met (i.e., the POC of all reference pictures of the current picture is smaller than the POC of the current picture) and, if necessary, predicts the motion vector MVy of each subCU (where X is equal to 0 or 1 and Y is equal to 1-X) using the motion vector MVx (the motion vector corresponding to the reference picture list X).

[0102] 2.3.1.2 Spatial-Temporal Motion Vector Prediction

[0103] In this method, the motion vectors of the subCUs are derived recursively along the order of the raster scan. This concept is illustrated in Figure 20. Consider an 8x8 CU containing four 4x4 subCUs, A, B, C, and D. The 4x4 blocks in the neighborhood of the current frame are labeled a, b, c, and d.

[0104] The derivation of the motion of sub-CU A begins by identifying its two spatial neighborhoods. The first neighborhood is the N×N block above sub-CU A (block c). If block c is unavailable or intra-encoded, check the other N×N blocks above sub-CU A (starting from block c and going from left to right). The second neighborhood is the block to the left of sub-CU A (block b). If block b is unavailable or intra-encoded, check the other blocks to the left of sub-CU A (starting from block b and going from top to bottom). The motion information obtained from the neighborhood blocks of each list is scaled to the first reference frame of the given list. Next, the TMVP (Temporal Motion Vector Predictor) of sub-block A is derived following a procedure similar to the TMVP derivation specified in HEVC. The motion information of the arranged blocks at position D is fetched and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors (up to 3) are averaged separately for each reference list. This averaged motion vector will be taken as the motion vector of the current sub-CU.

[0105] Figure 20 shows an example of a CU having four subblocks (ADs) and their neighboring blocks.

[0106] 2.3.1.3 SubCU Motion Prediction Mode Signal Notification

[0107] Sub-CU modes are valid as additional merge candidates, and no additional syntactic elements are required to signal the mode. Two additional merge candidates are added to the merge candidate list for each CU to represent ATMVP and STMVP modes. Up to seven merge candidates are used if the sequence parameter set indicates that ATMVP and STMVP are valid. The coding logic for the additional merge candidates is the same as for the merge candidates in HM, meaning that for each CU in the P or B slice, two or more RD checks are required for the two additional merge candidates.

[0108] In JEM, all bins in a merge index are contexts coded by CABAC. In contrast, in HEVC, only the first bin is a coded context, while the remaining bins are bypass-coded contexts.

[0109] 2.3.2 Adaptive motion vector difference resolution

[0110] In HEVC, when use_integer_mv_flag is 0 in the slice header, the Motion Vector Difference (MVD) (the difference between the motion vector and the PU's predicted motion vector) is signaled in units of 1 / 4 luminance samples. In JEM, Locally Adaptive Motion Vector Resolution (LAMVR) is introduced. In JEM, the MVD can be coded in units of 1 / 4 luminance samples, integer luminance samples, or 4 luminance samples. The MVD resolution is controlled at the coding unit (CU) level, and the MVD resolution flag is conditionally signaled for each CU that has at least one non-zero MVD component.

[0111] For CUs having at least one non-zero MVD component, a first flag is signaled to indicate whether 1 / 4 luminance sample MV precision is used in the CU. If the first flag (equal to 1) indicates that 1 / 4 luminance sample MV precision is not used, another flag is signaled to indicate whether integer luminance sample MV precision or 4 luminance sample MV precision is used.

[0112] If the first MVD resolution flag for the CU is zero or not coded for the CU (i.e., all MVDs in the CU are zero), then a quarter luminance sample MV resolution is used for the CU. If the CU uses integer luminance sample MV precision or 4 luminance sample MV precision, the MVPs in the CU's AMVP candidate list are rounded to the corresponding precision.

[0113] In the encoder, CU-level RD checks are used to determine which MVD resolution to use for the CU. That is, three CU-level RD checks are performed for each MVD resolution. To speed up the encoder, the following encoding scheme is applied in JEM.

[0114] During the RD check of a CU with a normal 1 / 4 luminance sample MVD resolution, the motion information of the current CU (integer luminance sample precision) is stored. During the RD check of the same CU with integer luminance sample and 4 luminance sample MVD resolutions, the stored motion information (after rounding) is used as a starting point for further small-range motion vector improvements, thus avoiding the time-consuming motion estimation process being repeated three times.

[0115] The RD check for CUs with a 4-luminance sample MVD resolution is conditionally invoked. For CUs, if the RD cost of an integer luminance sample MVD resolution is much greater than that of a quarter-luminance sample MVD resolution, the RD check for the 4-luminance sample MVD resolution for the CU is omitted.

[0116] 2.3.3 Pattern Matching Motion Vector Derivation

[0117] PMMVD (Pattern Matched Motion Vector Derivation) mode is a special merging mode based on FRUC (Frame-Rate Up Conversion) technology. In this mode, block motion information is not transmitted via signal but is derived by the decoder.

[0118] If the merge flag is true, the FRUC flag is signaled to the CU. If the FRUC flag is false, the merge index is signaled and the normal merge mode is used. If the FRUC flag is true, an additional FRUC mode flag is signaled to indicate which method (bilateral matching or template matching) is used to derive the block's motion information.

[0119] On the encoder side, the decision of whether to use the FRUC merge mode for a CU is based on RD cost selection, just as it is done for regular merge candidates. That is, RD cost selection is used to check both matching modes (bilateral matching and template matching) for a single CU. The one that leads to the lowest cost is then compared with the other CU modes. If the FRUC matching mode is the most efficient, the FRUC flag is set to true for the CU and the associated matching mode is used.

[0120] The motion derivation process in FRUC merge mode consists of two steps. First, a motion search is performed at the CU level, and then motion refinement is performed at the sub-CU level. At the CU level, an initial motion vector for the entire CU is derived based on bilateral matching or template matching. First, a list of MV candidates is generated, and the candidate that leads to the minimum matching cost is selected as the starting point for further CU level refinement. Then, a local search is performed near the starting point based on bilateral matching or template matching, and the MV result with the minimum matching cost is taken as the MV for the entire CU. Subsequently, the derived CU motion vector is used as a starting point to further refine the motion information at the sub-CU level.

[0121] For example, the following derivation process is performed to derive W×H CU motion information. In the first stage, the MV for the entire W×H CU is derived. In the second stage, the CU is further divided into M×M subCUs. The value of M is calculated as in (16), where D is a predefined division depth, which is set to 3 by default in JEM. Then, the MV for each subCU is derived.

[0122]

number

[0123] As shown in Figure 21, this bilateral matching is used to derive motion information for the current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. Assuming a continuous motion trajectory, the motion vectors MV0 and MV1 pointing to the two reference blocks are proportional to the temporal distance between the current picture and the two reference pictures, e.g., TD0 and TD1. In a special case, if the current picture is temporally between the two reference pictures and the temporal distance from the current picture to the two reference pictures is the same, the bilateral matching becomes a mirror-based bidirectional MV.

[0124] As shown in Figure 22, template matching is used to derive motion information for the current CU by finding the closest match between the template in the current picture (the neighboring block above and / or to the left of the current CU) and the block in the reference picture (the same size as the template). In addition to the FRUC merge mode described above, template matching is also applied to AMVP mode. In JEM, as with HEVC, AMVP has two candidates. A new candidate is derived using the template matching method. If the newly derived candidate by template matching differs from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and then the list size is set to 2 (meaning the second existing AMVP candidate is removed). When applied to AMVP mode, only CU-level search is applied.

[0125] 2.3.3.1 CU Level MV Candidate Set

[0126] The CU-level MV candidate set consists of the following: (i) The original AMVP candidate when the current CU is in AMVP mode (ii) All merge candidates, (iii) Multiple MVs within an interpolated MV field. (iv) Motion vectors of the upper and left neighbors

[0127] When using bilateral matching, each valid MV of a merge candidate is used as input to generate MV pairs assuming bilateral matching. For example, one valid MV of a merge candidate is (MVa, refa) in reference list A. Then, the reference picture refb of the bilateral MV that forms that pair is found in another reference list B, where refa and refb are on different sides of the current picture in time. If such a refb is not available in reference list B, refb is determined to be a different reference from refa, and its temporal distance from the current picture is the minimum in list B. After determining refb, MVb is derived by scaling MVa based on the temporal distance between the current picture and refa,refb.

[0128] The four MVs from the interpolated MV field are also added to the CU level candidate list. More specifically, the interpolated MVs for the current CU positions (0,0), (W / 2,0), (0,H / 2), and (W / 2,H / 2) are added.

[0129] When applying FRUC in AMVP mode, the original AMVP candidate is also added to the CU-level MV candidate set.

[0130] At the CU level, up to 15 MVs for AMVP CUs and up to 13 MVs for merged CUs are added to the candidate list.

[0131] 2.3.3.2 SubCU Level MV Candidate Set

[0132] The subCU-level MV candidate set consists of the following: (i) MV determined from CU level search, (ii) MV near the top, left, upper left, and upper right, (iii) A scaled version of the juxtaposed MV from the reference picture, (iv) Up to four ATMVP candidates, (v) Up to four STMVP candidates

[0133] The scaled MV from the reference picture is derived as follows: Traverse all reference pictures in both lists. The MVs at the subCU array positions in the reference picture are scaled relative to the reference of the starting CU level MV.

[0134] The candidates for ATMVP and STMVP are limited to the first four candidates.

[0135] At the subCU level, up to 17 MVs are added to the candidate list.

[0136] 2.3.3.3 Generation of Interpolated MV Fields

[0137] Before coding the frame, an interpolated motion field is generated for the entire picture based on one of the MEs. This motion field may then be used later as a MV candidate at the CU level or subCU level.

[0138] First, the motion field of each reference picture in both reference lists is traversed at the 4x4 block level. In each 4x4 block, if interpolated motion has not yet been assigned to the motion associated with the block passing through the 4x4 block of the current picture (shown in Figure 23), the motion of the reference block is scaled to the current picture based on the temporal distances TD0 and TD1 (similar to MV scaling of TMVP in HEVC), and the scaled motion is assigned to the block in the current frame. If a scaled MV has not been assigned to the 4x4 block, the block's motion is marked as unavailable in the interpolated motion field.

[0139] 2.3.3.4 Interpolation and Matching Costs

[0140] When a single motion vector points to a fractional sample position, motion-compensated interpolation is required. To reduce complexity, bilinear interpolation is used for both bilateral and template matching instead of the usual 8-tap HEVC interpolation.

[0141] The calculation of matching costs differs slightly at different steps. When selecting a candidate from a CU-level candidate set, the matching cost is the sum of the absolute differences (SAD) of bilateral matching or template matching. After determining the starting MV, the matching cost C for bilateral matching in sub-CU-level searches is calculated as follows:

[0142]

number

[0143] Here, w is a weighting coefficient empirically set to 4, and MV and MV S These represent the current MV and the starting MV, respectively. SAD is still used as the matching cost for template matching in subCU-level searches.

[0144] In FRUC mode, the MV is derived by using only luminance samples. The derived motion is used for both luminance and saturation for MC interpretation. After determining the MV, the final MC is performed using an 8-tap interpolation filter for luminance and a 4-tap interpolation filter for saturation.

[0145] 2.3.3.5 Improvements to MV

[0146] MV improvement is a pattern-based MV search with a criterion of bilateral matching cost or template matching cost. JEM supports two search patterns, namely UCBDS (Unrestricted Center-Biased Diamond Search) and adaptive cross-search for MV improvement at the CU level and sub-CU level, respectively. For both CU and sub-CU level MV improvement, the MV is searched directly with the accuracy of a 1 / 4 luminance sample MV, followed by improvement of a 1 / 8 luminance sample MV. The search range for MV improvement for the CU and sub-CU steps is set equally to eight luminance samples.

[0147] 2.3.3.6 Selection of prediction direction in template matching FRUC merge mode

[0148] In bilateral matching merge mode, bidirectional prediction is always applied to derive CU motion information based on the closest matching between two blocks along the current CU motion trajectory in two different reference pictures. There is no such limitation in template matching merge mode. In template matching merge mode, the encoder can choose from a single prediction from list0, a single prediction from list1, or bidirectional prediction for the CU. The selection is made based on the template matching cost, as follows: If costBi ≤ factor * min(cost0, cost1) Use bidirectional prediction. In all other cases, if cost0 ≤ cost1 We will use a single prediction from list0. Otherwise, Use a single prediction from list1.

[0149] Here, cost0 is the SAD for list0 template matching, cost1 is the SAD for list1 template matching, and costBi is the SAD for bidirectional predictive template matching. A factor value of 1.25 means that the selection process is biased towards bidirectional prediction. This inter-predictive direction selection applies only to CU-level template matching processes.

[0150] 2.3.4 Decoder-side motion vector improvement

[0151] In bidirectional prediction calculations, two prediction blocks are combined to form a single prediction signal, each consisting of a motion vector (MV) from list0 and an MV from list1, respectively, in order to predict a single block region. In the DMVR (Decoder-side Motion Vector Refinement) method, the two motion vectors of the bidirectional prediction are further refined by bilateral template matching. To obtain the refined MV without transmitting additional motion information, bilateral template matching is applied in the decoder, performing a search based on the distortion between the bilateral template and the reconstructed sample in the reference picture.

[0152] In DMVR, as shown in Figure 23, a bilateral template is generated from the first MV0 of list0 and MV1 of list1 as a weighted combination (i.e., average) of two prediction blocks, respectively. The template matching operation consists of calculating a cost metric between the generated template and the sample region in the reference picture (near the first prediction block). For each of the two reference pictures, the MV with the minimum template cost is considered the updated MV in that list and replaces the original MV. In JEM, nine MV candidates are searched for for each list. The nine MV candidates include the original MV and eight surrounding MVs, each having one luminance sample offset from the original MV in either the horizontal or vertical direction, or both. Finally, the two new MVs, i.e., MV0' and MV1' as shown in Figure 24, are used to generate the final bilateral prediction results. The sum of the absolute values ​​of the differences (SAD) is used as the cost metric.

[0153] DMVR can take one M from a past reference picture without sending additional syntactic elements. This applies to a bidirectional prediction merge mode between V and one MV from a future reference picture. In JEM, relative to CU, LIC, affine motion, FRUC, or sub-CU If a merge candidate is valid, the DMVR will not be applied.

[0154] 2.3.5 Merge / Skip Mode with Improved Bilateral Matching

[0155] First, the merge candidate list is constructed by inserting the motion vectors and reference indices of spatially and temporally neighboring blocks into a redundant candidate list until the number of available candidates reaches the maximum candidate size of 19. The merge candidate list in merge / skip mode is constructed by inserting spatial candidates (Figure 11) used for HEVC (combine candidates and zero candidates), temporal candidates, affine candidates, ATMVP (Advanced Temporal MVP) candidates, STMVP (Spatial Temporal MVP) candidates, and additional candidates based on a predetermined insertion order.

[0156] - Spatial candidates for blocks 1-4

[0157] - Extrapolated affine candidates for blocks 1-4

[0158] -ATMVP

[0159] -STMVP

[0160] - Virtual Affine Candidates

[0161] - Spatial candidates (block 5) (used only if the number of available candidates is less than 6).

[0162] -Extrapolated affine candidates (Block 5)

[0163] - Temporal candidates (derived like HEVC)

[0164] - Non-adjacent spatial candidates following extrapolated affine candidates (blocks 6-49 shown in Figure 25).

[0165] -Combination candidates

[0166] - Zero candidate

[0167] Note that the IC flag is inherited from the merge candidate, except for STMVP and affine. Also, for the first four spatial candidates, the bidirectional prediction is inserted before the single prediction.

[0168] [8] allows access to blocks that are not connected to the current block. If the non-adjacent block is coded in non-intra mode, the relevant motion information may be added as an additional merge candidate.

[0169] 3. Examples of problems that the embodiments disclosed herein seek to solve.

[0170] Current HEVC designs can correlate the current block with its neighboring blocks (next to the current block) to better code motion information. However, neighboring blocks may correspond to different objects with different motion trajectories. In this case, prediction from those neighboring blocks is not efficient.

[0171] Predicting movement information from non-adjacent blocks incurs the cost of storing all movement information (typically 4x4 levels) in a cache, resulting in additional coding gains and significantly increasing the complexity of the hardware implementation.

[0172] 4. Some examples of technical solutions

[0173] Embodiments of the technology of this disclosure overcome the shortcomings of existing implementations and thereby provide video coding with higher coding efficiency. In this description, the LUT may be a table, a list, an array, or any other array of indexed entries.

[0174] To overcome the shortcomings of existing implementations, various embodiments can implement a motion vector prediction technique based on a LUT that uses one or more tables (e.g., lookup tables) storing at least one motion candidate to predict block motion information, thereby providing video coding with higher coding efficiency. A lookup table is just one example of a table that can be used when including motion candidates to predict block motion information, and other implementations are possible. Each LUT may include one or more motion candidates, each associated with corresponding motion information. The motion information of a motion candidate may include the prediction direction, reference index / picture, motion vector, LIC flag, affine flag, MVD (Motion Vector Derivation) accuracy, and / or some or all of the MVD value. The motion information may further include block position information to indicate where the motion information originates.

[0175] The LUT-based motion vector prediction based on the disclosed technology can improve both existing and future video coding standards and is illustrated in the following examples for various embodiments. Since the LUT allows coding / decoding to be performed based on historical data (e.g., already processed blocks), the LUT-based motion vector prediction can also be called the History-based Motion Vector Prediction (HMVP) method. In the LUT-based motion vector prediction method, one or more tables containing motion information from previously coded blocks are maintained during the coding / decoding process. These motion candidates stored in the LUT are referred to as HMVP candidates. During the coding / decoding of one block, the associated motion information in the LUT may be added to a motion candidate list (e.g., a merge / AMVP candidate list), and the LUT may be used after coding / decoding one block. The updated LUT is then used to decode subsequent blocks. Thus, the updating of motion candidates in the LUT is based on the order of coding / decoding of the blocks. The following examples should be considered as examples to illustrate the general concept.

[0176] The following examples should be considered as illustrative examples for illustrating general concepts. These examples should not be interpreted in a narrow sense. Furthermore, these examples can be combined in any way.

[0177] In some embodiments, one or more lookup tables, each storing at least one motion candidate, may be used to predict motion information for a block. The embodiments may use the motion candidates to indicate a set of motion information stored in the lookup tables. In the case of conventional AMVP or merge mode, embodiments may use AMVP or merge candidates to store motion information.

[0178] The following examples illustrate the general concept.

[0179] Example of a lookup table

[0180] Each lookup table may contain one or more motion candidates associated with each candidate's motion information.

[0181] LUT Selection

[0182] Example B1: When coding a block, you can sequentially check some or all of the move candidates from a single lookup table. As you check a move candidate while coding a block, you may add this move candidate to a move candidate list (e.g., AMVP, merge candidate list).

[0183] How to use a lookup table

[0184] The total number of possible movements in the lookup table to be checked may be predetermined. a. It may further depend on coded information, block size, block shape, etc. 1. For example, in AMVP mode, only m motion candidates may be checked, while in merge mode, n motion candidates may be checked (for example, m=2, n=44). b. In one example, the total number of motion candidates to be checked may be signaled in an area including a VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), slice header, tile header, CTU (Coding Tree Unit), CTB (Coding Tree Block), CU (Coding Unit) or PU (Prediction Unit), or multiple CTUs / CTBs / CUs / PUs.

[0185] Example C1: One or more motion candidates contained in a lookup table may be directly inherited by a single block. a. They may be used in merge mode coding; that is, motion candidates may be checked in the merge candidate list derivation process. b. These may be used for affine merge mode coding. i. If the affine flag is 1, move candidates in the lookup table can be added as affine merge candidates. c. In the following cases, you may enable checking for movement candidates in the lookup table. i. After inserting a TMVP candidate, the merge candidate list is not full. ii. After checking for specific spatially neighboring blocks to derive a spatial merge candidate, the merge candidate list is not full. iii. After all spatial merge candidates have been processed, the merge candidate list is not full. iv. After predicting merge candidates in both directions, the merge candidate list is not full. v. If the number of spatial or temporal (e.g., adjacent and non-adjacent spaces, TMVP, STMVP, ATMVP, etc.) merge candidates added to the merge candidate list from other coding schemes (e.g., HEVC design or JEM design merge derivation process) is less than the number obtained by subtracting a given threshold from the maximum allowable number of merge candidates. 1. In one example, the threshold is set to 1 or 0. 2. Alternatively, thresholds may be signaled or predefined in SPS / PPS / sequence, picture, or slice header / tile. 3. Alternatively, the threshold may be adaptively changed for each block. For example, it may depend on coded block information such as block size / block shape / slice type, and / or on the number of available spatial or temporal merge candidates. 4. In another example, if the number of certain merge candidates already added to the merge candidate list is less than the number obtained by subtracting a given threshold from the maximum allowed merge candidates. "Certain merge candidates" may be spatial candidates such as HEVC, or non-adjacent merge candidates. vi. Pruning may be applied before adding motion candidates to the merge candidate list. In various implementations of this example and other examples disclosed in this patent specification, pruning may include a) comparing motion information with existing entries for uniqueness, or b) adding the motion information to the list if it is unique, or c) if it is not unique, c1) not adding the motion information, or c2) adding the motion information and removing the matching existing entry. In some implementations, the pruning step is not performed when adding motion candidates from the table to the candidate list. 1. In one example, the motion candidates may be pruned to all or part of the available spatial or temporal (e.g., adjacent and non-adjacent spaces, TMVP, STMVP, ATMVP, etc.) merge candidates from other coding methods of the merge candidate list. 2. The move candidates do not need to be pruned to move candidates based on subblocks, such as ATMVP and STMVP. 3. In one example, the current move candidate may be pruned to all or part of the available move candidates in the merge candidate list (inserted before the current move candidate). 4. The number of pruning steps associated with a motion candidate (e.g., the number of times a motion candidate needs to be compared with other candidates in the merge list) may depend on the number of available spatial or temporal merge candidates. For example, when checking a new motion candidate, if there are M candidates available in the merge list, the new motion candidate may be compared only with the first K candidates (K ≤ M). If the pruning function returns false (e.g., not identical to any of the first K candidates), this new motion candidate is considered different from all M candidates and may be added to the merge candidate list. In one example, K is set to min(K, 2). 5. In one example, the newly added motion candidate is simply compared with the first N candidates in the merge candidate list. For example, N = 3, 4, or 5. N may be signaled from the encoder to the decoder. 6. In one example, the new motion candidate to be checked is compared only with the last N candidates in the merge candidate list. For example, N = 3, 4, or 5. N may be signaled from the encoder to the decoder. 7. In one example, the method for selecting a candidate previously added to the list and comparing it to a new move candidate from the table may depend on where the previously added candidate was derived from. a. In one example, the move candidates in the lookup table may be compared with candidates derived from blocks in a given temporal and / or spatial proximity. b. In one example, different entries for motion candidates in the lookup table may be compared with different candidates that were added previously (i.e., derived from different locations).

[0186] Example C2: The motion candidates included in the lookup table may be used as a prediction module to code the motion information of the blocks. a. They may be used for AMVP mode coding. That is, motion candidates may be checked in the AMVP candidate list derivation process. b. You may enable checking for movement candidates in the lookup table in the following cases: i. After inserting TMVP candidates, the AMVP candidate list is not full. ii. The AMVP candidate list is selected from spatial neighbors, pruned, and is not full immediately before inserting TMVP candidates. iii. If no AMVP candidates exist from the upper adjacent block without scaling, and / or if no AMVP candidates exist from the left adjacent block without scaling. iv. Pruning may be applied before adding motion candidates to the AMVP candidate list. c. Check for motion candidates that have the same reference picture as the current reference picture. i. Alternatively, check motion candidates that have a different reference picture than the current reference picture (along with the scaled MV). ii. Alternatively, first check all motion candidates that have the same reference picture as the current reference picture, and then check motion candidates that have a different reference picture than the current reference picture. iii. Alternatively, merges can be performed in a similar manner to check for candidates.

[0187] Example C3: You may use lookup table motion candidates from a previously coded frame / slice / tile to predict block motion information in a different frame / slice / tile.

[0188] Updating the lookup table

[0189] Example D1: After coding a block containing motion information (i.e., in IntraBC mode, inter-encoded mode), one or more lookup tables may be updated.

[0190] Example D2: If one block is located at the boundary of one picture / slice / tile, updating the lookup table is never allowed.

[0191] Example D3: To code the current LCU row, you may disable the motion information for the above LCU row.

[0192] Example D4: When starting to code a slice / tile using a new time layer index, the number of available motion candidates can be reset to 0.

[0193] Example D5: The lookup table may be updated sequentially with rows / slices of a single slice / tile / LCU that have the same time-layer index.

[0194] Example D6: The lookup table update process may be called within different steps.

[0195] Additional exemplary embodiments

[0196] A History-based MVP (HMVP) method is proposed, which defines HMVP candidates as motion information for previously coded blocks. During the coding / decoding process, a table containing multiple HMVP candidates is maintained. Whenever a new slice is encountered, the table is emptied. Whenever there are intercoded blocks, the relevant motion information is added to the last entry in the table as a new HMVP candidate. The overall coding flow is shown in Figure 31.

[0197] In one example, the table size is set to L (for example, L = 16, 6, or 44), which indicates that up to L HMVP candidates can be added to the table.

[0198] In one embodiment (corresponding to Example 11.gi), if there are more than L HMVP candidates from previously coded blocks, a First-In-First-Out (FIFO) rule is applied so that the table always contains the most recent L previously coded move candidates. Figure 32 shows an example of applying the FIFO rule to remove HMVP candidates and adding new ones to the table used in the proposed method.

[0199] In another embodiment (corresponding to Invention 11.g.iii), whenever a new motion candidate is added (for example, if the current block is intercoded and in non-affine mode), a redundancy check is first applied to identify whether there are any identical or similar motion candidates in the LUT.

[0200] Some examples are shown below.

[0201] Figure 33A shows an example where the LUT is full before adding new motion candidates.

[0202] Figure 33B shows an example where the LUT is not full before adding a new motion candidate.

[0203] Figures 33A and 33B both show examples of LUT update methods based on redundancy removal (removing one candidate for redundant motion).

[0204] Figures 34A and 34B illustrate two example implementations of a LUT update method based on redundancy removal (removing multiple redundancy motion candidates; two candidates are shown in the figures).

[0205] Figure 34A shows an example where the LUT is full before adding new motion candidates.

[0206] Figure 34B shows an example where the LUT is not full before adding a new motion candidate.

[0207] HMVP candidates may be used in the merge candidate list construction process. After the TMVP candidates, all HMVP candidates from the last entry to the first entry in the table (or the HMVP with the last K0, e.g., K0=16 or 6) are inserted. Pruning is applied to the HMVP candidates. The merge candidate list construction process terminates when the total number of available merge candidates reaches the signaled maximum allowable merge candidate. Alternatively, the fetching of motion candidates from the LUT terminates when the total number of added motion candidates reaches a given value.

[0208] Similarly, HMVP candidates may be used in the AMVP candidate list construction process. After the TMVP candidates, the motion vectors of the last K1 HMVP candidates in the table are inserted. The AMVP candidate list is constructed using only HMVP candidates that have the same reference picture as the AMVP target reference picture. Pruning is applied to the HMVP candidates. In one example, K1 is set to 4.

[0209] Figure 28 is a block diagram of the video processing device 2800. The device 2800 may be used to implement one or more of the methods described herein. The device 2800 may be implemented by a smartphone, tablet, computer, IoT (Internet of Things) receiver, etc. The device 2800 may include one or more processors 2802, one or more memories 2804, and video processing hardware 2806. One or more processors 2802 may be configured to implement one or more of the methods described herein. One or more memories 2804 may be used to store data and code used to implement the methods and techniques described herein. The video processing hardware 2806 may be used to implement the techniques described herein in hardware circuitry.

[0210] Figure 29 is a flowchart of an example of video processing method 2900. Method 2900 includes maintaining a set of tables in step 2902, each table containing motion candidates, and each motion candidate associated with corresponding motion information. Method 2900 further includes updating the motion candidate list in step 2904 using a pruning process for motion candidates, based on motion candidates in one or more tables. Method 2900 further includes, in step 2906, using the constructed motion candidate list to perform a conversion between a first video block and a bitstream representation of the video containing the first video block.

[0211] Figure 30 is a flowchart of an example of video processing method 3000. Method 3000 includes, in step 3002, receiving a bitstream representation of video containing a first video block of video data. Method 3000 further includes, in step 3004, applying a pruning process to motion candidates in one or more tables in order to update a candidate list, each table containing motion candidates and each motion candidate associated with corresponding motion information. Method 3000 further includes, in step 3006, performing a conversion between the bitstream representation and the first video block using the constructed candidate list.

[0212] The features and embodiments of the above-described method / technology are described below using a section-based format.

[0213] 1. A video processing method comprising: maintaining a set of tables, each containing motion candidates, and each motion candidate associated with corresponding motion information; updating a candidate list based on motion candidates in one or more tables using a pruning process for motion candidates; and performing a conversion between a first video block and a bitstream representation of the video containing the first video block using the configured candidate list.

[0214] 2. A video processing method comprising: receiving a bitstream representation of video including a first video block; applying a pruning process to motion candidates in one or more tables to constitute a candidate list, wherein each table contains motion candidates and each motion candidate is associated with corresponding motion information; and performing a conversion between the bitstream representation and the first video block using the constituted candidate list.

[0215] 3. The method according to the first or second paragraph, wherein at least one table contains motion candidates derived from previously decoded video blocks that were decoded before the first video block.

[0216] 4. The method according to the first or second paragraph, wherein the pruning step is performed before adding the candidate movements from the table to the candidate list.

[0217] 5. The candidate list is the method described in paragraph 3, corresponding to the merge candidate list.

[0218] 6. The method according to the first or second paragraph, wherein the pruning step includes comparing a candidate movement from the table with at least a portion of the available spatial or temporal merge candidates.

[0219] 7. The method according to the first or second paragraph, wherein the pruning step includes comparing a candidate movement from a table with all available spatial merge candidates.

[0220] 8. The method according to paragraphs 1, 2, or 4, wherein the pruning step is performed without changing the motion candidates from a table to motion candidates based on subblocks.

[0221] 9. The method of paragraph 1 or 2, wherein the pruning step includes comparing motion candidates from a table with at least one of the available motion candidates in a merge candidate list before adding motion candidates.

[0222] 10. The method according to the first or second paragraph, wherein the pruning process comprises a plurality of steps, the number of which is a function of a plurality of spatial or temporal merge candidates.

[0223] 11. The method according to paragraph 9, wherein when M candidates are available in the merge candidate list, pruning is applied to only K merge candidates, K ≤ M, and K and M are integers.

[0224] 12. The method described in item 10, which sets K to min(K,2).

[0225] 13. The method according to paragraph 1, wherein the pruning process includes comparing a motion candidate from a table with a first N candidates in the candidate list, where N is an integer, before adding the motion candidate to the candidate list.

[0226] 14. The method according to the first or second paragraph, wherein the pruning step includes comparing a motion candidate from the table to be checked with the last N candidates in the candidate list, where N is an integer, before adding the motion candidate to the candidate list.

[0227] 15. The method of paragraph 12 or 13, further comprising signaling the value of N.

[0228] 16. The method of paragraph 1, wherein the pruning process includes comparing a move candidate from a table with a candidate in the candidate list, based on how the candidate was derived.

[0229] 17. The method according to paragraph 15, wherein the pruning process includes comparing a candidate movement from a table with a candidate movement derived from a given temporally or spatially adjacent block.

[0230] 18. The method described in Section 15, which compares entries for different movement candidates in a table with previously added different candidates derived from different locations.

[0231] 19. The method according to claim 1, further comprising not calling a pruning process when adding a motion candidate from a table to the candidate list.

[0232] 20. The method according to any one of claims 1 to 19, wherein performing the conversion includes generating the bitstream representation from the video block.

[0233] 21. The method according to any one of claims 1 to 19, wherein performing the conversion includes generating the video block from the bitstream representation.

[0234] 22. The motion candidate is associated with motion information including at least one of a prediction direction, a reference picture index, a motion vector value, an intensity compensation flag, an affine flag, a motion vector difference precision, or a motion vector difference value. The method according to any one of claims 1 to 21.

[0235] 23. The method according to any one of claims 1 to 22, wherein the motion candidate corresponds to a motion candidate of an intra prediction mode for intra mode coding.

[0236] 24. The method according to any one of claims 1 to 22, wherein the motion candidate corresponds to a motion candidate including illumination compensation parameters for IC parameter coding.

[0237] 25. The method according to any one of claims 1 to 24, further comprising updating one or more tables based on the conversion.

[0238] 26. The method according to claim 25, wherein updating one or more tables includes updating one or more tables based on the motion information of the first video block after performing the conversion.

[0239] 27. The method according to claim 26, further comprising performing a conversion between a subsequent video block of the video and the bitstream representation of the video based on the updated table.

[0240] 28. An apparatus comprising a processor and a non - transient memory storing instructions, wherein when the instructions are implemented by the processor, the processor is caused to perform the method according to any one of items 1 to 27.

[0241] 29. A computer program product stored in a non - transient computer - readable medium, the computer program product including program code for executing the method according to any one of items 1 to 27.

[0242] The specific embodiments of the technology of the present disclosure have been described above for the purpose of explanation. It will be understood that various modifications can be made without departing from the scope of the present invention. Therefore, the technology of the present disclosure is not limited except as defined by the appended claims.

[0243] The disclosed and other embodiments, modules, and functional operation implementations described herein may be implemented in digital electronic circuits, or in computer software, firmware, or hardware, or in one or more combinations thereof, including the structures disclosed herein and their structural equivalents. The disclosed and other embodiments may be implemented as one or more modules of computer program instructions encoded on a computer-readable medium for implementation by one or more computer program products, i.e., data processing devices, or for controlling the operation of data processing devices. This computer-readable medium may be a machine-readable storage device, a machine-readable storage board, a storage device, a composition of a material that provides a machine-readable propagating signal, or one or more combinations thereof. The term “data processing device” includes, for example, all devices and machines for processing data, including a programmable processor, a computer, or multiple processors or computers. In addition to hardware, this device may include code that makes up the execution environment of the computer program, such as processor firmware, a protocol stack, a database management system, an operating system, or code that constitutes one or more combinations thereof. A propagated signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, produced to encode information for transmission to a suitable receiving device.

[0244] A computer program (also called a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed as a standalone program or in any form, including modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be recorded in part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), stored in a single file dedicated to that program, or stored in multiple coordinating files (e.g., a file containing one or more modules, subprograms, or parts of code). A computer program can be deployed to run on one computer located at one site, or on multiple computers distributed across multiple sites and interconnected by a communication network.

[0245] The processes and logic flows described herein can be carried out by one or more programmable processors that run one or more computer programs to perform a function by operating on input data and generating outputs. The processes and logic flows can also be carried out by special-purpose logic circuits, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits), and the devices can also be implemented as special-purpose logic circuits.

[0246] Processors suitable for executing computer programs include, for example, both general-purpose and dedicated microprocessors, as well as any one or more processors in any type of digital computer. Generally, a processor receives instructions and data from read-only memory or random-access memory or both. Essential elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Generally, a computer may include one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or may be operablely coupled to receive data from or transfer data to these mass storage devices. However, a computer is not required to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, such as EPROM, EEPROM, flash memory, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and semiconductor storage devices such as CD-ROM and DVD-ROM disks. Processors and memory may be complemented by or incorporated into dedicated logic circuits.

[0247] This patent specification contains many details, but these should not be interpreted as limiting the scope of any invention or claims, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this patent specification in the context of another embodiment may be implemented in combination in one example. Conversely, various features described in the context of a single example may be implemented separately or in any suitable subcombination in multiple embodiments. Furthermore, features may be described above as acting in a particular combination and initially asserted as such, but one or more features from an asserted combination may, in some cases, be extracted from the combination, and the asserted combination may be directed towards a subcombination or a variation of a subcombination.

[0248] Similarly, although the operations are shown in a specific order in the drawings, this should not be understood as requiring that such operations be performed in a specific or sequential order shown, or that all shown operations be performed, in order to achieve the desired result. Furthermore, the separation of various system components in the examples described in this patent specification should not be understood as requiring such separation in all embodiments.

[0249] Only a few implementation forms and examples are described, and other embodiments, extensions, and modifications are possible based on what is described and illustrated in this patent specification.

Claims

1. A method for processing video data, Maintaining a table, wherein the table includes one or more candidates derived from one or more blocks of video coded prior to the current block, the one or more candidates being history-based motion vector prediction (HMVP) candidates, and the arrangement of the one or more candidates in the table being based on the order in which the one or more candidates were added to the table. The configuration of a candidate list, wherein at least one first candidate in the table is checked during the configuration if the number of spatial or temporal candidates placed in the candidate list is less than the maximum allowable merge candidate minus a predetermined threshold, and a pruning process is used for the at least one first candidate to determine whether to add the at least one first candidate to the candidate list, the pruning process includes comparing the at least one first candidate with at least one second candidate included in the candidate list, each of the at least one second candidate is selected from the candidate list based on where each of the at least one second candidate is derived from, each of the at least one second candidate is derived from a predetermined spatially neighboring block of the current block, and each of the at least one second candidate is not a subblock-based candidate. Using the aforementioned list of candidates, motion information is derived, Based on the motion information, perform a conversion between the current block and the bitstream of the video. A method of having.

2. The at least one second candidate includes the first N candidates in the candidate list, The method according to claim 1, wherein N is an integer.

3. The total number of the aforementioned at least one second candidate is K. The method according to claim 1, wherein K is 2 or less.

4. The aforementioned at least one second candidate includes at least one of the spatial candidates derived from block A1 and the spatial candidates derived from block B1. Block A1 is the left neighboring block to the current block, The aforementioned block B1 is an upper neighboring block to the current block. The method according to claim 3.

5. The method according to claim 1, wherein the at least one second candidate is part of the candidates present in the candidate list.

6. The method according to claim 5, wherein some of the candidates in the candidate list are some of the spatial candidates.

7. The method according to claim 2, wherein if one of the at least one first candidate is different from any of the at least one second candidate, that candidate is added to the candidate list.

8. The method according to claim 1, wherein the candidate in the table is associated with motion information including at least one of the prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference accuracy, and motion vector difference value.

9. The method according to claim 1, further comprising updating the table using the movement information of the current block.

10. The method according to claim 9, wherein if the table becomes full before candidates corresponding to the motion information are added to the table, candidates are removed from the table.

11. The method according to claim 1, wherein the candidate list is a merge candidate list.

12. The method according to claim 1, wherein the conversion comprises encoding the current block into the bitstream.

13. The method according to claim 1, wherein the conversion comprises decoding the current block from the bitstream.

14. A device for processing video data, comprising a processor and non-temporary memory having instructions, When the aforementioned instruction is executed by the processor, the processor will be instructed to: Maintaining a table, the table including one or more candidates derived from one or more blocks of video coded prior to the current block, The one or more candidates are history-based motion vector prediction (HMVP) candidates, and the arrangement of the one or more candidates in the table is based on the order in which the one or more candidates were added to the table. The configuration of a candidate list, wherein at least one first candidate in the table is checked during the configuration if the number of spatial or temporal candidates placed in the candidate list is less than the maximum allowable merge candidate minus a predetermined threshold, and a pruning process is used for the at least one first candidate to determine whether to add the at least one first candidate to the candidate list, the pruning process includes comparing the at least one first candidate with at least one second candidate included in the candidate list, each of the at least one second candidate is selected from the candidate list based on where each of the at least one second candidate is derived from, each of the at least one second candidate is derived from a predetermined spatially neighboring block of the current block, and each of the at least one second candidate is not a subblock-based candidate. Using the aforementioned list of candidates, motion information is derived, Based on the motion information, perform a conversion between the current block and the bitstream of the video. A device having.

15. In the processor, Maintaining a table, the table including one or more candidates derived from one or more blocks of video coded prior to the current block, The one or more candidates are history-based motion vector prediction (HMVP) candidates, and the arrangement of the one or more candidates in the table is based on the order in which the one or more candidates were added to the table. The configuration of a candidate list, wherein at least one first candidate in the table is checked during the configuration if the number of spatial or temporal candidates placed in the candidate list is less than the maximum allowable merge candidate minus a predetermined threshold, and a pruning process is used for the at least one first candidate to determine whether to add the at least one first candidate to the candidate list, the pruning process includes comparing the at least one first candidate with at least one second candidate included in the candidate list, each of the at least one second candidate is selected from the candidate list based on where each of the at least one second candidate is derived from, each of the at least one second candidate is derived from a predetermined spatially neighboring block of the current block, and each of the at least one second candidate is not a subblock-based candidate. Using the aforementioned list of candidates, motion information is derived, Based on the motion information, perform a conversion between the current block and the bitstream of the video. A non-temporary computer-readable storage medium that stores instructions to perform a certain action.

16. A method for storing a video bitstream, The aforementioned method, Maintaining a table, the table including one or more candidates derived from one or more blocks of video coded prior to the current block, The one or more candidates are history-based motion vector prediction (HMVP) candidates, and the arrangement of the one or more candidates in the table is based on the order in which the one or more candidates were added to the table. The configuration of a candidate list, wherein at least one first candidate in the table is checked during the configuration if the number of spatial or temporal candidates placed in the candidate list is less than the maximum allowable merge candidate minus a predetermined threshold, and a pruning process is used for the at least one first candidate to determine whether to add the at least one first candidate to the candidate list, the pruning process includes comparing the at least one first candidate with at least one second candidate included in the candidate list, each of the at least one second candidate is selected from the candidate list based on where each of the at least one second candidate is derived from, each of the at least one second candidate is derived from a predetermined spatially neighboring block of the current block, and each of the at least one second candidate is not a subblock-based candidate. Using the aforementioned list of candidates, motion information is derived, Based on the motion information, the bitstream of the video is generated, Storing the bitstream on a non-temporary computer-readable recording medium, A method of having.

Images