Update of HMVP table

Abstract

An example method of video processing includes maintaining a history-based motion vector prediction (HMVP) table including at least one motion candidate; performing a conversion between a current video block of a video and a bitstream representation of the video; and updating the table by removing a Kth entry in the table and adding the motion candidate to the HMVP table, wherein K is a positive integer and is determined based on a predetermined rule, and the motion candidate added to the HMVP table is derived from motion information derived during the conversion.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is to timely claim the priority and benefits of International Patent Application No. PCT / CN2019 / 123670 filed on December 6, 2019, in accordance with applicable patent laws and / or rules pursuant to the Paris Convention. The entire disclosure of International Patent Application No. PCT / CN2019 / 123670 is incorporated by reference as part of the disclosure of this application. Technical Field

[0003] This patent document relates to image and video encoding and decoding. Background Art

[0004] Digital video accounts for the largest use of bandwidth on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, bandwidth demand for digital video usage is expected to continue to grow. Summary of the invention

[0005] This document discloses techniques that can be used by video encoders and decoders to perform cross-component adaptive loop filtering during video encoding or decoding.In one example aspect, a method of video processing is disclosed.

[0006] In one example aspect, a method of video processing is disclosed. The method includes: performing conversion between a video block of a video and a bitstream representation of the video; and selectively updating a table of motion candidates for another video block by removing a Kth entry of the table, where K is a rule-based positive integer.

[0007] In one example aspect, a method of video processing is disclosed. The method includes: performing conversion between a video block of a video and a bitstream representation of the video; and updating a table of motion candidates for another video block by removing existing candidates in the table and adding new candidates, selectively based on a rule, wherein the rule depends on characteristics of the existing candidates.

[0008] In one example aspect, a method of video processing is disclosed. The method includes: performing conversion between a video block of a video and a bitstream representation of the video; and optionally based on a rule, updating a table of motion candidates for another video block by adding a new candidate based on the conversion at a Kth position starting from a 0th position in the table, where K depends on the rule.

[0009] In one example aspect, a method of video processing is disclosed. The method includes: performing conversion between a video block of a video processing unit of the video and a bitstream representation of the video; and selectively updating a table of motion candidates for another video block based on a rule, wherein the rule depends on the video processing unit.

[0010] In one example aspect, a method of video processing is disclosed. The method includes: maintaining a history-based motion vector prediction (HMVP) table including at least one motion candidate; performing a conversion between a current video block of a video and a bitstream representation of the video; and updating the table by removing a Kth entry in the table and adding the motion candidate to the HMVP table, wherein K is a positive integer and is determined based on a predetermined rule, and the motion candidate added to the HMVP table is derived from motion information derived during the conversion.

[0011] In one example aspect, a method of video processing is disclosed. The method includes: maintaining a history-based motion vector prediction (HMVP) table including at least one motion candidate; performing a conversion between a current video block of a video and a bitstream representation of the video; and updating the table with a motion candidate based on a position of a redundant candidate in the HMVP table, wherein the motion candidate is derived from motion information derived during the conversion, and the redundant candidate is the same as or similar to the motion candidate.

[0012] In one example aspect, a method of video processing is disclosed. The method includes: maintaining a history-based motion vector prediction (HMVP) table including at least one motion candidate; performing a conversion between a current video block of a video and a bitstream representation of the video; and updating the table by adding the motion candidate to a Kth entry in the HMVP table, wherein the motion candidate added to the HMVP table is derived from motion information derived during the conversion, and K is a positive integer and is determined based on a predetermined rule.

[0013] In one example aspect, a method of video processing is disclosed. The method includes: maintaining a history-based motion vector prediction (HMVP) table including at least one motion candidate; performing conversion between a video processing unit of a video block of a video and a bitstream representation of the video; and updating the HMVP table based on one or more rules associated with the video processing unit, wherein the rule changes from one video processing unit to another video processing unit.

[0014] In yet another example aspect, a video encoder apparatus is disclosed. The video encoder includes a processor configured to implement the above method.

[0015] In yet another example aspect, a video encoder apparatus is disclosed. The video encoder includes a processor configured to implement the above method.

[0016] In yet another exemplary aspect, a computer readable medium having stored thereon code is disclosed. The code is in the form of processor executable code embodying one of the methods described herein.

[0017] These and other features are described throughout this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A typical HEVC video encoder is shown (decoder modeling elements are shaded in light grey).

[0019] Figure 2 An example of video block segmentation is shown.

[0020] Figure 3 An example of allowed prediction blocks for an MxM codec unit or video block is shown.

[0021] Figure 4A-4B The partitioning pattern is shown.

[0022] Figure 5A-Figure 5B An example of a partitioning pattern using a quadtree binary tree is shown.

[0023] Figure 6 (a) to (e) show various block division modes.

[0024] Figure 7 The derivation process of constructing the Merge candidate list is shown.

[0025] Figure 8 Example locations of spatial merge candidates are shown.

[0026] Fig. 9 Candidate pairs considered for redundancy check of spatial merge candidates are shown.

[0027] Figure 10A-10B The location of the second PU of the N×2N and 2N×N partitions is shown.

[0028] Fig.11 is an illustration of motion vector scaling for temporal merge candidates.

[0029] Fig.12 The candidate positions of the temporal Merge candidates C0 and C1 in the collocated pictures are shown.

[0030] Fig.13 An example of a combined bi-predictive Merge candidate is shown.

[0031] Fig.14 The derivation process of motion vector prediction candidates is shown.

[0032] Fig.15 is an illustration of motion vector scaling for spatial motion vector candidates.

[0033] Fig.16An example encoding and decoding flow of history-based motion vector prediction (HMVP) is shown.

[0034] Fig.17 The modified Merge list building process is shown.

[0035] Figure 18A-18B A HMVP update method based on redundancy removal (removing one redundant motion candidate) is shown.

[0036] Figure 19A-19B An example of a HMVP updating method based on redundancy removal (removing one redundant motion candidate) is shown.

[0037] Fig. 20 CU and PU partitioning examples are shown.

[0038] Fig.21 An example of a current block and its neighboring blocks used in AVS3 is shown.

[0039] Figure 22A-22D An example of the HMVP update method based on redundancy removal (table size is set to L) is depicted.

[0040] Figure 23A-23F An example of the HMVP table update process is depicted.

[0041] Fig.24 is a block diagram of an example video processing system in which the disclosed technology may be implemented.

[0042] Fig.25 is a block diagram of an example hardware platform for video processing.

[0043] Fig.26 is a flow chart of an example method of video processing.

[0044] Fig. 27 is a flow chart of an example method of video processing.

[0045] Fig.28 is a flow chart of an example method of video processing.

[0046] Fig.29 is a flow chart of an example method of video processing.

[0047] Fig.30 is a flow chart of an example method of video processing. DETAILED DESCRIPTION

[0048] The section headings used in this document are intended to facilitate understanding, rather than to limit the applicability of the techniques and embodiments disclosed in each section to that section. In addition, the use of H.266 terminology in some descriptions is intended only to facilitate understanding, rather than to limit the scope of the disclosed techniques. Therefore, the techniques described herein are also applicable to other video codec designs.

[0049] 1. Introduction

[0050] This patent document relates to video coding technology. Specifically, it relates to motion vector coding in image / video coding. It can be applied to existing video coding standards (such as HEVC), or to-be-completed standards (Multi-function Video Codec), the third generation of the Chinese audio and video coding standard (AVS3). It can also be applied to future video coding standards or video codecs.

[0051] 2. Brief description

[0052] Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, ISO / IEC developed MPEG-1 and MPEG-4 Visual, and the two organizations jointly developed the H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Codec (AVC) and H.265 / HEVC standards. Since H.262, video codec standards are based on a hybrid video codec structure that utilizes temporal prediction plus transform coding. An example of a typical HEVC encoder framework is shown in Figure 1 Depicted in.

[0053] 2.1. Segmentation structure

[0054] 2.1.1 Segmentation Tree Structure in H.264 / AVC

[0055] The core of the codec layer in previous standards is the macroblock, which contains a 16×16 block of luma samples and, in the usual case of 4:2:0 color sampling, two corresponding 8×8 blocks of chroma samples.

[0056] The intra-codec block uses spatial prediction to exploit the spatial correlation between pixels. Two partitions are defined: 16×16 and 4×4.

[0057] The inter-codec block uses temporal prediction instead of spatial prediction by estimating the motion between pictures. Motion can be estimated independently for a 16×16 macroblock or any of its sub-macroblock partitions (16×8, 8×16, 8×8, 8×4, 4×8, 4×4) (see Figure 2 ). Only one motion vector (MV) is allowed per sub-macroblock partition.

[0058] 2.1.2. Partition tree structure in HEVC

[0059] In HEVC, CTU is divided into CUs to accommodate various local characteristics using a quadtree structure represented as a codec tree. The decision to use inter-picture (temporal domain) or intra-picture (spatial domain) prediction to encode and decode a picture area is made at the CU level. Each CU can be further divided into one, two, or four PUs according to the PU partition type. Within a PU, the same prediction process is applied, and relevant information is sent to the decoder based on the PU. After obtaining the residual block by applying the prediction process based on the PU partition type, the CU can be partitioned into transform units (TUs) according to another quadtree structure similar to the codec tree of the CU. One of the key features of the HEVC structure is that it has multiple partitioning concepts, including CU, PU, ​​and TU.

[0060] In the following, various features involved in hybrid video codec using HEVC are highlighted as follows.

[0061] 1) Codec Tree Unit and Codec Tree Block (CTB) Structure: The analog structure in HEVC is the Codec Tree Unit (CTU), which has a size selected by the encoder and can be larger than a traditional macroblock. A CTU consists of a luma CTB and a corresponding chroma CTB, as well as syntax elements. The size L×L of the luma CTB can be selected to be L=16, 32, or 64 samples, where larger sizes generally enable better compression. HEVC then supports splitting the CTB into smaller blocks using a tree structure and quadtree-like signaling.

[0062] 2) Codec Unit (CU) and Codec Block (CB): The quadtree syntax of a CTU specifies the size and position of its luma CB and chroma CB. The root of the quadtree is associated with the CTU. Therefore, the size of the luma CTB is the maximum supported size of the luma CB. The partitioning of the CTU into luma CB and chroma CB is jointly signaled. One luma CB and typically two chroma CBs together with the associated syntax form a codec unit (CU). A CTB may contain only one CU or may be partitioned to form multiple CUs, and each CU has an associated partitioning of prediction units (PUs) and a tree of transform units (TUs).

[0063] 3) Prediction Unit and Prediction Block (PB): The decision to use inter-picture or intra-picture prediction to encode or decode a picture region is made at the CU level. The root of the PU partition structure is at the CU level. Depending on the basic prediction type decision, the luma CB and chroma CB can then be further divided in size and predicted based on luma and chroma prediction blocks (PB). HEVC supports variable PB sizes from 64×64 down to 4×4 samples. Figure 3An example of allowed PBs for an M×M CU is shown.

[0064] 4) TU and transform blocks: The prediction residuals are encoded and decoded using block transforms. The root of the TU tree structure is at the CU level. The luma CB residual can be the same as the luma transform block (TB), or can be further divided into smaller luma TBs. The same applies to chroma TBs. Integer basis functions similar to the functions of the Discrete Cosine Transform (DCT) are defined for square TB sizes 4×4, 8×8, 16×16, and 32×32. For the 4×4 transform of the luma intra picture prediction residual, an integer transform derived from the form of the Discrete Sine Transform (DST) can be specified instead.

[0065] 2.1.2.1. Tree structure partitioning into transform blocks and units

[0066] For residual codecs, CBs can be recursively partitioned into transform blocks (TBs). The partitioning is signaled by a residual quadtree. Only square CB and TB partitions are specified, where blocks can be recursively divided into quadrants, such as Figure 4A-4B As shown. For a given luma CB of size M×M, a flag signals whether it is divided into four blocks of size M / 2×M / 2. If further division is possible, as signaled by the maximum depth of the residual quadtree indicated in the SPS, a flag indicating whether it is divided into four quadrants is assigned to each quadrant. The leaf node blocks generated by the residual quadtree are transform blocks that are further processed by transform coding and decoding. The encoder indicates the maximum and minimum luma TB sizes it will use. When the CB size is larger than the maximum TB size, the division is implicit. When the division will result in a luma TB size smaller than the indicated minimum value, no division is implicit. Except when the luma TB size is 4×4 (in this case, a single 4×4 chroma TB is used for the area covered by four 4×4 luma TBs), the chroma TB size is half the luma TB size in each dimension. In the case of an intra picture predicted CU, the decoded samples of the nearest neighboring TB (inside or outside the CB) are used as reference data for intra picture prediction.

[0067] Contrary to previous standards, the HEVC design allows a TB to span multiple PBs of an inter-picture predicted CU to maximize the potential codec efficiency benefits of the quadtree-structured TB partitioning.

[0068] 2.1.2.2. Parent Node and Child Node

[0069] CTB is divided according to a quadtree structure, and the nodes of the quadtree structure are codec units. A plurality of nodes in the quadtree structure include leaf nodes and non-leaf nodes. A leaf node has no child nodes in the tree structure (i.e., the leaf node is not further divided). Non-leaf nodes include the root node of the tree structure. The root node corresponds to an initial video block (e.g., CTB) of video data. For each corresponding non-root node in a plurality of nodes, the corresponding non-root node corresponds to a video block, which is a sub-block of a video block corresponding to a parent node in the tree structure of the corresponding non-root node. Each corresponding non-leaf node in a plurality of non-leaf nodes has one or more child nodes in the tree structure.

[0070] 2.1.3. Quadtree plus binary tree block structure with larger CTU in JEM

[0071] To explore future video codec technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and put them into reference software called the Joint Exploration Model (JEM).

[0072] 2.1.3.1.QTBT block segmentation structure

[0073] Different from HEVC, the QTBT structure removes the concept of multiple partition types, that is, it removes the separation of CU, PU and TU concepts, and supports more flexibility in CU partition shape. In the QTBT block structure, a CU can have a square or rectangular shape. Figure 5A-Figure 5B As shown, the codec tree unit (CTU) is first partitioned by a quadtree structure. The quadtree leaf nodes are further partitioned by a binary tree structure. There are two types of partitioning in the binary tree partitioning, symmetrical horizontal partitioning and symmetrical vertical partitioning. The binary tree leaf nodes are called codec units (CUs), and the segmentation is used for prediction and transform processing without any further partitioning. This means that CU, PU, ​​and TU have the same block size in the QTBT codec block structure. In JEM, a CU sometimes consists of codec blocks (CBs) of different color components, for example, a CU contains one luminance CB and two chrominance CBs in the case of P slices and B slices in a 4:2:0 chrominance format, and sometimes consists of CBs of a single component, for example, a CU contains only one luminance CB or only two chrominance CBs in the case of an I slice.

[0074] The following parameters are defined for the QTBT segmentation scheme.

[0075] –CTU size: The root node size of the quadtree, which is the same concept as in HEVC.

[0076] –MinQTSize: Minimum allowed quadtree leaf node size.

[0077] –MaxBTSize: Maximum allowed binary tree root node size.

[0078] –MaxBTDepth: Maximum allowed binary tree depth.

[0079] –MinBTSize: Minimum allowed binary tree leaf node size.

[0080] In one example of a QTBT partitioning structure, the CTU size is set to 128×128 luma samples with two corresponding 64×64 chroma sample blocks, MinQTSize is set to 16×16, MaxBTSize is set to 64×64, MinBTSize (for both width and height) is set to 4×4, and MaxBTDepth is set to 4. Quadtree partitioning is first applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can have sizes from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the leaf quadtree node is 128×128, it will not be further divided by the binary tree because its size exceeds MaxBTSize (i.e., 64×64). Otherwise, the leaf quadtree node can be further divided by the binary tree. Therefore, the quadtree leaf node is also the root node of the binary tree, and its binary tree depth is 0. When the binary tree depth reaches MaxBTDepth (i.e., 4), no further division is considered. When the width of the binary tree node is equal to MinBTSize (i.e. 4), no further horizontal partitioning is considered. Similarly, when the height of the binary tree node is equal to MinBTSize, no further vertical partitioning is considered. The leaf nodes of the binary tree are further processed through the prediction and transformation processes without any further partitioning. In JEM, the maximum CTU size is 256×256 luma samples.

[0081] Figure 5A An example of block segmentation by using QTBT is shown, and Figure 5B The corresponding tree representation is shown. Solid lines indicate quadtree partitioning, and dashed lines indicate binary tree partitioning. In each partition (i.e., non-leaf) node of the binary tree, a flag is signaled to indicate which partition type (i.e., horizontal or vertical) is used, where 0 indicates horizontal partitioning and 1 indicates vertical partitioning. For quadtree partitioning, there is no need to indicate the partition type, because quadtree partitioning always divides the block horizontally and vertically to produce 4 sub-blocks of the same size.

[0082] In addition, the QTBT scheme supports the ability for luma and chroma to have separate QTBT structures. Currently, for P and B slices, the luma CTB and chroma CTB in one CTU share the same QTBT structure. However, for I slices, the luma CTB is split into CUs by the QTBT structure, and the chroma CTB is split into chroma CUs by another QTBT structure. This means that the CU in the I slice consists of the codec block of the luma component or the codec blocks of the two chroma components, and the CU in the P slice or B slice consists of the codec blocks of all three color components.

[0083] In HEVC, inter prediction of small blocks is restricted to reduce memory access for motion compensation, so that bidirectional prediction is not supported for 4×8 blocks and 8×4 blocks, and inter prediction is not supported for 4×4 blocks. In JEM's QTBT, these restrictions are removed.

[0084] 2.1.4. VVC Ternary Tree

[0085] As proposed, tree types other than quadtrees and binary trees are supported. In an embodiment, more than two ternary tree (TT) partitions are introduced, namely horizontal and vertical center-side ternary trees, such as Figure 6 As shown in (d) and (e).

[0086] Figure 6 (a) Quadtree partition 6 (b) Vertical binary tree partition 6 (c) Horizontal binary tree partition 6 (d) Vertical center side ternary tree partition 6 (e) Horizontal center side ternary tree partition.

[0087] There are two levels of trees, the region tree (quadtree) and the prediction tree (binary tree or ternary tree). The CTU is first segmented by the region tree (RT). The RT leaf can be further divided by the prediction tree (PT). The PT leaf can also be further divided by the PT until the maximum PT depth is reached. The PT leaf is the basic coding unit. For convenience, it is still called CU. The CU cannot be further divided. Prediction and transformation are applied to the CU in the same way as JEM. The entire segmentation structure is named "multi-type tree".

[0088] 2.2. Inter-frame prediction in HEVC / H.265

[0089] Each inter-predicted PU has motion parameters for one or two reference picture lists. The motion parameters include motion vectors and reference picture indexes. The use of one of the two reference picture lists can also be notified using inter_pred_idc signaling. The motion vector can be explicitly encoded and decoded as a delta relative to the predictor, and such a coding mode is called AMVP mode.

[0090] When a CU is encoded or decoded in skip mode, one PU is associated with the CU and there are no significant residual coefficients, no coded motion vector increments, or reference picture indices. The Merge mode is specified, whereby the motion parameters of the current PU are obtained from neighboring PUs, including spatial and temporal candidates. The Merge mode can be applied to any inter-predicted PU, not just to the Skip mode. An alternative to the Merge mode is explicit transmission of motion parameters, where each PU explicitly signals the motion vectors, the corresponding reference picture indices for each reference picture list, and the reference picture list usage.

[0091] When signaling indicates that one of the two reference picture lists is to be used, a PU is generated from a block of samples. This is called "unidirectional prediction". Unidirectional prediction can be used for both P slices and B slices.

[0092] When signaling indicates that both of the reference picture lists are to be used, a PU is generated from two sample blocks. This is called "bi-prediction". Bi-prediction is only available for B slices.

[0093] The following provides details about the inter prediction modes specified in HEVC. The description will start with the Merge mode.

[0094] 2.2.1.Merge mode

[0095] 2.2.1.1. Merge mode candidate derivation

[0096] When predicting a PU using Merge mode, an index pointing to an entry in the Merge candidate list is parsed from the bitstream and used to retrieve motion information. The construction of this list is specified in the HEVC standard and can be outlined according to the sequence of the following steps:

[0097] Step 1: Initial candidate derivation

[0098] Step 1.1: Spatial Candidate Derivation

[0099] Step 1.2: Redundancy check of spatial candidates

[0100] οStep 1.3: Time Domain Candidate Derivation

[0101] Step 2: Additional candidate insertions

[0102] οStep 2.1: Create bidirectional prediction candidates

[0103] οStep 2.2: Insert zero motion candidates

[0104] These steps are also Figure 7 Schematically depicted in . For spatial Merge candidate derivation, up to four Merge candidates are selected from candidates located at five different positions. For temporal Merge candidate derivation, up to one Merge candidate is selected from two candidates. Since a constant number of candidates for each PU is assumed at the decoder, additional candidates are generated when the number of candidates does not reach the maximum number of Merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, truncated unary binarization (TU) is used to encode the index of the best Merge candidate. If the size of the CU is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

[0105] Hereinafter, operations associated with the aforementioned steps are described in detail.

[0106] 2.2.1.2. Spatial Candidate Derivation

[0107] In the derivation of spatial domain Merge candidates, Figure 8 Up to four Merge candidates are selected from the candidates at the positions depicted in . The order of derivation is A1, B1, B0, A0 and B2. Position B2 is considered only when any PU at position A1, B1, B0, A0 is not available (for example, because it belongs to another slice or piece) or is intra-coded. After adding the candidate at position A1, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only pairs with Fig. 9 The arrows in link pairs, and candidates are added to the list only if the corresponding candidates for redundancy checking do not have the same motion information. Another source of duplicate motion information is a "second PU" associated with a partition other than 2N×2N. As an example, Figure 10A-10BThe second PU is depicted for the N×2N and 2N×N cases, respectively. When the current PU is partitioned into N×2N, the candidate at position A1 is not considered for list construction. In fact, adding this candidate would result in two prediction units having the same motion information, which is redundant for having only one PU in the codec unit. Similarly, position B1 is not considered when the current PU is partitioned into 2N×N.

[0108] 2.2.1.3. Time Domain Candidate Derivation

[0109] In this step, only one candidate is added to the list. Specifically, in the derivation of the temporal Merge candidate, a scaled motion vector is derived based on the collocated PU belonging to the picture with the minimum POC (Picture Order Count) difference with the current picture in a given reference picture list. The reference picture list to be used to derive the collocated PU is explicitly signaled in the slice header. The scaled motion vector of the temporal Merge candidate is obtained as shown by the dotted line in 13, which is scaled from the motion vector of the collocated PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal Merge candidate is set equal to zero. The actual implementation of the scaling process is described in the HEVC specification. For B slices, two motion vectors (one for reference picture list 0 and the other for reference picture list 1) are obtained and combined to produce a bidirectional prediction Merge candidate.

[0110] Fig.11 is an illustration of motion vector scaling for temporal merge candidates.

[0111] In the collocated PU(Y) belonging to the reference frame, the position of the temporal candidate is selected between candidates C0 and C1, such as Fig.12 As depicted in . If the PU at position C0 is not available, is intra-coded, or is outside the current CTU, position C1 is used. Otherwise, position C0 is used in the derivation of the time domain Merge candidate.

[0112] 2.2.1.4. Additional candidate insertion

[0113] In addition to the spatio-temporal Merge candidates, there are two additional types of Merge candidates: combined bi-directional prediction Merge candidates and zero Merge candidates. Combined bi-directional prediction Merge candidates are generated by utilizing spatio-temporal Merge candidates. Combined bi-directional prediction Merge candidates are only used for B slices. Combined bi-directional prediction candidates are generated by combining the first reference picture list motion parameters of an initial candidate with the second reference picture list motion parameters of another initial candidate. If the two tuples provide different motion hypotheses, they will form a new bi-directional prediction candidate. As an example, Fig.13 Depicted is the situation when two candidates in the original list (left) with mvL0 and refIdxL0 or mvL1 and refIdxL1 are used to create a combined bi-predictive Merge candidate that is added to the final list (right). There are many rules about the combinations that are considered to generate these additional Merge candidates.

[0114] Zero motion candidates are inserted to fill the remaining entries in the Merge candidate list and thus reach the MaxNumMergeCand capacity. These candidates have zero spatial displacement and reference picture index, which starts from zero and increases every time a new zero motion candidate is added to the list. The number of reference frames used by these candidates is 1 and 2 for unidirectional prediction and bidirectional prediction, respectively. Finally, no redundancy check is performed on these candidates.

[0115] 2.2.1.5. Motion Estimation Regions for Parallel Processing

[0116] To speed up the encoding process, motion estimation can be performed in parallel, thereby deriving motion vectors for all prediction units within a given region at the same time. Deriving Merge candidates from spatial neighborhoods may interfere with parallel processing, because one prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is completed. In order to ease the trade-off between coding efficiency and processing latency, HEVC defines a motion estimation region (MER), whose size is signaled in the picture parameter set using the "log2_parallel_merge_level_minus2" syntax element. When MER is defined, Merge candidates falling into the same region are marked as unavailable and are therefore not considered in list construction.

[0117] 7.3.2.3 Picture parameter set RBSP syntax

[0118] 7.3.2.3.1 Generic Picture Parameter Set RBSP Syntax

[0119]

[0120] log2_parallel_merge_level_minus2 plus 2 specifies the value of the variable Log2ParMrgLevel, which is used in the derivation of luma motion vectors for Merge mode as specified in clause 8.5.3.2.2 and in the derivation of spatial Merge candidates as specified in clause 8.5.3.2.3. The value of log2_parallel_merge_level_minus2 shall be in the range of 0 to CtbLog2SizeY-2 (inclusive).

[0121] The variable Log2ParMrgLevel is derived as follows:

[0122] Log2ParMrgLevel=log2_parallel_merge_level_minus2+2 (7-37)

[0123] NOTE 3 - The value of Log2ParMrgLevel indicates the built-in capability of parallel derivation of Merge candidate lists. For example, when Log2ParMrgLevel is equal to 6, the Merge candidate lists of all prediction units (PUs) and coding units (CUs) contained in a 64×64 block can be derived in parallel.

[0124] 2.2.2. Motion Vector Prediction in AMVP Mode

[0125] Motion vector prediction exploits the spatio-temporal correlation of motion vectors with neighboring PUs, which is used for explicit transmission of motion parameters. It builds a motion vector candidate list by first checking the availability of neighboring PU positions in the left and above time domain, removing redundant candidates and adding zero vectors to make the candidate list a constant length. The encoder can then select the best prediction value from the candidate list and send the corresponding index indicating the selected candidate. Similar to Merge index signaling, the index of the best motion vector candidate is encoded using truncated unary. The maximum value to be encoded in this case is 2 (see Fig.14 ). In the following sections, details about the derivation process of motion vector prediction candidates are provided.

[0126] 2.2.2.1. Derivation of motion vector prediction candidates

[0127] Fig.14 The process of deriving motion vector prediction candidates is outlined.

[0128] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For spatial motion vector candidate derivation, based on the location of Figure 8 The motion vectors of each PU at five different positions are depicted, and two motion vector candidates are finally derived.

[0129] For temporal motion vector candidate derivation, one motion vector candidate is selected from two candidates derived based on two different collocated positions. After generating the first spatio-temporal candidate list, duplicate motion vector candidates in the list are removed. If the number of potential candidates is greater than two, motion vector candidates whose reference picture index within their associated reference picture list is greater than 1 are removed from the list. If the number of spatio-temporal motion vector candidates is less than two, an additional zero motion vector candidate is added to the list.

[0130] 2.2.2.2. Spatial Motion Vector Candidates

[0131] In the derivation of the spatial motion vector candidate, from the position such as Figure 8 A maximum of two candidates are considered out of the five potential candidates for PU derivation at the depicted position. Those positions are the same as the positions of the motion merge. The derivation order for the left side of the current PU is defined as A0, A1, and scaling A0, scaling A1. The derivation order for the upper side of the current PU is defined as B0, B1, B2, scaling B0, scaling B1, scaling B2. For each side, there are therefore four cases that can be used as motion vector candidates, two of which do not require the use of spatial scaling, and two of which use spatial scaling. The four different cases are outlined below.

[0132] No airspace scaling

[0133] -(1) Same reference picture list and same reference picture index (same POC)

[0134] -(2) Different reference picture lists, but same reference pictures (same POC)

[0135] Airspace scaling

[0136] -(3) Same reference picture list, but different reference pictures (different POC)

[0137] -(4) Different reference picture lists, and different reference pictures (different POC)

[0138] First check for non-spatial scaling, then spatial scaling. Spatial scaling is considered when the POC is different between the reference picture of the neighboring PU and the reference picture of the current PU regardless of the reference picture list. If all PUs of the left candidate are unavailable or intra-coded, then scaling is allowed for the upper motion vector to help parallel derivation of the left and upper MV candidates. Otherwise, spatial scaling is not allowed for the upper motion vector.

[0139] In the spatial domain scaling process, the motion vectors of neighboring PUs are scaled in a similar way to the temporal domain scaling, such as Fig.15 The main difference is that the reference picture list and the index of the current PU are given as input; the actual scaling process is the same as that of temporal scaling.

[0140] 2.2.2.3. Temporal motion vector candidates

[0141] Except for the reference picture index derivation, all the processes used to derive the temporal merge candidate are the same as those used to derive the spatial motion vector candidate (see Fig.12 ). Signaling the reference picture index to the decoder.

[0142] 2.2.2.4.Signaling Notification of Merge / AMVP Information

[0143] For AMVP mode, four parts may be signaled in the bitstream, namely, prediction direction, reference index, MVD and mv prediction value candidate index (highlighted in the syntax table below), while for Merge mode, only the Merge index may need to be signaled.

[0144] Syntax table:

[0145] 7.3.6.1 Generic slice segment header syntax

[0146]

[0147] 7.3.8.6 Prediction Unit Syntax

[0148]

[0149] 7.3.8.9 Motion Vector Difference Syntax

[0150]

[0151] Semantics

[0152] five_minus_max_num_merge_cand specifies the maximum number of Merge MVP candidates supported in a stripe subtracted from 5. The maximum number of Merge MVP candidates, MaxNumMergeCand, is derived as follows:

[0153] MaxNumMergeCand=5-five_minus_max_num_merge_cand (2-1)

[0154] The value of MaxNumMergeCand must be in the range of 1 to 5 (inclusive).

[0155] merge_flag[x0][y0] specifies whether the inter prediction parameters of the current prediction unit are inferred from the neighboring inter prediction partitions. Array index x0, y0 specifies the position (x0, y0) of the top left luma sample of the considered prediction block relative to the top left luma sample of the picture.

[0156] When merge_flag[x0][y0] is not present, it is inferred as follows:

[0157] - If CuPredMode[x0][y0] is equal to MODE_SKIP, merge_flag[x0][y0] is inferred to be equal to 1.

[0158] Otherwise, merge_flag[x0][y0] is inferred to be equal to 0.

[0159] merge_idx[x0][y0] specifies the Merge candidate index of the Merge candidate list, where x0, y0 specify the position (x0, y0) of the upper left luma sample of the considered prediction block relative to the upper left luma sample of the picture.

[0160] 2.3. Inter-mode encoding and decoding in VVC

[0161] In VVC, translational motion and affine motion are handled differently. For translational motion, enhanced AMVP and Merge designs are adopted.

[0162] 2.3.1. History-based Motion Vector Prediction (HMVP)

[0163] Different from the Merge list design, in VVC, a history-based motion vector prediction (HMVP) method is adopted, in which the HMVP table is maintained to store motion information from the previous coded block. Such a method is also called a lookup table-based MVP method.

[0164] LUT-based motion vector prediction technology can be implemented in various embodiments to provide video codecs with higher codec efficiency, which uses one or more tables (e.g., lookup tables) storing at least one motion candidate to predict the motion information of a block. The lookup table is an example of a table that can be used to include motion candidates to predict the motion information of a block, and other implementations are also possible. Each LUT may include one or more motion candidates, each motion candidate being associated with corresponding motion information. The motion information of the motion candidate may include part or all of the prediction direction, reference index / picture, motion vector, LIC flag, affine flag, motion vector derivation (MVD) accuracy and / or MVD value. The motion information may also include block position information to indicate which block the motion information comes from.

[0165] In the following examples described for various embodiments, LUT-based motion vector prediction based on the disclosed technology is described, which can enhance existing and future video coding standards. Because LUT allows the encoding / decoding process to be performed based on historical data (e.g., blocks that have been processed), LUT-based motion vector prediction can also be referred to as a history-based motion vector prediction (HMVP) method. In the LUT-based motion vector prediction method, one or more tables with motion information from previously encoded and decoded blocks are maintained during the encoding / decoding process. These motion candidates stored in the LUT are called HMVP candidates. During the encoding / decoding of a block, the associated motion information in the LUT can be added to a motion candidate list (e.g., Merge / AMVP candidate list), and after encoding / decoding a block, the LUT can be updated. Then, the updated LUT is used to encode and decode subsequent blocks. That is, the updating of motion candidates in the LUT is based on the encoding / decoding order of the block.

[0166] In HMVP, the motion information of the previous codec is stored. The motion information of the previously coded block is defined as the HMVP candidate. Multiple HMVP candidates are stored in a table called the HMVP table, and the table is maintained on-the-fly during the encoding / decoding process. When encoding / decoding a new slice / LCU row / slice starts, the HMVP table is cleared. Whenever there is an inter-coded block and a non-sub-block, non-triangle prediction mode (non-TPM), the associated motion information is added to the last entry of the table as a new HMVP candidate. Fig.16 The overall encoding and decoding process is depicted in .

[0167] 2.3.1.1. Conventional Merge List Building Process

[0168] The construction of a regular Merge list (for translational motion) can be outlined according to the sequence of following steps:

[0169] Step 1: Derive spatial merge candidates

[0170] Step 1: Derive the temporal Merge candidate (TMVP, temporal motion vector prediction)

[0171] Step 2: Insert HMVP candidates

[0172] Step 3: Insert pairwise average candidates

[0173] Step 4: Default motion candidates

[0174] HMVP candidates can be used in the AMVP and Merge candidate list construction process. Fig.17The modified Merge candidate list construction process is depicted (highlighted in blue). When the Merge candidate list is not full after the TMVP candidate is inserted, the HMVP candidates stored in the HMVP table can be used to fill the Merge candidate list. Considering that a block usually has a higher correlation with the nearest neighboring blocks in terms of motion information, the HMVP candidates in the table are inserted in descending order of index. The last entry in the table is added to the list first, and the first entry is added to the end. Similarly, redundancy removal is applied to the HMVP candidates. Once the total number of available Merge candidates reaches the maximum number of Merge candidates allowed for signaling, the Merge candidate list construction process terminates.

[0175] Note that all spatial / temporal / HMVP candidates should be coded and decoded in non-IBC mode. Otherwise, they are not allowed to be added to the regular Merge candidate list.

[0176] The HMVP table contains up to 5 motion candidates, and each of them is unique.

[0177] 2.3.1.2. Pruning process

[0178] Candidates are added to the list only if the corresponding candidates for redundancy check do not have the same motion information. Such a comparison process is called pruning process.

[0179] The pruning process between spatial candidates depends on the TPM usage of the current block.

[0180] When the current block is not encoded or decoded in TPM mode (eg, conventional Merge, MMVD, CIIP), the HEVC pruning process (ie, five-pass pruning) of spatial Merge candidates is utilized.

[0181] 2.3.1.3.HMVP table update

[0182] After decoding an inter-coded block in non-TPM, non-sub-block mode, the HMVP table is updated. The motion information of the current block may be pruned first with all available HMVP candidates in the HMVP table.

[0183] A counter is assigned to the HMVP table, which records the number of available HMVP candidates in the table. If the HMVP table is reset, it is initialized to 0.

[0184] There are three situations during the update process, how to handle each situation is described below:

[0185] 1. If an identical candidate is found, the redundant candidate is removed from the table and all remaining candidates are pushed forward with an index minus 1. The motion information of the current block is then added to the HMVP table after all existing candidates. The counter remains unchanged.

[0186] 2. If no identical candidate is found and the table is not full, the motion information of the current block is added to the HMVP table after all existing candidates. The counter is increased by 1.

[0187] 3. If no identical candidate is found and the table is full, the first entry in the HMVP table is removed and all remaining candidates are pushed forward with an index minus 1. The motion information of the current block is added to the HMVP table after all existing candidates. The counter remains unchanged.

[0188] The first and last two cases are respectively Figure 18A-18B and Figure 19A-19B Further described in.

[0189] Fig.18A Shown is the case when redundant candidates are found, and the LUT is full before adding new motion candidates.

[0190] Fig.18B Shown is the case when a redundant candidate is found and the LUT is not full before adding a new motion candidate.

[0191] Figure 18A-18B A HMVP update method based on redundancy removal (removing one redundant motion candidate) is shown.

[0192] ( Fig.19A ) no redundant candidates are found and the LUT is full before adding new motion candidates, while Fig.19B No redundant candidates are found and the LUT is not full before adding new motion candidates.

[0193] The relevant specifications are defined as follows:

[0194] 8.5.2.16 Update process of the candidate list of motion vector prediction values ​​based on history

[0195] The input to this process is:

[0196] – Luma motion vectors mvL0 and mvL1 with 1 / 16 fractional sample accuracy,

[0197] – reference indices refIdxL0 and refIdxL1,

[0198] – The prediction list uses the flags predFlagL0 and predFlagL1,

[0199] – Bidirectional prediction weight index bcwIdx,

[0200] – Half-sample interpolation filter index hpelIfIdx.

[0201] The MVP candidate hMvpCand includes luma motion vectors mvL0 and mvL1, reference indexes refIdxL0 and refIdxL1, prediction list utilization flags predFlagL0 and predFlagL1, bidirectional prediction weight index bcwIdx, and half-sample interpolation filter index hpelIfIdx.

[0202] The candidate list HmvpCandList is modified with the candidate hMvpCand by the following ordered steps:

[0203] 1. The variable identicalCandExist is set equal to FALSE, and the variable removeIdx is set equal to 0.

[0204] 2. When NumHmvpCand is greater than 0, for each index hMvpIdx (where hMvpIdx = 0..NumHmvpCand-1), the following steps apply until identicalCandExist equals TRUE:

[0205] – When hMvpCand and HmvpCandList[hMvpIdx] have the same motion vector and the same reference index, identicalCandExist is set equal to TRUE and removeIdx is set equal to hMvpIdx.

[0206] 3. The candidate list HmvpCandList is updated as follows:

[0207] – If identicalCandExist is equal to TRUE or NumHmvpCand is equal to 5, the following applies:

[0208] – For each index i, where (i=(removeIdx+1)..(NumHmvpCand-1)), HmvpCandList[i-1] is set equal to HmvpCandList[i].

[0209] –HmvpCandList[NumHmvpCand-1] is set equal to hMvpCand.

[0210] – Otherwise (identicalCandExist is equal to FALSE and NumHmvpCand is less than 5), the following applies:

[0211] –HmvpCandList[NumHmvpCand++] is set equal to hMvpCand.

[0212] 2.3.2. AMVP list construction for translational motion

[0213] HMVP was introduced to the AMVP list without pruning.

[0214] 2.4. Intra-block copy (IBC) mode in VVC

[0215] In IBC, the AMVP and Merge lists are shared, that is, the same construction process is used. For 4x4 blocks, only HMVP candidates and zero block vectors (BV) in the IBC HMVP table are added to the candidate list. For other cases, two spatial IBC candidates, HMVP candidates, and zero BV are added in sequence in the case of partial pruning.

[0216] After decoding the IBC codec block, the IBC HMVP table is updated accordingly, in a similar manner to the updating process of the HMVP table for inter mode.

[0217] The corresponding specifications are provided below:

[0218] 8.5.2.6 Updating the History-Based Block Vector Prediction Candidate List

[0219] The input to this process is:

[0220] – Luma block vector bvL, with 1 / 16 fractional sample accuracy.

[0221] The candidate list HmvpIbcCandList is modified by the following ordered steps:

[0222] 1. The variable identicalCandExist is set equal to FALSE, and the variable removeIdx is set equal to 0.

[0223] 2. When NumHmvpIbcCand is greater than 0, for each index hMvpIdx (where hMvpIdx = 0..NumHmvpIbcCand-1), the following steps apply until identicalCandExist is equal to TRUE:

[0224] – When bvL is equal to HmvpIbcCandList[hMvpIdx], identicalCandExist is set equal to TRUE and removeIdx is set equal to hMvpIdx.

[0225] 3. The candidate list HmvpIbcCandList is updated as follows:

[0226] – If identicalCandExist equals TRUE or NumHmvpIbcCand equals 5, the following applies:

[0227] – For each index i, (where i = (removeIdx+1)..(NumHmvpIbcCand-1)), HmvpIbcCandList[i-1] is set equal to HmvpIbcCandList[i].

[0228] –HmvpIbcCandList[NumHmvpIbcCand-1] is set equal to bvL.

[0229] – Otherwise (identicalCandExist is equal to FALSE and NumHmvpIbcCand is less than 5), the following applies:

[0230] –HmvpIbcCandList[NumHmvpIbcCand++] is set equal to bvL.

[0231] 2.5. Codec Structure in AVS2

[0232] Similar to HEVC, AVS2 also adopts the concept of codec / prediction / transform structure based on CU, PU and TU. First, the picture is divided into the largest codec unit (LCU), which consists of 2N×2N samples. An LCU can be a single CU, or it can be divided into four smaller CUs in a quadtree partitioning structure; Fig. 20 As shown, the CU can be recursively divided until it reaches the minimum CU size limit. Once the CU hierarchical tree is divided, the leaf node CU can be further divided into PUs. PU is the basic unit for intra-frame and inter-frame prediction, and allows multiple different shapes to encode irregular image patterns, such as Fig. 20 shown.

[0233] Fig. 20 (a) The largest possible recursive CU structure in AVS2 is shown. (LCU size = 64, maximum layer depth = 4). (b) Possible PU partitions for skip, intra mode, and inter mode in AVS2, including symmetric and asymmetric prediction (d = 1, 2 for intra prediction; and d = 0, 1, 2 for inter prediction).

[0234] 2.6. Inter-frame prediction in AVS3

[0235] Similar to HEVC, there are two inter-frame codec frame types, P-frames and B-frames. P-frames are forward predicted frames using a single reference picture, while B-frames are bidirectionally predicted frames including forward, backward, bidirectional prediction, and symmetric prediction using two reference frames. In B-frames, in addition to the traditional forward, backward, bidirectional, and skip / direct prediction modes, symmetric prediction is defined as a special bidirectional prediction mode, in which only one forward motion vector (MV) is encoded and decoded, and the backward MV is derived from the forward MV based on the picture order count (POC) distance. The symmetric mode can effectively represent the linear motion model of the object.

[0236] 2.6.1. Skip / Direct Mode Based on Multi-Motion Model

[0237] The motion information of the SKIP mode for the current block is derived from the previously decoded block, and no residual information is encoded. Similar to the SKIP mode, the DIRECT mode has no motion information to be sent, while the prediction residual and mode information are sent.

[0238] As an extension of the traditional SKIP and DIRECT modes in which motion information from only one time-domain prediction block (CuSubtypeIdx equals 0) is utilized, four additional motion model-based SKIP and DIRECT modes are introduced in AVS2 and AVS3, namely, bidirectional DIRECT / SKIP mode (CuSubtypeIdx equals 1), symmetric DIRECT / SKIP mode (CuSubtypeIdx equals 2), backward DIRECT / SKIP mode (CuSubtypeIdx equals 3) and forward DIRECT / SKIP mode (CuSubtypeIdx equals 4).

[0239] A priority-based motion information derivation method is designed, which takes into account the motion model (prediction direction) of the block. A higher priority is assigned to the motion information of neighboring blocks that have the same motion model as the current block.

[0240] For each of the four additional motion model-based skip / direct modes, the motion information derivation process can be divided into three steps that are performed sequentially.

[0241] 1) Motion model matching search: Fig.21 As shown, the initial process of finding neighboring blocks with the same motion model as the current block at positions F, G, C, A, B, D is performed in this order. Once the first block that shares the same motion model as the current block is found, the motion information of that block is assigned to the current block.

[0242] 2) Manual search: In the absence of neighboring blocks that share the same motion model as the current block (ie, no results were found from step 1), a second process of manually constructing the motion information of the current block is performed. This process varies from one motion model to another.

[0243] a. For bidirectional DIRECT / SKIP mode, how many spatial neighbors are forward coded (by N fw ) and how many spatial neighbors are there for subsequent encoding and decoding (denoted by N bw If N fw and N bw If both are equal to or greater than 1, the combination of motion information of the first forward and first backward prediction blocks is assigned to the current block. The search order is the same as the first step, i.e., starting from positions F, G, C, A, B, and D.

[0244] b. For symmetrical DIRECT / SKIP mode, perform the following steps in order:

[0245] – Count how many spatial neighbors are using bidirectional encoding and decoding, represented by N bi express

[0246] – If N bi If equal to or greater than 2, the motion information of the adjacent bidirectional prediction block of the last block in the scanning order of F, G, C, A, B, and D (equal to the first block in the scanning order of D, B, A, C, G, and F) is assigned to the current block.

[0247] – Otherwise (if N bi Less than 2), if N bw Equal to or greater than 1, where N bw Indicates how many spatial neighbors are decoded backward, the backward motion vector (denoted by MvE1) of the motion information of the first backward prediction block is assigned to the current block, and the forward motion vector MvE0 is set equal to Clip3 (-32768, 32767, -MvE1). The search order is the same as the first step, that is, starting from positions F, G, C, A, B, and D.

[0248] – Otherwise (if N bi Less than 2 and N bw Less than 1), if N fw Equal to or greater than 1, where N fw Indicates how many spatial neighbors are forward encoded, the forward motion vector of the motion information of the first forward prediction block (denoted by MvE0) is assigned to the current block, and the backward motion vector MvE1 is set equal to Clip3 (-32768, 32767, -MvE0). The search order is the same as the first step, that is, starting from positions F, G, C, A, B, and D.

[0249] c. For the backward DIRECT / SKIP mode, count how many spatial neighbors are bidirectionally encoded and decoded, and use N bi The backward motion information of the last neighboring bidirectional predicted block is assigned to the current block. The search order is the same as the first step, i.e., starting from positions F, G, C, A, B, and D.

[0250] d. For the forward DIRECT / SKIP mode, count how many spatial neighbors are encoded and decoded in bidirectional mode, and use N bi The forward motion information of the last neighboring bidirectional predicted block is assigned to the current block. The search order is the same as the first step, i.e., starting from positions F, G, C, A, B, and D.

[0251] 3) Default MV construction: This step is called only when both step 1) and step 2) cannot find a usable motion vector. In AVS2 / AVS3, the following applies:

[0252] a. For bidirectional / symmetric skip / direct mode, set zero motion vector for both forward and backward.

[0253] b. For backward DIRECT / SKIP mode, the backward motion vector is set to zero MV, ie, (0,0).

[0254] c. For forward DIRECT / SKIP mode, the forward motion vector is set to zero MV, ie, (0,0).

[0255] Since there are multiple SKIP and DIRECT modes, some side information needs to be brought in when signaling the motion models of SKIP and DIRECT modes. In the AVS2 reference software RD3.0, the rate-distortion optimization method is used to select the mode of the coding block. When calculating the RD cost of the new SKIP and DIRECT modes, the side information is taken into account to produce a more accurate cost value.

[0256] 2.6.2. HMVP of AVS3

[0257] The basic idea of ​​HMVP design in AVS3 and VVC is the same. However, unlike VVC, the HMVP candidate index in the merge / amvp list changes dynamically due to pruning between spatial / temporal candidates. In AVS3, the HMVP candidate is fixed to start from an index equal to 4 to 11 for B slices and 2 to 9 for P slices.

[0258] Table 1. Sub-CU types of P pictures

[0259]

[0260] Table 2. Sub-CU types of B pictures

[0261]

[0262] The update process is the same as in VVC.

[0263] 3. Examples of technical problems solved by the disclosed technical solutions

[0264] The current design of the HMVP update process has the following issues:

[0265] 1. It follows the assumption that candidates added earlier have lower chances of being selected as MVPs for subsequent blocks. However, this is not always true.

[0266] 4. List of Techniques and Embodiments

[0267] The following list should be considered as an example to explain the general concept. These items should not be interpreted in a narrow way. In addition, these items can be combined in any way. The combination between this IDF and other IDFs also applies.

[0268] In this document, the HMVP table is a table that stores candidates, where the candidate is not limited to motion candidates (e.g., block vector (BV) with prediction direction, reference index / picture, motion vector, LIC flag, affine flag, MVD precision, MVD value, block position information, IBC codec block), it can also be an intra mode candidate or others.

[0269] Assume that the HMVP table size is denoted by L, and each candidate in the table is associated with an index in the range [0, L-1].

[0270] 1. During the HMVP table update process, instead of removing the first (earliest) added candidate in the HMVP table (e.g., the candidate index may be set to 0), it is proposed to remove the Kth entry in the HMVP table (i.e., the HMVP candidate with index equal to K), where K is a non-zero integer value.

[0271] a. In one example, K is predefined, for example, it is set to 1.

[0272] b. In one example, K depends on the HMVP table size, ie, the maximum number of candidates to be stored in the HMVP table.

[0273] c. In one example, K depends on how the HMVP table is utilized.

[0274] i. In one example, when the HMVP table for the inter-codec block is updated, K may be set to K0, and when the HMVP table for the IBC codec block is updated, K may be set to K1, where K0 is not equal to K1.

[0275] ii. In one example, when updating the HMVP table for an intra-frame codec block, K may be set equal to K2. K2 may not be equal to K1 or / and K0.

[0276] d. In one example, K depends on codec information such as block dimensions, block position, how many times the table has been updated, and the parity of the number of times the table has been updated.

[0277] e. In one example, K may be signaled in the bitstream, such as at a sequence / video / picture / slice / slice / sub-picture / brick / other video processing unit level.

[0278] i. In one example, K may be signaled in SPS / PPS / picture header / slice header / VPS / DPS.

[0279] ii. In one example, zero is excluded from the possible values ​​to be signaled for K.

[0280] f. In one example, for each entry in the HMVP table, a variable may be further assigned, such as to indicate the frequency of the candidate.

[0281] i. Furthermore, alternatively, the determination of which entry to remove from the table may be made based on a variable (eg, the determination of K).

[0282] g. In one example, K can be adaptively changed on the fly.

[0283] i. In one example, for a first video region, K may be set to a first value, and for a second video region, K may be set to a second value, wherein the first value and the second value are different.

[0284] ii. Furthermore, alternatively, the first value or the second value may be equal to 0.

[0285] h. Furthermore, alternatively, the above method may be applied under certain conditions.

[0286] i. In one example, when updating the HMVP table with a new candidate, if the table is full and there are no redundant candidates in the HMVP table (ie, the candidate in the HMVP table is the same or similar to the new candidate), the Kth entry in the HMVP table is removed.

[0287] ii. In one example, when updating the HMVP table with a new candidate, if there is a redundant candidate in the HMVP table (ie, the candidate in the HMVP table is the same or similar to the new candidate) and the redundant candidate is associated with an index equal to 0, the Kth entry in the HMVP table is removed.

[0288] 2. How to update the HMVP table with new candidates may depend on where the redundant (or similar) candidates are in the table (eg, the index of the redundant candidate).

[0289] a. In one example, if a redundant candidate is associated with an index equal to N (eg, N=0 or L-1), the redundant candidate may first be modified and stored in the HMVP table.

[0290] i. In one example, the motion vector / BV associated with the redundant candidate may be modified by adding one or more offsets or being scaled.

[0291] ii. In one example, in the updated HMVP table, the modified redundant candidate may be associated with the same index (ie, N).

[0292] 3. When a new candidate is used to update the HMVP table, instead of always adding the new candidate after all existing candidates after pruning, it is proposed to add the new candidate to the Kth last entry (eg, index equal to L-1-K) that is not the last entry.

[0293] a. In one example, K is less than or equal to T, where T is the number of candidates already in the table.

[0294] b. How to add new candidates may depend on the index of the redundant HMVP candidates in the HMVP table.

[0295] c. In one example, K is predefined, for example, 1.

[0296] i. In one example, a new candidate is added as a candidate with index set to (L-2), where K=1.

[0297] d. In one example, K depends on the HMVP table size, ie, the maximum number of candidates to be stored in the HMVP table.

[0298] e. In one example, K depends on how the HMVP table is utilized.

[0299] i. In one example, when the HMVP table for the inter-codec block is updated, K may be set to K0, and when the HMVP table for the IBC codec block is updated, K may be set to K1, where K0 is not equal to K1.

[0300] ii. In one example, when updating the HMVP table for an intra-frame codec block, K may be set equal to K2. K2 may not be equal to K1 or / and K0.

[0301] f. In one example, K depends on codec information such as block dimensions, block position, how many times the table has been updated, and the parity of the number of times the table has been updated.

[0302] g. In one example, K may be signaled in the bitstream, such as at a sequence / video / picture / slice / slice / sub-picture / tile / other video processing unit level.

[0303] i. In one example, K may be signaled in SPS / PPS / picture header / slice header / VPS / DPS.

[0304] h. In one example, for each entry in the HMVP table, a variable may be further assigned, such as to indicate the frequency of the candidate.

[0305] i. Furthermore, alternatively, the determination of which entry to remove from the table may be made based on a variable (eg, the determination of K).

[0306] i. In one example, K can be adaptively changed on the fly.

[0307] i. In one example, for a first video region, K may be set to a first value, and for a second video region, K may be set to a second value, wherein the first value and the second value are different.

[0308] ii. Furthermore, alternatively, the first value or the second value may be equal to 0.

[0309] j. In one example, K may depend on the pruning result, for example, whether there is an identical or similar candidate in the HMVP table before being updated.

[0310] 4. The rules for updating the HMVP table may vary from one video processing unit to another.

[0311] a. In one example, for the first rule, pruning is applied, and for the second rule, non-pruning is applied.

[0312] b. In one example, for the first rule, partial pruning is applied (i.e., the new candidate to be added is compared with some of the existing candidates in the table), and for the second rule, full pruning is applied (i.e., the new candidate to be added is compared with all the existing candidates in the table).

[0313] c. In one example, it may depend on how the HMVP table will be used, eg, for blocks in prediction mode utilizing the HMVP table.

[0314] d. In one example, the codec information may include codec mode information.

[0315] i. In one example, when updating the HMVP table for the inter-codec block, the above method or the existing method (according to the adaptive order of the pruning results) can be applied, and when updating the HMVP table for the IBC codec block, the FIFO rule can be applied.

[0316] 5. Whether and / or how the above bullet points are applied may depend on the codec information.

[0317] a. In one example, the codec information may include codec mode information.

[0318] i. In one example, when updating the HMVP table for an inter-codec block, the above method may be applied, and when updating the HMVP table for an IBC-codec block, the existing method may be applied.

[0319] 5. Example of Embodiment

[0320] 5.1 Example of a diagram depicting the HMVP table update process

[0321] exist Fig. 22C In, the HMVP table update process is modified for the case when the table is full and no redundant candidates are found among the existing candidates in the HMVP table.

[0322] Fig.22A The case is shown where a redundant candidate is found and the LUT is full before adding a new motion candidate.

[0323] Fig. 22B The case is shown where a redundant candidate is found and the LUT is not full before adding a new motion candidate.

[0324] Fig. 22C The case is shown where no redundant candidates are found and the LUT is full before adding new motion candidates.

[0325] Fig.22D The case is shown where no redundant candidates are found and the LUT is not full before adding a new motion candidate.

[0326] exist Figure 23A-23F In the example, the HMVP table update process (corresponding to K=1 in bullet point 1 of Section 4) is modified for the following cases:

[0327] When the table is full and no redundant candidates are found among the existing candidates in the HMVP table, refer to Fig.23E .

[0328] When the table is full and a redundant candidate is found as the first candidate in the HMVP table (HMVP0), refer to Fig.23C .

[0329] When the table is not full and the redundant candidate is found to be the first candidate in the HMVP table (HMVP0), refer to Fig.23D .

[0330] Fig.23A The case is shown where a redundant candidate is found (index not equal to 0) and the LUT is full before adding a new motion candidate.

[0331] Fig. 23B The case is shown where a redundant candidate is found (index not equal to 0) and the LUT is not full before adding a new motion candidate.

[0332] Fig.23C The case is shown where a redundant candidate is found (index equal to 0) and the LUT is full before adding a new motion candidate.

[0333] Fig.23D The case is shown where a redundant candidate is found (index equal to 0) and the LUT is not full before adding a new motion candidate.

[0334] Fig.23E The case is shown where no redundant candidates are found and the LUT is full before adding new motion candidates.

[0335] Fig.23F The case is shown where no redundant candidates are found and the LUT is not full before adding a new motion candidate.

[0336] 5.2. Example #1 based on AVS3 specification

[0337] Deleted text is marked with a strikethrough, and newly added text is highlighted.

[0338] 9.16 Update historical sports information table

[0339] After completing the decoding of the current prediction unit, if the current prediction unit is an intra-frame prediction unit and not an affine prediction unit, when NumOfHmvpCand is greater than 0, update the historical motion information table HmvpCandidateList according to the motion information of the current prediction block; otherwise, do not perform the operations defined in this clause.

[0340] a) Initialize hmvpIdx to 0.

[0341] b) If CntHmvp is equal to 0, HmvpCandidateList[CntHmvp] is equal to the motion information of the current prediction unit, and CntHmvp is increased by 1.

[0342] c) Otherwise, determine whether the motion information of the current prediction block is the same as HmvpCandidateList[hmvpIdx] according to the method defined in 9.15:

[0343] 1) If the motion information is the same, execute step d); otherwise, add 1 to hmvpIdx.

[0344] 2) If hmvpIdx is less than CntHmvp, execute step c); otherwise, execute step d).

[0345] d) If hmvpIdx is less than CntHmvp, then:

[0346] 1) i from hmvpIdx to CntHmvp-1, let HmvpCandidateList[i] equal to HmvpCandidateList[i+1];

[0347] 2) HmvpCandidateList[CntHmvp-1] is equal to the motion information of the current prediction unit. Otherwise, if hmvpIdx is equal to CntHmvp and CntHmvp is equal to NumOfHmvpCand, then:

[0348] 1) i from 1 To CntHmvp-1, let HmvpCandidateList[i] equal to HmvpCandidateList[i+1];

[0349] 2) HmvpCandidateList[CntHmvp-1] is equal to the motion information of the current prediction unit. Otherwise, if hmvpIdx is equal to CntHmvp and CntHmvp is less than NumOfHmvpCand, HmvpCandidateList[CntHmvp] is equal to the motion information of the current prediction unit, and CntHmvp is increased by 1.

[0350] 5.3. Example 1 based on VVC specification

[0351] 8.5.2.6 Update process of the candidate list of block vector prediction values ​​based on history The input of this process is:

[0352] – Luma block vector bvL, with 1 / 16 fractional sample accuracy.

[0353] The candidate list HmvpIbcCandList is modified by the following ordered steps:

[0354] 4. The variable identicalCandExist is set equal to FALSE, and the variable removeIdx is set equal to 0.

[0355] 5. When NumHmvpIbcCand is greater than 0, for each index hMvpIdx (where hMvpIdx = 0..NumHmvpIbcCand-1), the following steps apply until identicalCandExist is equal to TRUE:

[0356] – When bvL is equal to HmvpIbcCandList[hMvpIdx], identicalCandExist is set equal to TRUE and removeIdx is set equal to hMvpIdx.

[0357] 6. The candidate list HmvpIbcCandList is updated as follows:

[0358] – If identicalCandExist is equal to FALSE and NumHmvpIbcCand is equal to 5, then removeIdx is Set to 1.

[0359] – If identicalCandExist equals TRUE or NumHmvpIbcCand equals 5, the following applies:

[0360] – For each index i, (where i = (removeIdx+1)..(NumHmvpIbcCand-1)), HmvpIbcCandList[i-1] is set equal to HmvpIbcCandList[i].

[0361] –HmvpIbcCandList[NumHmvpIbcCand-1] is set equal to bvL.

[0362] – Otherwise (identicalCandExist is equal to FALSE and NumHmvpIbcCand is less than 5), the following applies:

[0363] –HmvpIbcCandList[NumHmvpIbcCand++] is set equal to bvL.

[0364] 8.5.2.16. Update process of the history-based motion vector prediction candidate list

[0365] The input to this process is:

[0366] – luma motion vectors mvL0 and mvL1 with 1 / 16 fractional sample accuracy,

[0367] – reference indices refIdxL0 and refIdxL1,

[0368] – The prediction list uses the flags predFlagL0 and predFlagL1,

[0369] – Bidirectional prediction weight index bcwIdx,

[0370] – Half-sample interpolation filter index hpelIfIdx.

[0371] The MVP candidate hMvpCand includes luma motion vectors mvL0 and mvL1, reference indexes refIdxL0 and refIdxL1, prediction list utilization flags predFlagL0 and predFlagL1, bidirectional prediction weight index bcwIdx, and half-sample interpolation filter index hpelIfIdx.

[0372] The candidate list HmvpCandList is modified with the candidate hMvpCand by the following ordered steps:

[0373] 1. The variable identicalCandExist is set equal to FALSE, and the variable removeIdx is set equal to 0.

[0374] 2. When NumHmvpCand is greater than 0, for each index hMvpIdx (where hMvpIdx = 0..NumHmvpCand-1), the following steps apply until identicalCandExist equals TRUE:

[0375] – When hMvpCand and HmvpCandList[hMvpIdx] have the same motion vector and the same reference index, identicalCandExist is set equal to TRUE and removeIdx is set equal to hMvpIdx.

[0376] 3. The candidate list HmvpCandList is updated as follows:

[0377] – If identicalCandExist is equal to FALSE and NumHmvpIbcCand is equal to 5, then removeIdx is Set to 1.

[0378] – If identicalCandExist is equal to TRUE or NumHmvpCand is equal to 5, the following applies:

[0379] – For each index i, where (i=(removeIdx+1)..(NumHmvpCand-1)), HmvpCandList[i-1] is set equal to HmvpCandList[i].

[0380] –HmvpCandList[NumHmvpCand-1] is set equal to hMvpCand.

[0381] – Otherwise (identicalCandExist is equal to FALSE and NumHmvpCand is less than 5), the following applies:

[0382] –HmvpCandList[NumHmvpCand++] is set equal to hMvpCand.

[0383] Fig.24 1 is a block diagram illustrating an example video processing system 1900 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of system 1900. System 1900 may include an input 1902 for receiving video content. The video content may be received in a raw or uncompressed format, such as 8 or 10 bit multi-component pixel values, or may be in a compressed or encoded format. Input 1902 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, a Passive Optical Network (PON), and the like, and wireless interfaces such as Wi-Fi or cellular interfaces.

[0384] System 1900 may include a codec component 1904 that can implement various codecs or encoding methods described in this document. Codec component 1904 can reduce the average bit rate of the video from input 1902 to the output of codec component 1904 to generate a codec representation of the video. Codec technology is therefore sometimes referred to as video compression or video transcoding technology. The output of codec component 1904 can be stored or sent via a communication connection such as represented by component 1906. The bitstream (or codec) representation of the storage or communication transmission of the video received at input 1902 can be used by component 1908 to generate pixel values ​​or transmit to a displayable video of display interface 1910. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "codec" operations or tools, it will be understood that the codec tool or operation is used at the encoder, and the corresponding decoding tool or operation of the inverse encoding result will be performed by the decoder.

[0385] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or Displayport, etc. Examples of storage interfaces include SATA (Serial Advanced Technology Attachment), PCI, IDE interfaces, etc. The technology described in this document may be embodied in various electronic devices, such as mobile phones, laptops, smart phones, or other devices capable of performing digital data processing and / or video display.

[0386] Fig.25 3600 is a block diagram of a video processing device 3600. Device 3600 can be used to implement one or more of the methods described herein. Device 3600 can be embodied in a smart phone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. Device 3600 may include one or more processors 3602, one or more memories 3604, and video processing hardware 3606. (Multiple) processors 3602 can be configured to implement one or more methods described in this document. Memory (multiple memories) 3604 can be used to store data and code for implementing the methods and techniques described herein. Video processing hardware 3606 can be used to implement some of the techniques described in this document in hardware circuits.

[0387] Next, a list of preferred solutions for some embodiments is provided.

[0388] The following solution illustrates an example embodiment of the techniques discussed in the previous section (eg, bullet 1).

[0389] 1. A method of video processing (eg, Fig.26 ), comprising: performing (2602) a conversion between a video block of a video and a bitstream representation of the video; and selectively updating (2604) a table of motion candidates for another video block by removing a Kth entry of the table, where K is a positive integer based on a rule. For example, the rule specifies conditions under which the update is completed and conditions under which the update is not performed (e.g., when there are no new candidates to be added).

[0390] 2. The method according to solution 1, wherein the rule specifies K=1.

[0391] 3. The method according to solution 1, wherein the rule specifies that K depends on the size of the table.

[0392] 4. The method of solution 1, wherein the rule specifies that K depends on characteristics of the transformation of the video block.

[0393] 5. The method of solution 1, wherein the rule specifies the conditions for performing the update.

[0394] 6. The method of solution 5, wherein the condition is that the table has no redundant candidates similar to the new candidate, and the rule specifies that the Kth entry in the table is removed.

[0395] 7. The method of solution 5, wherein the condition is that the table includes a redundant candidate that is similar to the new candidate, and the redundant candidate corresponds to index 0, and wherein the rule specifies removing the Kth entry in the table.

[0396] The following solution illustrates an example embodiment of the technique discussed in the previous section (eg, Bullet 2).

[0397] 8. A method of video processing, comprising: performing conversion between a video block of a video and a bitstream representation of the video; and selectively updating a table of motion candidates for another video block by removing existing candidates in the table and adding new candidates based on a rule, wherein the rule depends on characteristics of the existing candidates.

[0398] 9. The method according to solution 8, wherein the characteristic of the existing candidate is an index of the existing candidate.

[0399] The following solution illustrates an example embodiment of the techniques discussed in the previous section (eg, Bullet 3).

[0400] 10. A method of video processing, comprising: performing conversion between a video block of a video and a bitstream representation of the video; and selectively based on a rule, updating a table of motion candidates for another video block by adding a new candidate based on the conversion at the Kth position starting from the 0th position in the table, where K depends on the rule.

[0401] 11. The method of solution 10, wherein the rule specifies that K depends on the index of the redundant candidate in the table.

[0402] 12. The method according to solution 10, wherein K=1, or K depends on the size of the table.

[0403] The following solution illustrates an example embodiment of the technique discussed in the previous section (eg, bullet 5).

[0404] 13. A method of video processing, comprising: performing conversion between a video block of a video processing unit of a video and a bitstream representation of the video; and selectively updating a table of motion candidates for another video block based on a rule, wherein the rule depends on the video processing unit.

[0405] 14. The method of solution 13, wherein the rule specifies whether pruning is applied to updates to the video processing unit.

[0406] 15. A method according to any of solutions 13-14, wherein, depending on the characteristics of the video processing unit, the rule is one of a first rule specifying application of partial pruning or a second rule specifying application of full pruning.

[0407] 16. A method according to any one of solutions 1-15, wherein the table is a history-based motion vector prediction table.

[0408] 17. A method according to any of solutions 1-16, wherein performing the conversion includes encoding the video to generate a codec representation.

[0409] 18. A method according to any of solutions 1-16, wherein performing the conversion includes parsing and decoding the codec representation to generate the video.

[0410] 19. A video decoding device comprising a processor configured to implement the method according to one or more of solutions 1 to 18.

[0411] 20. A video encoding device comprising a processor configured to implement the method according to one or more of solutions 1 to 18.

[0412] 21. A computer program product storing a computer code, which, when executed by a processor, causes the processor to implement the method according to any one of solutions 1 to 18.

[0413] 22. A method, apparatus or system as described in this document.

[0414] Fig. 27 A flow chart of an example method for video processing is shown. The method includes maintaining (2702) a history-based motion vector prediction (HMVP) table including at least one motion candidate; performing (2704) a conversion between a current video block of a video and a bitstream representation of the video; and updating (2706) the table by removing a Kth entry in the table and adding the motion candidate to the HMVP table, wherein K is a positive integer and is determined based on a predetermined rule, and the motion candidate added to the HMVP table is derived from motion information derived during the conversion.

[0415] In some examples, performing the conversion is based at least in part on the HMVP table.

[0416] In some examples, the predetermined rule specifies that K is predefined.

[0417] In some examples, K is set to 1.

[0418] In some examples, the predetermined rule specifies that K depends on the size of the HMVP table.

[0419] In some examples, the predetermined rule specifies that K depends on one or more characteristics of the transformation of the video block.

[0420] In some examples, when the HMVP table is used for an inter-codec block, K is set to K0, and when the HMVP table is used for an IBC codec block, K is set to K1, where K0 is not equal to K1.

[0421] In some examples, when the HMVP table is used for an intra-frame codec block, K is set equal to K2, where K2 is not equal to K1 or / and K0.

[0422] In some examples, the predetermined rule specifies that K depends on at least one of codec information associated with the current video block, how many times the table has been updated, and the parity of the number of times the table has been updated, wherein the codec information includes at least one of block dimensions and block positions.

[0423] In some examples, K is signaled in the bitstream at at least one of a sequence level, a video level, a picture level, a slice level, a slice level, a sub-picture level, a tile level, or other video processing unit level.

[0424] In some examples, for each entry in the table, a variable is assigned to indicate the frequency of the candidate associated with the entry, where a predetermined rule specifies that K is determined according to the variable.

[0425] In some examples, predetermined rules specify that K changes adaptively on the fly.

[0426] In some examples, for a first video region, K is set to a first value, and for a second video region, K is set to a second value, where the first value and the second value are different.

[0427] In some examples, the first value or the second value is equal to zero.

[0428] In some examples, the update is applied under certain conditions.

[0429] In some examples, when updating the table with a motion candidate, the Kth entry in the table is removed if the table is full and there are no redundant candidates in the table, where the redundant candidate is the same or similar to the new candidate.

[0430] In some examples, when updating the table with a motion candidate, if there is a redundant candidate in the table and the redundant candidate is associated with an index equal to 0, the Kth entry in the table is removed, where the redundant candidate is the same or similar to the new candidate.

[0431] Fig.28 A flow chart of an example method for video processing is shown. The method includes maintaining (2802) a history-based motion vector prediction (HMVP) table including at least one motion candidate; performing (2804) a conversion between a current video block of a video and a bitstream representation of the video; and updating (2806) the table with a motion candidate based on a position of a redundant candidate in the HMVP table, wherein the motion candidate is derived from motion information derived during the conversion and the redundant candidate is the same as or similar to the motion candidate.

[0432] In some examples, performing the conversion is based at least in part on the HMVP table.

[0433] In some examples, if a redundant candidate is associated with an index in the HMVP table that is equal to N, the redundant candidate is first modified and stored in the table, where N is an integer.

[0434] In some examples, N=0 or L-1, where L is the size of the HMVP table.

[0435] In some examples, a motion vector or block vector associated with a redundant candidate is modified by adding one or more offsets or being scaled.

[0436] In some examples, the modified redundant candidate is associated with the same index N in the updated HMVP table.

[0437] Fig.29 A flow chart of an example method for video processing is shown. The method includes maintaining (2902) a history-based motion vector prediction (HMVP) table including at least one motion candidate; performing (2904) a conversion between a current video block of a video and a bitstream representation of the video; and updating (2906) the HMVP table by adding the motion candidate to a Kth entry in the HMVP table, wherein the motion candidate added to the HMVP table is derived from motion information derived during the conversion, and K is a positive integer and is determined based on a predetermined rule.

[0438] In some examples, performing the conversion is based at least in part on the HMVP table.

[0439] In some examples, the predetermined rule specifies that K is less than or equal to T, where T is the number of candidates already in the HMVP table.

[0440] In some examples, how to add motion candidates depends on the index of a redundant candidate in the HVMP table, where the redundant candidate is the same or similar to the motion candidate.

[0441] In some examples, the predetermined rule specifies that K is predefined.

[0442] In some examples, K is set to 1.

[0443] In some examples, the motion candidate is added as a candidate with an index set to (L-2), where L is the size of the HMVP table.

[0444] In some examples, the predetermined rule specifies that K depends on the size of the HMVP table.

[0445] In some examples, the predetermined rule specifies that K depends on one or more characteristics of the transformation of the video block.

[0446] In some examples, when the HMVP table is used for an inter-codec block, K is set to K0, and when the HMVP table is used for an IBC codec block, K is set to K1, where K0 is not equal to K1.

[0447] In some examples, when the HMVP table is used for an intra-frame codec block, K is set equal to K2, where K2 is not equal to K1 or / and K0.

[0448] In some examples, the predetermined rule specifies that K depends on at least one of codec information associated with the current video block, how many times the table has been updated, and the parity of the number of times the table has been updated, wherein the codec information includes at least one of block dimensions and block positions.

[0449] In some examples, K is signaled in the bitstream at at least one of a sequence level, a video level, a picture level, a slice level, a slice level, a sub-picture level, a tile level, or other video processing unit level.

[0450] In some examples, for each entry in the HMVP table, a variable is assigned to indicate the frequency of the candidate associated with the entry, where a predetermined rule specifies that K is determined according to the variable.

[0451] In some examples, predetermined rules specify that K changes adaptively on the fly.

[0452] In some examples, for a first video region, K is set to a first value, and for a second video region, K is set to a second value, where the first value and the second value are different.

[0453] In some examples, the first value or the second value is equal to zero.

[0454] In some examples, the predetermined rule specifies that K depends on a pruning result indicating whether there is an identical or similar candidate in the HMVP table before being updated.

[0455] Fig.30 A flow chart of an example method for video processing is shown. The method includes maintaining (3002) a history-based motion vector prediction (HMVP) table including at least one motion candidate; performing (3004) a conversion between a current video block of a video and a bitstream representation of the video; and updating (3006) the HMVP table based on one or more rules associated with a video processing unit, wherein the rules vary from one video processing unit to another video processing unit.

[0456] In some examples, performing the conversion is based at least in part on the HMVP table.

[0457] In some examples, for a first rule, pruning is applied, and for a second rule, non-pruning is applied.

[0458] In some examples, for the first rule, partial pruning is applied, and for the second rule, full pruning is applied.

[0459] In some examples, the rules are determined based on one or more characteristics of the conversion of the video processing unit.

[0460] In some examples, the one or more characteristics include a prediction mode of the processing unit.

[0461] In some examples, the rules are determined based on codec mode information of the video processing unit.

[0462] In some examples, when the HMVP table is used for inter-codec blocks, the rule changes from one video processing unit to another, or the rule specifies an adaptive order according to the pruning results.

[0463] In some examples, when the HMVP table is used for an IBC codec block, the rule is a FIFO rule.

[0464] In some examples, the updating depends on codec information associated with the video block.

[0465] In some examples, the codec information includes codec mode information.

[0466] In some examples, the transform generates video blocks of the video from the bitstream representation.

[0467] In some examples, the transform generates a bitstream representation from video blocks of the video.

[0468] In this document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during conversion from a pixel representation of a video to a corresponding bitstream representation, or vice versa. The bitstream representation of the current video block may correspond, for example, to bits that are concatenated or interspersed at different locations within the bitstream, as defined by the syntax. For example, a macroblock may be encoded based on transformed and coded error residual values ​​and also using bits in the header and other fields in the bitstream.

[0469] The disclosed and other solutions, examples, embodiments, modules and functional operations described in this document can be implemented in digital electronic circuits, or in computer software, firmware or hardware (including the structures disclosed in this document and their structural equivalents), or in a combination of one or more of them. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium, which are used to be executed by a data processing device or control the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances that affect machine-readable propagation signals, or a combination of one or more of them. The term "data processing device" includes all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal generated to encode information for transmission to a suitable receiver device, such as a machine-generated electrical signal, an optical signal, or an electromagnetic signal.

[0470] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language (including compiled or interpreted languages), and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). A computer program may be deployed to execute on one computer or on multiple computers located on one site or distributed across multiple sites and interconnected by a communication network.

[0471] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, and the apparatus can also be implemented as special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits).

[0472] Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operably coupled to receive data from or transfer data to or from the one or more mass storage devices. However, a computer does not require such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

[0473] Although this patent document contains many details, these details should not be interpreted as limitations on any subject matter or the scope of possible protection, but rather as descriptions of features specific to particular embodiments of particular technologies. Certain features described in the context of separate embodiments in this patent document may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be excluded from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0474] Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed, to achieve the desired results. In addition, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0475] Only a few implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and shown in this patent document.

Claims

1. A method for video processing, comprising: Maintaining a history-based motion vector prediction (HMVP) table including at least one motion candidate; performing conversion between a current video block of a video and a bitstream of said video; as well as updating the HMVP table by removing a Kth entry in the HMVP table and adding a motion candidate to the HMVP table, wherein K is a positive integer and is determined based on a predetermined rule, and the motion candidate added to the HMVP table is derived from the motion information derived during the conversion, When the HMVP table is used for an inter-frame codec block, K is set to K0, and when the HMVP table is used for an IBC codec block, K is set to K1, wherein K0 is not equal to K1.

2. The method according to claim 1, wherein: Performing the conversion is based at least in part on the HMVP table.

3. The method according to claim 1 or 2, wherein: The predetermined rule specifies that K is predefined.

4. The method according to claim 3, wherein: K0 or K1 is set to 1.

5. The method according to claim 1, wherein: The predetermined rule specifies that K depends on the size of the HMVP table.

6. The method according to claim 1, wherein: The predetermined rule specifies that K depends on one or more characteristics of the transformation of the video block.

7. The method according to claim 1, wherein: When the HMVP table is used for an intra-frame codec block, K is set equal to K2, wherein K2 is not equal to K1 or / and K0.

8. The method according to claim 1, wherein: The predetermined rule specifies that K depends on at least one of codec information associated with the current video block, how many times the HMVP table has been updated, and a parity of the number of times the HMVP table has been updated, wherein the codec information includes at least one of block dimensions and block positions.

9. The method according to claim 1 or 2, wherein: K is signaled in the bitstream at at least one of a sequence level, a video level, a picture level, a slice level, a slice level, a sub-picture level, a tile level, or other video processing unit level.

10. The method according to claim 1, wherein: For each entry in the HMVP table, a variable is assigned to indicate the frequency of the candidate associated with the entry, wherein the predetermined rule specifies that K is determined according to the variable.

11. The method according to claim 1, wherein: The predetermined rule specifies that K changes adaptively on the fly.

12. The method according to claim 11, wherein: For a first video region, K is set to a first value, and for a second video region, K is set to a second value, wherein the first value and the second value are different.

13. The method according to claim 1 or 2, wherein: The updates are applied under certain conditions.

14. The method according to claim 13, wherein: When updating the HMVP table with the motion candidate, if the HMVP table is full and there is no redundant candidate in the HMVP table, the Kth entry in the HMVP table is removed, wherein the redundant candidate is the same as or similar to the new candidate.

15. The method according to claim 13, wherein: When updating the HMVP table with the motion candidate, if there is a redundant candidate in the HMVP table and the redundant candidate is associated with an index equal to 0, the Kth entry in the HMVP table is removed, wherein the redundant candidate is identical or similar to the new candidate.

16. The method according to claim 1, further comprising: The HMVP table is updated with a motion candidate based on a position of a redundant candidate in the HMVP table, wherein the motion candidate is derived from the motion information derived during the conversion and the redundant candidate is the same as or similar to the motion candidate.

17. The method according to claim 16, wherein: Performing the conversion is based at least in part on the HMVP table.

18. The method according to claim 16 or 17, wherein: If the redundant candidate is associated with an index in the HMVP table equal to N, the redundant candidate is first modified and stored in the HMVP table, N being an integer.

19. The method according to claim 18, wherein: N=0 or L-1, where L is the size of the HMVP table.

20. The method according to claim 19, wherein: The motion vector or block vector associated with the redundant candidate is modified by adding one or more offsets or being scaled.

21. The method according to claim 20, wherein: The modified redundant candidate is associated with the same index N in the updated HMVP table.

22. The method of claim 1, further comprising: The HMVP table is updated by adding a motion candidate to a K3 th entry in the HMVP table, wherein the motion candidate added to the HMVP table is derived from the motion information derived during the converting, and K3 is a positive integer and is determined based on a predetermined rule.

23. The method according to claim 22, wherein: Performing the conversion is based at least in part on the HMVP table.

24. The method according to claim 22 or 23, wherein: The predetermined rule specifies that K3 is less than or equal to T, where T is the number of candidates already in the HMVP table.

25. The method according to claim 22, wherein: How to add motion candidates depends on the index of a redundant candidate in the HVMP table, where the redundant candidate is the same or similar to the motion candidate.

26. The method of claim 22, wherein: The predetermined rule specifies that K3 is predefined.

27. The method according to claim 26, wherein: K3 is set to 1.

28. The method according to claim 27, wherein: The motion candidate is added as a candidate with index set to (L-2), where L is the size of the HMVP table.

29. The method of claim 22, wherein: The predetermined rule specifies that K3 depends on the size of the HMVP table.

30. The method of claim 22, wherein: The predetermined rule specifies that K3 depends on one or more characteristics of the transformation of the video block.

31. The method according to claim 30, wherein: When the HMVP table is used for an inter-frame codec block, K3 is set to K4, and when the HMVP table is used for an IBC codec block, K3 is set to K5, where K4 is not equal to K5.

32. The method according to claim 31, wherein: When the HMVP table is used for an intra-frame codec block, K3 is set equal to K6, wherein K6 is not equal to K5 and / or K4.

33. The method of claim 22, wherein: The predetermined rule specifies that K3 depends on at least one of codec information associated with the current video block, how many times the HMVP table has been updated, and the parity of the number of times the HMVP table has been updated, wherein the codec information includes at least one of block dimension and block position.

34. The method according to claim 22 or 23, wherein: K3 is signaled in the bitstream at at least one of a sequence level, a video level, a picture level, a slice level, a slice level, a sub-picture level, a tile level, or other video processing unit level.

35. The method of claim 22, wherein: For each entry in the HMVP table, a variable is assigned to indicate the frequency of the candidate associated with the entry, wherein the predetermined rule specifies that K3 is determined according to the variable.

36. The method of claim 22, wherein: The predetermined rule specifies that K3 changes adaptively on the fly.

37. The method of claim 36, wherein: For a first video region, K3 is set to a first value, and for a second video region, K3 is set to a second value, wherein the first value and the second value are different.

38. The method of claim 22, wherein: The predetermined rule specifies that K3 depends on a pruning result, which indicates whether there is an identical or similar candidate in the HMVP table before being updated.

39. The method of claim 1, further comprising: a video processing unit that performs conversion between video blocks of a video and a bitstream of said video; as well as The HMVP table is updated based on one or more rules associated with the video processing unit, wherein the rules vary from one video processing unit to another.

40. The method of claim 39, wherein: Performing the conversion is based at least in part on the HMVP table.

41. The method according to claim 39 or 40, wherein: For the first rule, pruning is applied, and for the second rule, non-pruning is applied.

42. The method according to claim 39 or 40, wherein: For the first rule, partial pruning is applied, and for the second rule, full pruning is applied.

43. The method according to claim 39 or 40, wherein: The rules are determined based on one or more characteristics of the conversion of the video processing unit.

44. The method of claim 43, wherein: The one or more characteristics include a prediction mode of the processing unit.

45. The method of claim 39, wherein: The rule is determined based on codec mode information of the video processing unit.

46. ​​The method of claim 45, wherein: When the HMVP table is used for an inter-frame codec block, the rules vary from one video processing unit to another, or the rules specify an adaptive order according to the pruning result.

47. The method of claim 45, wherein: When the HMVP table is used for an IBC codec block, the rule is a FIFO rule.

48. The method according to any one of claims 1-2, 16-17, 22-23, 39-40, wherein The updating depends on codec information associated with the video block.

49. The method of claim 48, wherein: The codec information includes codec mode information.

50. The method according to any one of claims 1-2, 16-17, 22-23, 39-40, wherein The conversion generates video blocks of the video from the bitstream.

51. The method according to any one of claims 1-2, 16-17, 22-23, 39-40, wherein The conversion generates the bitstream from video blocks of the video.

52. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method of any one of claims 1 to 51.

53. A computer program product stored on a non-transitory computer readable medium, the computer program product comprising program code for executing the method according to any one of claims 1 to 51.

Images