LUT update call

Abstract

A video processing method is provided, comprising: maintaining one or more motion candidate tables; using the motion candidate tables to sequentially use multiple video processing operations to perform conversion between a current video block of a video and a coded representation of the current video block; and updating the motion candidate tables due to completion of a specific processing operation.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is to timely claim the priority and benefits of International Patent Application No. PCT / CN2019 / 071214 filed on January 10, 2019, in accordance with applicable patent laws and / or under the rules of the Paris Convention. For all purposes under relevant laws, the entire disclosure of the aforementioned application is incorporated by reference as part of the disclosure of this application. Technical Field

[0003] This patent document relates to video coding and decoding technology, equipment and systems. Background Art

[0004] Despite advances in video compression, digital video still accounts for the largest use of bandwidth on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, bandwidth demand for digital video usage is expected to continue to grow. Summary of the invention

[0005] This document discloses methods, systems, and devices for encoding and decoding digital video using a Merge list of motion vectors.

[0006] In one example aspect, a video processing method is disclosed. The method includes: maintaining one or more motion candidate tables; using the motion candidate tables to sequentially use multiple video processing operations to perform a conversion between a current video block of a video and a codec representation of the current video block; and updating the motion candidate tables due to completion of a particular processing operation.

[0007] In another example aspect, another video processing method is disclosed. The method includes: maintaining one or more motion candidate tables; determining at least one of the following based on characteristics of a current video block including a codec mode, size, video processing data unit, or low latency check: i) whether to update a motion candidate table in the one or more motion candidate tables, or ii) how to update the motion candidate table; and performing conversion between a current video block of a video and a codec representation of the video based on the determination.

[0008] In yet another example aspect, a video encoder device implementing the video encoding method described herein is disclosed.

[0009] In yet another representative aspect, the various techniques described herein may be embodied as a computer program product stored on a non-transitory computer-readable medium. The computer program product includes program code for executing the methods described herein.

[0010] In yet another representative aspect, a video decoder device may implement a method as described herein.

[0011] The details of one or more embodiments are set forth in the accompanying drawings, and in the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram illustrating an example of a video encoder implementation.

[0013] Figure 2 Macroblock segmentation in the H.264 video codec standard is shown.

[0014] Figure 3 An example of dividing a coding block (CB) into prediction blocks (PU) is shown.

[0015] Figure 4 An example implementation of subdividing a CTB into CBs and transform blocks (TBs) is shown. Solid lines indicate CB boundaries, and dashed lines indicate TB boundaries, including an example CTB and its partitions, and the corresponding quadtree.

[0016] Figure 5 An example of a Quad Tree Binary Tree (QTBT) structure for segmenting video data is shown.

[0017] Figure 6 An example of video block segmentation is shown.

[0018] Figure 7 An example of quadtree partitioning is shown.

[0019] Figure 8 An example of tree type signaling is shown.

[0020] Fig. 9 An example of the derivation process of Merge candidate list construction is shown.

[0021] Fig.10 Example locations of spatial merge candidates are shown.

[0022] Fig.11 An example of candidate pairs considered for redundancy check of spatial merge candidates is shown.

[0023] Fig.12 Examples of the location of the second PU for Nx2N and 2NxN partitions are shown.

[0024] Fig.13 Motion vector scaling of temporal Merge candidates is shown.

[0025] Fig.14 The candidate positions of the temporal Merge candidates and their collocated pictures are shown.

[0026] Fig.15 An example of a combined bi-predictive Merge candidate is shown.

[0027] Fig.16 An example of a derivation process of motion vector prediction candidates is shown.

[0028] Fig.17 An example of motion vector scaling of spatial motion vector candidates is shown.

[0029] Fig.18 An example Alternative Temporal Motion Vector Prediction (ATMVP) for motion prediction of a CU is shown.

[0030] Fig.19 An example of identification of source blocks and source pictures is pictorially depicted.

[0031] Fig. 20 An example of one CU having four sub-blocks and neighboring blocks is shown.

[0032] Fig.21 An example of bilateral matching is shown.

[0033] Fig. 22 An example of template matching is shown.

[0034] Fig.23 An example of unilateral motion estimation (ME) in frame rate up conversion (FRUC) is depicted.

[0035] Fig.24 An example of DMVR based on bilateral template matching is shown.

[0036] Fig.25 An example of spatial neighboring blocks used to derive spatial Merge candidates is shown.

[0037] Fig.26 Depicted is an example of how to select a representative location for lookup table update.

[0038] Fig.27A and Fig.27B An example of updating a lookup table with a new set of motion information is shown.

[0039] Fig.28A and Fig.28BAn example of a hardware platform for implementing the visual media decoding or visual media encoding techniques described in this document is shown.

[0040] Fig.29 is an example of a flowchart of a video processing method based on some embodiments of the disclosed technology.

[0041] Fig.30 is an example of a flowchart of a video processing method based on some embodiments of the disclosed technology.

[0042] Fig.31 An example of a decoding flow chart using the proposed HMVP method is shown.

[0043] Fig.32 An example of updating a table using the proposed HMVP method is shown.

[0044] Fig.33A and Fig.33B An example of a LUT updating method based on redundancy removal (one redundant motion candidate is removed) is shown.

[0045] Fig.34A and Fig.34B An example of a redundancy removal-based LUT updating method (a plurality of redundant motion candidates are removed) is shown. DETAILED DESCRIPTION

[0046] In order to improve the compression rate of video, researchers are constantly looking for new technologies to encode video.

[0047] 1. Overview

[0048] This document relates to video codec technology. Specifically, it relates to motion information coding (such as Merge mode, AMVP mode) in video codec. It can be applied to existing video codec standards, such as HEVC, or standards to be finalized (e.g., Versatile Video Codec). It can also be applied to future video codec standards or video codecs.

[0049] Brief Discussion

[0050] Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, ISO / IEC developed MPEG-1 and MPEG-4 Visual, and the two organizations jointly developed the H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Coding (AVC) and H.265 / HEVC standards. Since H.262, video codec standards are based on a hybrid video codec structure that utilizes temporal prediction plus transform coding. Figure 1 An example of a typical HEVC encoder framework is depicted in .

[0051] 2.1 Segmentation Structure

[0052] 2.1.1 Segmentation Tree Structure in H.264 / AVC

[0053] In previous standards, the core of the codec layer is the macroblock, which contains 16×16 blocks of luminance samples and, in the usual case of 4:2:0 color sampling, two corresponding blocks of 8×8 chrominance samples.

[0054] The intra codec block uses spatial prediction to exploit the spatial correlation between pixels. Two partitions are defined: 16x16 and 4x4.

[0055] The inter-frame codec uses temporal prediction instead of spatial prediction by estimating the motion between pictures. The 16×16 macroblock or any of its sub-macroblocks can be partitioned into: 16×8, 8×16, 8×8, 8×4, 4×8, 4×4 (see Figure 2 ) estimates motion independently. Only one motion vector (MV) is allowed per sub-macroblock partition.

[0056] 2.1.2 Partition Tree Structure in HEVC

[0057] In HEVC, CTU is divided into CUs using a quadtree structure represented as a codec tree to accommodate various local characteristics. A decision is made at the CU level whether to use inter-frame (temporal domain) or intra-frame (spatial domain) prediction to encode and decode the picture area. Depending on the PU partition type, each CU can be further divided into one, two or four PUs. Within a PU, the same prediction process is applied, and relevant information is transmitted to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU partition type, the CU can be partitioned into transform units (TUs) according to another quadtree structure similar to the codec tree of the CU. An important feature of the HEVC structure is that it has a multi-partition concept, including CU, PU, ​​and TU.

[0058] In the following, various features involved in hybrid video coding using HEVC are highlighted as follows.

[0059] 1) Codec tree unit and coding tree block (CTB) structure: A similar structure in HEVC is the coding tree unit (CTU), which has a size selected by the encoder and can be larger than a traditional macroblock. A CTU includes a luma CTB and corresponding chroma CTBs and syntax elements. The size L×L of the luma CTB can be selected as L=16, 32, or 64 samples, where larger sizes generally enable better compression. HEVC then supports splitting the CTB into smaller blocks using a tree structure and quadtree-like signaling.

[0060] 2) Coding unit (CU) and coding block (CB): The quadtree syntax of a CTU specifies the size and position of its luma CB and chroma CB. The root of the quadtree is associated with the CTU. Therefore, the size of the luma CTB is the maximum supported size of the luma CB. The partitioning of the CTU into luma CB and chroma CB is signaled jointly. One luma CB and generally two chroma CBs, together with the associated syntax, form a coding unit (CU). A CTB may contain only one CU or may be partitioned to form multiple CUs, and each CU has an associated tree partitioned into prediction units (PUs) and transform units (TUs).

[0061] 3) Prediction unit and prediction block (PB): The decision whether to use inter-frame or intra-frame prediction to encode or decode the picture area is made at the CU level. The root of the PU partition structure is at the CU level. Depending on the basic prediction type decision, the luma CB and chroma CB can be further divided in size and predicted from the luma and chroma prediction blocks (PB). HEVC supports various PB sizes from 64×64 to 4×4 samples. Figure 3 An example of PBs allowed for an M×M CU is shown.

[0062] 4) TU and transform blocks: The prediction residuals are encoded and decoded using block transforms. The root of the TU tree structure is at the CU level. The luma CB residual can be the same as the luma transform block (TB), or can be further divided into smaller luma TBs. The same applies to chroma TBs. For square TBs of size 4×4, 8×8, 16×16, and 32×32, integer basis functions similar to the discrete cosine transform (DCT) are defined. For the 4×4 transform of the luma intra prediction residual, an integer transform derived from the form of the discrete sine transform (DST) can be specified instead.

[0063] Figure 4 An example of subdividing a CTB into CBs and transform blocks (TBs) is shown. Solid lines indicate CB boundaries and dashed lines indicate TB boundaries. (a) CTB and its partitioning, (b) corresponding quadtree.

[0064] 2.1.2.1 Tree structure divided into transform blocks and units

[0065] For residual coding, CBs can be recursively partitioned into transform blocks (TBs). The partitioning is signaled by the residual quadtree. Only square CB and TB partitioning is specified, where a block can be recursively divided into four quadrants, such as Figure 4 As shown. For a given luma CB of size M×M, a flag signals whether it is divided into four blocks of size M / 2×M / 2. If further division is possible, as signaled by the maximum depth of the residual quadtree indicated in the SPS, each quadrant is assigned a flag indicating whether it is divided into four quadrants. The leaf node blocks resulting from the residual quadtree are transform blocks that are further processed by the transform codec. The encoder indicates the maximum and minimum luma TB sizes it will use. When the CB size is larger than the maximum TB size, the division is implicit. When the division would cause the luma TB size to be smaller than the indicated minimum value, no division is implicit. The chroma TB size is half the luma TB size in each dimension, except for the case where the luma TB size is 4×4, in which case a single 4×4 chroma TB is used for the area covered by 4 4×4 luma TBs. In the case of an intra-predicted CU, the decoded samples of the nearest TB (inside or outside the CB) are used as reference data for intra prediction.

[0066] Contrary to previous standards, the HEVC design allows TB to span multiple PBs for inter-predicted CUs to maximize the potential codec efficiency advantage of quadtree-structured TB partitioning.

[0067] 2.1.2.2 Parent Node and Child Node

[0068] The CTB is divided according to a quadtree structure, and its nodes are codec units. The multiple nodes in the quadtree structure include leaf nodes and non-leaf nodes. In the tree structure, the leaf node has no child nodes (i.e., the leaf node is not further divided). The non-leaf node includes the root node of the tree structure. The root node corresponds to the initial video block (e.g., CTB) of the video data. For each corresponding non-root node in the multiple nodes, the corresponding non-root node corresponds to a video block, which is a child block of the video block corresponding to the parent node in the tree structure of the corresponding non-root node. Each corresponding non-leaf node in the multiple non-leaf nodes has one or more child nodes in the tree structure.

[0069] 2.1.3 Quadtree plus binary tree block structure with larger CTU in JEM

[0070] In order to explore future video codec technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and input them into a reference software called the Joint Exploration Model (JEM).

[0071] 2.1.3.1QTBT block segmentation structure

[0072] Unlike HEVC, the QTBT structure removes the concept of multiple partition types, that is, it removes the separation of the concepts of CU, PU, ​​and TU, and supports greater flexibility in the CU partition shape. In the QTBT block structure, the CU can be square or rectangular. Figure 5 As shown, the codec tree unit (CTU) is first partitioned by a quadtree structure. The quadtree leaf nodes are further partitioned by a binary tree structure. There are two types of partitions in the binary tree partition, symmetrical horizontal partition and symmetrical vertical partition. The binary tree leaf nodes are called codec units (CUs), and the partition is used for prediction and transformation processes without any further partitioning. This means that in the QTBT codec block structure, CU, PU and TU have the same block size. In JEM, CU sometimes includes codec blocks (CBs) of different color components, for example, in the case of P slices and B slices of 4:2:0 chroma format, a CU contains one luminance CB and two chroma CBs, and CU sometimes includes CBs of a single component, for example, in the case of I slices, a CU contains only one luminance CB, or only two chroma CBs.

[0073] For the QTBT segmentation scheme, the following parameters are defined.

[0074] –CTU size: the root node size of the quadtree, the same concept as HEVC

[0075] –MinQTSize: Minimum allowed quadtree leaf node size

[0076] –MaxBTSize: Maximum allowed binary tree root node size

[0077] –MaxBTDepth: Maximum allowed binary tree depth

[0078] –MinBTSize: minimum allowed binary tree leaf node size

[0079] In one example of a QTBT partitioning structure, the CTU size is set to 128×128 luma samples with two corresponding 64×64 chroma sample blocks, MinQTSize is set to 16×16, MaxBTSize is set to 64×64, MinBTSize is set to 4×4 (for both width and height), and MaxBTDepth is set to 4. Quadtree partitioning is first applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can have sizes from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the leaf quadtree node is 128×128, it will not be further divided by the binary tree because its size exceeds MaxBTSize (i.e., 64×64). Otherwise, the leaf quadtree node can be further divided by the binary tree. Therefore, the quadtree leaf node is also the root node of the binary tree, and its binary tree depth is 0. When the binary tree depth reaches MaxBTDepth (i.e. 4), no further partitioning is considered. When the width of the binary tree node is equal to MinBTSize (i.e. 4), no further horizontal partitioning is considered. Similarly, when the height of the binary tree node is equal to MinBTSize, no further vertical partitioning is considered. The leaf nodes of the binary tree are further processed through the prediction and transformation process without any further segmentation. In JEM, the maximum CTU size is 256×256 luminance samples.

[0080] Figure 5 (left) shows an example of block segmentation by using QTBT, and Figure 5 (Right) shows the corresponding tree representation. Solid lines indicate quadtree partitions, and dashed lines indicate binary tree partitions. In each partition (i.e., non-leaf) node of the binary tree, a flag is signaled to indicate which partition type (i.e., horizontal or vertical) is used, where 0 indicates horizontal partitioning and 1 indicates vertical partitioning. For quadtree partitioning, there is no need to indicate the partition type, because quadtree partitioning always divides the block horizontally and vertically to produce four sub-blocks of equal size.

[0081] In addition, the QTBT scheme supports the ability of luma and chroma to have separate QTBT structures. Currently, for P and B slices, the luma CTB and chroma CTB in one CTU share the same QTBT structure. However, for I slices, the luma CTB is partitioned into CUs by the QTBT structure, and the chroma CTB is partitioned into chroma CUs by another QTBT structure. This means that the CU in the I slice includes a codec block of the luma component or a codec block of two chroma components, and the CU in the P slice or B slice includes codec blocks of all three color components.

[0082] In HEVC, inter prediction of small blocks is restricted to reduce memory access for motion compensation, such as 4×8 and 8×4 blocks do not support bidirectional prediction, and 4×4 blocks do not support inter prediction. In JEM's QTBT, these restrictions are removed.

[0083] 2.1.4 VVC Ternary Tree

[0084] Tree types other than quadtrees and binary trees are supported. Figure 6 As shown in (d) and (e), two other ternary tree (TT) partitions are introduced, namely horizontal and vertical center-side ternary trees.

[0085] Figure 6 Shown are: (a) quadtree partition (b) vertical binary tree partition (c) horizontal binary tree partition (d) vertical center-side ternary tree partition (e) horizontal center-side ternary tree partition.

[0086] There are two levels of trees, the region tree (quadtree) and the prediction tree (binary tree or ternary tree). The CTU is first partitioned by the region tree (RT). The RT leaves can be further partitioned by the prediction tree (PT). The PT leaves can also be further partitioned by the PT until the maximum PT depth is reached. The PT leaf is the basic coding unit. For convenience, it is still referred to as the CU. The CU cannot be further partitioned. Both prediction and transformation are applied to the CU in the same way as in JEM. The entire partitioning structure is named "multi-type tree".

[0087] 2.1.5 Segmentation Structure

[0088] The tree structure used in this response is called Multi-Tree Type (MTT), which is a generalization of QTBT. In QTBT, Figure 5 As shown in FIG. 1 , the codec tree unit (CTU) is firstly divided by a quadtree structure, and the quadtree leaf nodes are further divided by a binary tree structure.

[0089] The basic structure of MTT includes two types of tree nodes: region tree (RT) and prediction tree (PT), which supports nine types of segmentation, such as Figure 7 shown.

[0090] Figure 7 Shown are: (a) quadtree partition (b) vertical binary tree partition (c) horizontal binary tree partition (d) vertical ternary tree partition (e) horizontal ternary tree partition (f) horizontal upward asymmetric binary tree partition (g) horizontal downward asymmetric binary tree partition (h) vertical left asymmetric binary tree partition (i) vertical right asymmetric binary tree partition.

[0091] The region tree can recursively divide the CTU into blocks until it reaches a 4x4 size region tree leaf node. At each node of the region tree, a prediction tree can be formed from one of three tree types: binary tree (BT), ternary tree (TT), and asymmetric binary tree (Asymmetric Binary Tree, ABT). In PT partitioning, it is prohibited to have quadtree partitions in the branches of the prediction tree. As in JEM, the luminance tree and chrominance tree are separated in the I stripe. The signaling notification method of RT and PT is as follows Figure 8 shown.

[0092] 2.2 Inter-frame prediction in HEVC / H.265

[0093] Each inter-predicted PU has motion parameters for one or two reference picture lists. Motion parameters include motion vectors and reference picture indices. Inter_pred_idc can also be used to signal the use of one of the two reference picture lists. Motion vectors can be explicitly encoded and decoded as delta relative to the predicted value. Such encoding and decoding mode is called AMVP mode.

[0094] When a CU is encoded or decoded in skip mode, one PU is associated with the CU and there are no significant residual coefficients, no coded motion vector deviations or reference picture indices. A Merge mode is defined so that the motion parameters of the current PU are obtained from neighboring PUs including spatial and temporal candidates. Merge mode can be applied to any inter-predicted PU, not just for skip mode. An alternative to Merge mode is explicit transmission of motion parameters, where the motion vectors, the corresponding reference picture index for each reference picture list, and the use of the reference picture list are explicitly signaled per PU.

[0095] When signaling indicates that one of the two reference picture lists is to be used, a PU is generated from a block of samples. This is called "unidirectional prediction". Unidirectional prediction can be used for both P slices and B slices.

[0096] When signaling indicates that two reference picture lists are to be used, the PU is generated from two sample blocks. This is called "bidirectional prediction". Bidirectional prediction is only available for B slices.

[0097] The following text provides details about the inter prediction modes specified in HEVC. The description will start with the Merge mode.

[0098] 2.2.1Merge mode

[0099] 2.2.1.1 Merge Mode Candidate Derivation

[0100] When predicting a PU using Merge mode, an index pointing to an entry in the Merge candidate list is parsed from the bitstream and used to retrieve motion information. The construction of this list is specified in the HEVC standard and can be outlined according to the following sequential steps:

[0101] Step 1: Initial candidate derivation

[0102] Step 1.1: Spatial Candidate Derivation

[0103] Step 1.2: Redundancy check of spatial candidates

[0104] οStep 1.3: Time Domain Candidate Derivation

[0105] Step 2: Additional candidate insertions

[0106] οStep 2.1: Create bidirectional prediction candidates

[0107] οStep 2.2: Insert zero motion candidates

[0108] These steps are also schematically depicted in Fig. 9 For spatial Merge candidate derivation, up to four Merge candidates are selected from candidates located at five different positions. For temporal Merge candidate derivation, up to one Merge candidate is selected from two candidates. Since a constant number of candidates for each PU is assumed at the decoder, additional candidates are generated when the number of candidates does not reach the maximum number of Merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, the index of the best Merge candidate is encoded using truncated unary binarization (TU). If the size of the CU is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

[0109] Hereinafter, operations associated with the aforementioned steps will be described in detail.

[0110] 2.2.1.2 Spatial Candidate Derivation

[0111] In the derivation of spatial merge candidates, Fig.10 Up to four Merge candidates are selected from the candidates at the positions depicted in . The order of derivation is A1, B1, B0, A0, and B2. Position B2 is considered only when any PU at position A1, B1, B0, A0 is unavailable (for example, because it belongs to another stripe or slice) or is intra-coded. After adding the candidate at position A1, a redundancy check is performed on the addition of the remaining candidates to ensure that candidates with the same motion information are excluded from the list, thereby improving coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only pairs with the same motion information are considered. Fig.11 The arrows in , and candidates are added to the list only if the corresponding candidates for redundancy checking do not have the same motion information. Another source of duplicate motion information is a "second PU" associated with a partition other than 2N×2N. As an example, Fig.12 The second PU is depicted for the N×2N and 2N×N cases, respectively. When the current PU is partitioned into N×2N, the candidate at position A1 is not considered for list construction. In fact, by adding this candidate it would result in two prediction units having the same motion information, which is redundant for having only one PU in the codec unit. Similarly, when the current PU is partitioned into 2N×N, position B1 is not considered.

[0112] 2.2.1.3 Time Domain Candidate Derivation

[0113] In this step, only one candidate is added to the list. Specifically, in the derivation of the temporal Merge candidate, the scaled motion vector is derived based on the collocated PU belonging to the picture with the minimum POC (Picture Order Count) difference with the current picture in the given reference picture list. The reference picture list to be used for derivation of the collocated PU is explicitly signaled in the slice header. The scaled motion vector of the temporal Merge candidate is obtained, as in Fig.13 As shown by the dotted line in the figure, the scaled motion vector is scaled from the motion vector of the collocated PU using the POC distances, tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal merge candidate is set equal to zero. The actual implementation of the scaling process is described in the HEVC specification. For B slices, two motion vectors (one for reference picture list 0 and the other for reference picture list 1) are obtained and combined to generate a bidirectional prediction merge candidate. Illustration of motion vector scaling for temporal merge candidates.

[0114] In the collocated PU(Y) belonging to the reference frame, the position of the temporal candidate is selected between candidates C0 and C1, such as Fig.14 If the PU at position C0 is not available, is intra-coded, or is outside the current CTU, position C1 is used. Otherwise, position C0 is used in the derivation of the time-domain Merge candidate.

[0115] 2.2.1.4. Additional candidate insertion

[0116] In addition to the spatio-temporal Merge candidates, there are two additional types of Merge candidates: combined bi-directional prediction Merge candidates and zero Merge candidates. Combined bi-directional prediction Merge candidates are generated by utilizing spatio-temporal Merge candidates. Combined bi-directional prediction Merge candidates are only used for B slices. Combined bi-directional prediction candidates are generated by combining the first reference picture list motion parameters of an initial candidate with the second reference picture list motion parameters of another initial candidate. If the two tuples provide different motion hypotheses, they will form a new bi-directional prediction candidate. As an example, Fig.15 Depicted when two candidates in the original list (on the left) (which have mvL0 and refIdxL0 or mvL1 and refIdxL1) are used to create a combined bi-predictive Merge candidate that is added to the final list (on the right). There are many rules about combining that are considered to generate these additional Merge candidates.

[0117] Zero-motion candidates are inserted to fill the remaining entries in the Merge candidate list and thus reach the MaxNumMergeCand capacity. These candidates have zero spatial displacement and a reference picture index that starts at zero and increases each time a new zero-motion candidate is added to the list. The number of reference frames used by these candidates is 1 and 2 for unidirectional prediction and bidirectional prediction, respectively. Finally, no redundancy check is performed on these candidates.

[0118] 2.2.1.5 Motion Estimation Regions for Parallel Processing

[0119] To speed up the encoding process, motion estimation can be performed in parallel, thereby deriving motion vectors for all prediction units within a given region at the same time. Deriving Merge candidates from spatial neighborhoods may interfere with parallel processing because one prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is completed. In order to ease the trade-off between coding efficiency and processing latency, HEVC defines a motion estimation region (MER), and the size of the MER is signaled in the picture parameter set using the "log2_parallel_merge_level_minus2" syntax element. When defining a MER, Merge candidates that fall into the same region are marked as unavailable and are therefore not considered in list construction.

[0120] 7.3.2.3 Picture parameter set RBSP syntax

[0121] 7.3.2.3.1 Common Picture Parameter Set RBSP Syntax

[0122]

[0123] log2_parallel_merge_level_minus2 plus 2 specifies the value of the variable Log2ParMrgLevel, which is used in the derivation of luma motion vectors for Merge mode as specified in clause 8.5.3.2.2 and in the derivation of spatial Merge candidates as specified in clause 8.5.3.2.3. The value of log2_parallel_merge_level_minus2 shall be in the range of 0 to CtbLog2SizeY-2, inclusive.

[0124] The variable Log2ParMrgLevel is derived as follows:

[0125] Log2ParMrgLevel=log2_parallel_merge_level_minus2+2 (7-37)

[0126] Note 3 - The value of Log2ParMrgLevel indicates the built-in capability of parallel derivation of Merge candidate lists. For example, when Log2ParMrgLevel is equal to 6, Merge candidate lists for all prediction units (PUs) and coding units (CUs) contained in a 64x64 block can be derived in parallel.

[0127] 2.2.2 Motion Vector Prediction in AMVP Mode

[0128] Motion vector prediction exploits the spatiotemporal correlation of motion vectors with neighboring PUs for explicit transmission of motion parameters. A motion vector candidate list is constructed by first checking the availability of neighboring PU positions in the left and upper time domains, removing redundant candidates, and adding zero vectors to make the candidate list a constant length. The encoder can then select the best prediction value from the candidate list and send a corresponding index indicating the selected candidate. Similar to Merge index signaling, a truncated unary is used to encode the index of the best motion vector candidate. In this case, the maximum value to be encoded is 2 (for example, Figures 2 to 8 ). In the following sections, details about the derivation process of motion vector prediction candidates are provided.

[0129] 2.2.2.1 Derivation of Motion Vector Prediction Candidates

[0130] Fig.16 The derivation process of motion vector prediction candidates is summarized.

[0131] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For spatial motion vector candidate derivation, based on the location of Fig.11 The motion vectors of each PU at five different positions are depicted, and two motion vector candidates are finally derived.

[0132] For temporal motion vector candidate derivation, one motion vector candidate is selected from two candidates derived based on two different collocated positions. After generating the first list of spatio-temporal candidates, duplicate motion vector candidates in the list are removed. If the number of potential candidates is greater than two, motion vector candidates whose reference picture index within the associated reference picture list is greater than 1 are removed from the list. If the number of spatio-temporal motion vector candidates is less than two, an additional zero motion vector candidate is added to the list.

[0133] 2.2.2.2 Spatial Motion Vector Candidates

[0134] In the derivation of spatial motion vector candidates, the Fig.11 Consider up to two candidates out of the five potential candidates for PU derivation at the depicted position, and those positions are the same as the positions of the motion Merge. The derivation order on the left side of the current PU is defined as A0, A1, and scaled A0, scaled A1. The derivation order on the upper side of the current PU is defined as B0, B1, B2, scaled B0, scaled B1, scaled B2. Therefore, for each side, there are four cases that can be used as motion vector candidates, two of which do not require spatial scaling and two use spatial scaling. The four different cases are summarized as follows:

[0135] No airspace scaling

[0136] (1) Same reference picture list and same reference picture index (same POC)

[0137] (2) Different reference picture lists, but the same reference picture (same POC)

[0138] Airspace scaling

[0139] (3) Same reference picture list, but different reference pictures (different POC)

[0140] (4) Different reference picture lists and different reference pictures (different POCs)

[0141] First check for no spatial scaling, then spatial scaling. When the POC is different between the reference picture of the neighboring PU and the reference picture of the current PU, spatial scaling is considered regardless of the reference picture list. If all PUs of the left candidate are unavailable or intra-coded, scaling of the upper motion vector is allowed to help parallel derivation of the left and upper MV candidates. Otherwise, spatial scaling is not allowed for the upper motion vector.

[0142] like Fig.17 As depicted, in the spatial scaling process, the motion vectors of neighboring PUs are scaled in a similar manner as in the temporal scaling. The main difference is that the reference picture list and the index of the current PU are given as input; the actual scaling process is the same as that of the temporal scaling process.

[0143] 2.2.2.3 Temporal Motion Vector Candidates

[0144] Except for the reference picture index derivation, all the processes for deriving the temporal merge candidate are the same as those for deriving the spatial motion vector candidate (see e.g. Figure 6 ). The reference picture index is signaled to the decoder.

[0145] 2.2.2.4 Signaling of AMVP Information

[0146] For the AMVP mode, four parts may be signaled in the bitstream, namely prediction direction, reference index, MVD and mv prediction value candidate index.

[0147] Syntax table:

[0148]

[0149] 7.3.8.9 Motion Vector Difference Syntax

[0150]

[0151] 2.3 New inter-frame prediction method in JEM (Joint Exploration Model)

[0152] 2.3.1 Motion vector prediction based on sub-CU

[0153] In JEM with QTBT, each CU can have at most one motion parameter set for each prediction direction. Two sub-CU level motion vector prediction methods are considered in the encoder by dividing the large CU into sub-CUs and deriving the motion information of all sub-CUs of the large CU. The optional temporal motion vector prediction (ATMVP) method allows each CU to extract multiple motion information sets from multiple blocks smaller than the current CU in the collocated reference picture. In the Spatial-TemporalMotion Vector Prediction (STMVP) method, the motion vector of the sub-CU is recursively derived by using the temporal motion vector prediction value and the spatial neighboring motion vector.

[0154] In order to maintain a more accurate motion field for sub-CU motion prediction, motion compression of reference frames is currently disabled.

[0155] 2.3.1.1 Optional temporal motion vector prediction

[0156] In the optional temporal motion vector prediction (ATMVP) method, the motion vector temporal motion vector prediction (TMVP) is modified by extracting multiple motion information sets (including motion vectors and reference indices) from blocks smaller than the current CU. Fig.18 As shown in , a sub-CU is a square N×N block (N is set to 4 by default).

[0157] ATMVP predicts the motion vectors of sub-CUs within a CU in two steps. The first step is to use the so-called time domain vector to identify the corresponding block in the reference picture. The reference picture is called the motion source picture. The second step is to divide the current CU into sub-CUs and obtain the motion vector and the reference index of each sub-CU from the block corresponding to each sub-CU, such as Fig.18 shown.

[0158] In the first step, the reference picture and the corresponding block are determined by the motion information of the spatial neighboring blocks of the current CU. In order to avoid repeated scanning of neighboring blocks, the first Merge candidate in the Merge candidate list of the current CU is used. The first available motion vector and its associated reference index are set to the temporal vector and the index of the motion source picture. In this way, in ATMVP, the corresponding block can be identified more accurately compared to TMVP, where the corresponding block (sometimes called the collocated block) is always in the lower right or center position relative to the current CU. In one example, if the first Merge candidate comes from the left neighboring block (i.e., Fig.19 A1 in ), the associated MV and reference picture are used to identify the source block and source picture.

[0159] Fig.19 An example of identifying source blocks and source pictures is shown.

[0160] In the second step, the corresponding block of the sub-CU is identified by the time domain vector in the motion source picture by adding the time domain vector to the coordinates of the current CU. For each sub-CU, the motion information of its corresponding block (the minimum motion grid covering the center sample point) is used to derive the motion information of the sub-CU. After the motion information of the corresponding N×N block is identified, it is converted into the motion vector and reference index of the current sub-CU in the same way as the TMVP of HEVC, where motion scaling and other processes apply. For example, the decoder checks whether the low latency condition is met (i.e., the POC of all reference pictures of the current picture is less than the POC of the current picture) and may use the motion vector MV x (corresponding to the motion vector of reference picture list X) to predict the motion vector MV of each sub-CU y (where X is equal to 0 or 1, and Y is equal to 1-X).

[0161] 2.3.1.2 Spatial Temporal Motion Vector Prediction (STMVP)

[0162] In this method, the motion vector of a sub-CU is recursively derived following the raster scan order. Fig. 20 This concept is illustrated. Let us consider an 8x8 CU containing four 4x4 sub-CUs A, B, C, and D. The neighboring 4x4 blocks in the current frame are labeled a, b, c, and d.

[0163] The motion derivation of sub-CU A starts by identifying its two spatial neighbors. The first neighbor is the N×N block (block c) on the upper side of sub-CU A. If block c is unavailable or intra-coded, check other N×N blocks on the upper side of sub-CU A (from left to right, starting from block c). The second neighbor is the block on the left side of sub-CU A (block b). If block b is unavailable or intra-coded, check other blocks on the left side of sub-CU A (from top to bottom, starting from block b). The motion information obtained from the adjacent blocks of each list is scaled to the first reference frame of a given list. Next, the temporal motion vector predictor (TMVP) of sub-block A is derived by following the same process as the TMVP derivation process specified in HEVC. The motion information of the collocated block at position D is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors (up to 3) are averaged for each reference list respectively. The averaged motion vector is assigned as the motion vector of the current sub-CU.

[0164] Fig. 20 An example of a CU with four sub-blocks (AD) and their neighboring blocks (ad) is shown.

[0165] 2.3.1.3 Sub-CU motion prediction mode signaling

[0166] Sub-CU mode is enabled as an additional Merge candidate, and no additional syntax elements are required to signal the mode. Two additional Merge candidates are added to the Merge candidate list of each CU to represent ATMVP mode and STMVP mode. If the sequence parameter set indicates that ATMVP and STMVP are enabled, a maximum of seven Merge candidates are used. The encoding logic of the additional Merge candidates is the same as that used for the Merge candidates in HM, which means that for each CU in a P slice or B slice, more than two RD checks are required for the two additional Merge candidates.

[0167] In JEM, all bins of the Merge index are decoded by CABAC context codec. In HEVC, only the first bin is context decoded and the remaining bins are context bypass decoded.

[0168] 2.3.2 Adaptive Motion Vector Difference Resolution

[0169] In HEVC, when use_integer_mv_flag in the slice header is equal to 0, the motion vector difference (MVD) (between the motion vector of the PU and the predicted motion vector) is signaled in units of quarter luma samples. In JEM, Locally Adaptive Motion Vector Resolution (LAMVR) is introduced. In JEM, MVD can be encoded and decoded in units of quarter luma samples, integer luma samples, or four luma samples. The MVD resolution is controlled at the codec unit (CU) level, and the MVD resolution flag is conditionally signaled for each CU with at least one non-zero MVD component.

[0170] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter luma sample MV precision is not used, another flag is signaled to indicate whether integer luma sample MV precision or quad luma sample MV precision is used.

[0171] When the first MVD resolution flag of a CU is zero or is not coded for the CU (meaning that all MVDs in the CU are zero), a quarter luma sample MV resolution is used for the CU. When the CU uses integer luma sample MV precision or four luma sample MV precision, the MVP in the AMVP candidate list of the CU is rounded to the corresponding precision.

[0172] In the encoder, the CU-level RD check is used to determine which MVD resolution to use for the CU. That is, for each MVD resolution, three CU-level RD checks are performed. In order to speed up the encoder, the following encoding scheme is applied in JEM:

[0173] During RD check of a CU with normal quarter luma sample MVD resolution, the motion information of the current CU (integer luma sample accuracy) is stored. The stored motion information (after rounding) is used as a starting point for further small-scale motion vector refinement during RD check of the same CU with integer luma sample and 4 luma sample MVD resolution, so that the time-consuming motion estimation process is not repeated three times.

[0174] Conditionally call RD check for CUs with 4 luma sample MVD resolution. For a CU, skip RD check for 4 luma sample MVD resolution of the CU when the RD cost of integer luma sample MVD resolution is much greater than the RD cost of quarter luma sample MVD resolution.

[0175] 2.3.3 Pattern Matching Motion Vector Derivation

[0176] The Pattern Matched Motion Vector Derivation (PMMVD) mode is a special Merge mode based on the Frame-Rate Up Conversion (FRUC) technology. In this mode, the motion information of the block is derived at the decoder side instead of signaling the motion information of the block.

[0177] When the Merge flag of the CU is true, the FRUC flag is signaled to the CU. When the FRUC flag is false, the Merge index is signaled and the normal Merge mode is used. When the FRUC flag is true, an additional FRUC mode flag is signaled to indicate which method (bilateral matching or template matching) will be used to derive the motion information of the block.

[0178] On the encoder side, the decision on whether to use the FRUC Merge mode for a CU is based on the RD cost selection made for normal Merge candidates. That is, the two matching modes (bilateral matching and template matching) of the CU are checked by using the RD cost selection. The matching mode that causes the minimum cost is further compared with other CU modes. If the FRUC matching mode is the most effective mode, the FRUC flag is set to true for the CU and the relevant matching mode is used.

[0179] The motion derivation process in FRUC Merge mode has two steps: first, a motion search at the CU level is performed, and then motion refinement at the sub-CU level is performed. At the CU level, the initial motion vector of the entire CU is derived based on bilateral matching or template matching. First, a list of MV candidates is generated, and the candidate that causes the minimum matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bilateral matching or template matching is performed near the starting point, and the MV result of the minimum matching cost is used as the MV of the entire CU. Subsequently, the motion information is further refined at the sub-CU level with the derived CU motion vector as the starting point.

[0180] For example, the following derivation process is performed for the W×H CU motion information derivation. In the first stage, the MV of the entire W×H CU is derived. In the second stage, the CU is further divided into M×M sub-CUs. The value of M is calculated as shown in (3), and D is a predefined division depth, which is set to 3 by default in JEM. Then the MV of each sub-CU is derived.

[0181]

[0182] like Fig.21 As shown, bilateral matching is used to derive the motion information of the current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. Under the assumption of continuous motion trajectory, the motion vectors MV0 and MV1 pointing to the two reference blocks should be proportional to the temporal distance (i.e. TD0 and TD1) between the current picture and the two reference pictures. As a special case, when the current picture is between the two reference pictures in the temporal domain and the temporal distance from the current picture to the two reference pictures is the same, bilateral matching becomes a mirror-based bidirectional MV.

[0183] like Fig. 22As shown, template matching is used to derive the motion information of the current CU by finding the closest match between the template in the current picture (the top and / or left adjacent blocks of the current CU) and the block in the reference picture (the same size as the template). In addition to the aforementioned FRUC Merge mode, template matching is also applied to AMVP mode. In JEM, as in HEVC, AMVP has two candidates. Using the template matching method, new candidates are derived. If the candidate newly derived by template matching is different from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and then the list size is set to 2 (meaning removing the second existing AMVP candidate). When applied to AMVP mode, only CU-level search is applied.

[0184] 2.3.3.1 CU-level MV candidate set

[0185] The MV candidates set at the CU level include:

[0186] (i) If the current CU is in AMVP mode, it is the original AMVP candidate

[0187] (ii) All Merge candidates,

[0188] (iii) Interpolate several MVs in the MV field.

[0189] (iv) Top and left neighboring motion vectors

[0190] When bilateral matching is used, each valid MV of the Merge candidate is used as input to generate an MV pair assuming bilateral matching. For example, at reference list A, one valid MV of the Merge candidate is (MVa, refa). Then, find the reference picture refb of its paired bilateral MV in other reference list B, so that refa and refb are on different sides of the current picture in the temporal domain. If such refb is not available in reference list B, refb is determined as a reference different from refa, and its temporal distance to the current picture is the minimum in list B. After determining refb, MVb is derived by scaling MVa based on the temporal distance between the current picture and refa, refb.

[0191] The four MVs from the interpolated MV field are also added to the candidate list at the CU level. More specifically, the interpolated MVs at positions (0,0), (W / 2,0), (0,H / 2), and (W / 2,H / 2) of the current CU are added.

[0192] When FRUC is applied in AMVP mode, the original AMVP candidates are also added to the MV candidate set at the CU level.

[0193] At the CU level, for an AMVP CU, up to 15 MVs are added to the candidate list, and for a Merge CU, up to 13 MVs are added to the candidate list.

[0194] 2.3.3.2 MV Candidates at Sub-CU Level

[0195] The MV candidates set at the sub-CU level include:

[0196] (i) MV determined from CU-level search,

[0197] (ii) neighboring MVs at the top, left, upper-left, and upper-right corners,

[0198] (iii) a scaled version of the collocated MV from the reference image,

[0199] (iv) Up to 4 ATMVP candidates,

[0200] (v) Up to 4 STMVP candidates

[0201] The scaled MV from the reference picture is derived as follows: All reference pictures in the two lists are traversed. The MV at the collocated position of the sub-CU in the reference picture is scaled to the reference of the MV at the starting CU level.

[0202] ATMVP and STMVP candidates are limited to the top four.

[0203] At the sub-CU level, up to 17 MVs are added to the candidate list.

[0204] 2.3.3.3 Generation of interpolated MV fields

[0205] Before encoding and decoding the frame, an interpolated motion field is generated for the entire picture based on the single-sided ME. The motion field can then be used as a MV candidate at the CU level or sub-CU level later.

[0206] First, the motion fields of each reference picture in both reference lists are traversed at the 4×4 block level. For each 4×4 block, if the motion associated with the block passes through a 4×4 block in the current picture (e.g. Fig.23 As shown) and the block is not assigned any interpolated motion, the motion of the reference block is scaled to the current picture according to the temporal distances TD0 and TD1 (in the same way as the MV scaling of TMVP in HEVC), and the scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to the 4×4 block, the motion of the block is marked as unavailable in the interpolated motion field.

[0207] 2.3.3.4 Interpolation and Matching Cost

[0208] When the motion vectors point to fractional sample locations, motion compensated interpolation is required. To reduce complexity, bilinear interpolation is used for both bilateral matching and template matching instead of conventional 8-tap HEVC interpolation.

[0209] The calculation of the matching cost is a little different at different steps. When selecting a candidate from the candidate set at the CU level, the matching cost is the absolute sum difference (SAD) of the bilateral match or template match. After determining the starting MV, the matching cost C of the bilateral match at the sub-CU level search is calculated as follows:

[0210]

[0211] Where w is a weighting factor set to 4 based on experience, MV and MV s Indicates the current MV and the starting MV respectively. SAD is still used as the matching cost of template matching at the sub-CU level search.

[0212] In FRUC mode, MV is derived by using luma samples only. The derived motion will be used for both luma and chroma for MC inter prediction. After deciding MV, final MC is performed using 8-tap interpolation filter for luma and 4-tap interpolation filter for chroma.

[0213] 2.3.3.5 MV Refinement

[0214] MV refinement is a pattern-based MV search with bilateral matching cost or template matching cost as the criterion. In JEM, two search modes are supported—Unrestricted Center-Biased Diamond Search (UCBDS) and Adaptive Cross Search, for MV refinement at CU level and sub-CU level, respectively. For both CU and sub-CU level MV refinement, MV is searched directly with quarter luma sample MV precision, and followed by eighth luma sample MV refinement. The search range for MV refinement for CU and sub-CU steps is set equal to 8 luma samples.

[0215] 2.3.3.6 Selection of prediction direction in template matching FRUC Merge mode

[0216] In bilateral matching Merge mode, bidirectional prediction is always applied because the motion information of the CU is derived based on the closest match between two blocks in two different reference pictures along the motion trajectory of the current CU. There is no such restriction for template matching Merge mode. In template matching Merge mode, the encoder can choose between unidirectional prediction according to list0, unidirectional prediction according to list1, or bidirectional prediction of the CU. The choice is based on the template matching cost, as shown below:

[0217] If costBi <= factor * min (cost0, cost1)

[0218] Use bidirectional prediction;

[0219] Otherwise, if cost0 <= cost1

[0220] Use one-way prediction according to list0;

[0221] otherwise,

[0222] Use one-way prediction based on list1;

[0223] Among them, cost0 is the SAD of list0 template matching, cost1 is the SAD of list1 template matching, and costBi is the SAD of bidirectional prediction template matching. The value of factor is equal to 1.25, which means that the selection process is biased towards bidirectional prediction.

[0224] The inter prediction direction selection is only applied to the template matching process at the CU level.

[0225] 2.3.4 Decoder-side motion vector refinement

[0226] In the bidirectional prediction operation, for the prediction of a block area, two prediction blocks formed using the motion vectors (MVs) of list 0 and the MVs of list 1, respectively, are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of the bidirectional prediction are further refined by a bilateral template matching process. Bilateral template matching is applied in the decoder to perform a distortion-based search between the bilateral template and the reconstructed samples in the reference picture to obtain the refined MV without sending additional motion information.

[0227] In DMVR, the bilateral template is generated as a weighted combination (i.e., average) of two prediction blocks from the initial MV0 of list 0 and MV1 of list 1, respectively, as Fig.23As shown. The template matching operation includes calculating a cost metric between the generated template and the sample area (around the initial prediction block) in the reference picture. For each of the two reference pictures, the MV that produces the minimum template cost is considered to be the updated MV in the list to replace the original MV. In JEM, 9 MV candidates are searched for each list. The 9 candidate MVs include the original MV and 8 surrounding MVs whose one luminance sample is offset relative to the original MV in the horizontal or vertical direction or both. Finally, the two new MVs (i.e., Fig.24 MV0′ and MV1′ shown) are used to generate the final bidirectional prediction result. The sum of absolute differences (SAD) is used as the cost metric.

[0228] DMVR is applied to the Merge mode of bi-directional prediction, where one MV comes from the past reference picture and the other comes from the future reference picture, without transmitting additional syntax elements. In JEM, DMVR is not applied when LIC, affine motion, FRUC or sub-CU Merge candidate is enabled for the CU.

[0229] 2.3.5 Merge / Skip Mode with Bilateral Match Refinement

[0230] First, the Merge candidate list is constructed by inserting the motion vectors and reference indices of spatially adjacent and temporally adjacent blocks into the candidate list with redundancy check until the number of available candidates reaches the maximum candidate size of 19. The spatial candidates are inserted according to a predefined insertion order ( Fig.11 ), temporal candidates, affine candidates, advanced temporal MVP (ATMVP) candidates, space-time MVP (STMVP) candidates, and additional candidates used in HEVC (combined candidates and zero candidates) to construct the Merge candidate list for Merge / skip mode:

[0231] - Spatial candidates for blocks 1-4.

[0232] - Extrapolated affine candidates for blocks 1-4.

[0233] -ATMVP.

[0234] -STMVP.

[0235] -Virtual affine candidate.

[0236] - Spatial candidates (block 5) (only used if the number of available candidates is less than 6).

[0237] - Extrapolate affine candidates (block 5).

[0238] - Temporal candidates (derived from HEVC).

[0239] - non-adjacent spatial candidates, followed by extrapolated affine candidates (blocks 6 to 49, e.g. Fig.25 depicted).

[0240] -Candidates for combination.

[0241] -Zero Candidates

[0242] Note that the IC flag is also inherited from the Merge candidates except STMVP and Affine. In addition, for the first four spatial candidates, the bidirectionally predicted candidates are inserted before the unidirectionally predicted candidates.

[0243] Blocks that are not connected to the current block can be accessed. If non-adjacent blocks are encoded and decoded in non-intra mode, the associated motion information can be added as an additional Merge candidate.

[0244] 3. Examples of Problems Solved by the Embodiments Disclosed in This Article

[0245] The current HEVC design can exploit the correlation between the current block and its neighboring blocks (next to the current block) to better encode and decode motion information. However, the neighboring blocks may correspond to different objects with different motion trajectories. In this case, the prediction from its neighboring blocks is invalid.

[0246] Prediction based on motion information of non-neighboring blocks can bring additional coding gain, at the cost of storing all motion information (usually on a 4x4 level) in cache, which significantly increases the complexity of hardware implementation.

[0247] 4. Some Examples

[0248] The following examples should be considered as examples to explain the general concept. These examples should not be interpreted narrowly. In addition, these examples can be combined in any way.

[0249] Some embodiments may use one or more lookup tables storing at least one motion candidate to predict motion information of a block. Embodiments may use motion candidates to indicate a set of motion information stored in a lookup table. For traditional AMVP or Merge mode, embodiments may use AMVP or Merge candidates for storing motion information.

[0250] The following example explains the general concept.

[0251] Example of a lookup table

[0252] Example A1: Each lookup table may contain one or more motion candidates, where each candidate is associated with its motion information.

[0253] a. Here, the motion information of the motion candidate may include part or all of the prediction direction, reference index / picture, motion vector, LIC flag, affine flag, MVD precision, and MVD value.

[0254] b. Motion information may also include block position information and / or block shape to indicate where the motion information comes from.

[0255] c. A counter may also be assigned to each lookup table.

[0256] i. At the start of encoding / decoding a picture / slice / LCU (CTU) line / slice, the counter may be initialized to zero.

[0257] ii. In one example, the counter may be updated after encoding / decoding a CTU / CTB / CU / CB / PU / certain region size (eg, 8×8 or 16×16).

[0258] iii. In one example, each time a candidate is added to the lookup table, a counter is incremented by one.

[0259] iv. In one example, the counter should be no larger than the table size (the number of allowed motion candidates).

[0260] v. Alternatively, a counter may be used to indicate how many motion candidates have been attempted to be added to the lookup table (some of them are in the lookup table, but may be removed from the table later). In this case, the counter may be larger than the table size.

[0261] d. The table size (number of allowed motion candidates) and / or the number of tables can be fixed or adaptive. The table size can be the same for all tables, or different tables can have different table sizes.

[0262] i. Alternatively, different sizes may be used for different lookup tables (eg, 1 or 2).

[0263] ii. In one example, the table size and / or the number of tables may be predefined.

[0264] iii. In one example, the table size and / or the number of tables may be signaled in a Video Parameter Set (VPS), a Sequence Parameter Set (SPS), a Picture Parameter Set (PPS), a slice header, a codec tree unit (CTU), a codec tree block (CTB), a codec unit (CU) or a prediction unit (PU), or in an area covering multiple CTUs / CTBs / CUs / PUs.

[0265] iv. The table size and / or number of tables may also depend on the slice type, the temporal layer index of the picture, the picture order count (POC) distance between one slice and the nearest intra slice.

[0266] e. Assuming there are N tables for encoding and decoding threads, N*P tables may be needed to encode and decode a slice, where P indicates the number of LCU rows or the number of slices.

[0267] i. Alternatively, only P tables may be needed to encode and decode a slice, where P indicates the number of LCU rows, where when slices are disabled, only one lookup table is used per LCU row even though N may be greater than 1.

[0268] LUT Selection

[0269] Example B1: To encode and decode a block, some or all motion candidates from a lookup table may be checked in order. When a motion candidate is checked during encoding and decoding of a block, it may be added to a motion candidate list (eg, AMVP, Merge candidate list).

[0270] a. Alternatively, motion candidates from multiple lookup tables may be checked sequentially.

[0271] b. The lookup table index may be signaled in a CTU, CTB, CU or PU, or in a region covering multiple CTUs / CTBs / CUs / PUs.

[0272] Example B2: The choice of the lookup table may depend on the location of the block.

[0273] a. It can depend on the CTU address covering the block. Here, we take two lookup tables (dual lookup tables, DLUT (Dual Look Up Tables)) as an example to illustrate this idea:

[0274] i. If the block is located in one of the first M CTUs in a CTU row, the first lookup table may be used to encode and decode the block, while for blocks located in the remaining CTUs in the CTU row, the second lookup table may be utilized.

[0275] ii. If the block is located in one of the first M CTUs in the CTU row, the motion candidates in the first lookup table may be checked first for encoding and decoding the block, and if there are not enough candidates in the first table, the second lookup table may be used. For blocks located in the remaining CTUs in the CTU row, the second lookup table may be used.

[0276] iii. Alternatively, for the blocks located in the remaining CTUs in the CTU row, the motion candidates in the second lookup table may be checked first for encoding and decoding the blocks, and if there are not enough candidates in the second table, the first lookup table may be used.

[0277] b. It may depend on the distance between the location of the block and a location associated with a motion candidate in one or more lookup tables.

[0278] iv. In one example, if one motion candidate is associated with a smaller distance to the block to be encoded, it may be checked earlier than another motion candidate.

[0279] Use of Lookup Table

[0280] Example C1: The total number of motion candidates to be checked in the lookup table may be predefined.

[0281] a. It may also depend on codec information, block size, block shape, etc. For example, for AMVP mode, only m motion candidates may be checked, while for Merge mode, n motion candidates may be checked (eg, m=2, n=44).

[0282] b. In one example, the total number of motion candidates to be checked may be signaled in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a slice header, a codec tree unit (CTU), a codec tree block (CTB), a codec unit (CU), or a prediction unit (PU), or in an area covering multiple CTUs / CTBs / CUs / PUs.

[0283] Example C2: The motion candidate(s) included in the lookup table may be directly inherited by the block.

[0284] a. They can be used for Merge mode encoding and decoding, that is, motion candidates can be checked during the Merge candidate list derivation process.

[0285] b. They can be used for affine Merge mode encoding and decoding.

[0286] i. If the affine flag of a motion candidate in the lookup table is 1, the motion candidate in the lookup table can be added as an affine Merge candidate.

[0287] c. Checking for motion candidates in the lookup table can be enabled in the following cases:

[0288] i. After inserting the TMVP candidate, the Merge candidate list is not full;

[0289] ii. After checking a certain spatial neighboring block used for spatial Merge candidate derivation, the Merge candidate list is not full;

[0290] iii. After all airspace merge candidates are merged, the merge candidate list is not full;

[0291] iv. After the combined bidirectional prediction Merge candidate, the Merge candidate list is not full;

[0292] v. When the number of spatial or temporal Merge candidates (for example, including adjacent spatial domains and non-adjacent spatial domains, TMVP, STMVP, ATMVP, etc.) that have been placed in the Merge candidate list according to other coding and decoding methods (for example, the Merge derivation process of HEVC design or JEM design) is less than the maximum allowed Merge candidate minus a given threshold.

[0293] 1. In one example, the threshold is set to 1 or 0.

[0294] 2. Alternatively, the threshold may be signaled or predefined in the SPS / PPS / sequence, picture, slice header / slice.

[0295] 3. Alternatively, the threshold may be adaptively changed from block to block. For example, it may depend on codec block information such as block size / block shape / slice type, and / or on the number of available spatial or temporal Merge candidates.

[0296] 4. In another example, when the number of certain Merge candidates that have been put into the Merge candidate list is less than the maximum allowed Merge candidate minus a given threshold. "Some Merge candidates" may be spatial candidates or non-adjacent Merge candidates as in HEVC.

[0297] vi. Pruning can be applied before adding motion candidates to the Merge candidate list.

[0298] 1. In one example, the motion candidate may be pruned from all or part of the available spatial or temporal Merge candidates (e.g., including adjacent spatial and non-adjacent spatial, TMVP, STMVP, ATMVP, etc.) in the Merge candidate list from other coding methods.

[0299] 2. Motion candidates and sub-block based motion candidates (eg, ATMVP, STMVP) may not be pruned.

[0300] 3. In one example, the current motion candidate and all or part of the available motion candidates in the Merge candidate list (inserted before the current motion candidate) may be pruned.

[0301] 4. The number of pruning operations related to motion candidates (i.e., the number of times a motion candidate needs to be compared with other candidates in the Merge list) can depend on the number of available spatial or temporal Merge candidates. For example, when checking a new motion candidate, if there are M candidates available in the Merge list, the new motion candidate can be compared only with the first K (K≤M) candidates. If the pruning function returns false (e.g., not the same as any of the first K candidates), the new motion candidate is considered different from all of the M candidates and it can be added to the Merge candidate list. In one example, K is set to min(K, 2).

[0302] 5. In one example, the newly appended motion candidate is only compared with the first N candidates in the Merge candidate list. For example, N = 3, 4 or 5. N can be signaled from the encoder to the decoder.

[0303] 6. In one example, new motion candidates to be checked are compared only with the last N candidates in the Merge candidate list. For example, N = 3, 4, or 5. N can be signaled from the encoder to the decoder.

[0304] 7. In one example, how to select the candidates previously added in the list to compare with the new motion candidates from the table may depend on where the previously added candidates were derived from.

[0305] a. In one example, the motion candidates in the lookup table can be compared with the candidates derived from given temporal and / or spatial neighboring blocks.

[0306] b. In one example, different entries of motion candidates in the lookup table may be compared with different previously added candidates (ie, candidates derived from different locations).

[0307] Example C3: The motion candidate(s) included in the lookup table may be used as a predictor of the motion information of the codec block.

[0308] a. They can be used for AMVP mode encoding and decoding, i.e., motion candidates can be checked during the AMVP candidate list derivation process.

[0309] b. Checking for motion candidates in the lookup table can be enabled in the following cases:

[0310] i. After inserting the TMVP candidate, the AMVP candidate list is not full;

[0311] ii. Just before inserting the TMVP candidate, after selecting from the spatial neighbors and pruning, the AMVP candidate list is not full;

[0312] iii. When there is no AMVP candidate from the upper neighboring block that does not need to be scaled and / or there is no AMVP candidate from the left neighboring block that does not need to be scaled;

[0313] iv. Pruning can be applied before adding motion candidates to the AMVP candidate list.

[0314] v. Rules similar to those mentioned in Item 5.(3)(4) can be applied to the AMVP model.

[0315] c. Check motion candidates that have the same reference picture as the current reference picture.

[0316] i. Alternatively, additionally, motion candidates with reference pictures different from the current reference picture are also checked (in case of scaling MV).

[0317] ii. Alternatively, first check all motion candidates having the same reference picture as the current reference picture, and then check motion candidates having a reference picture different from the current reference picture.

[0318] iii. Alternatively, follow the same steps in Merge to check motion candidates.

[0319] Example C4: The order of checking motion candidates in the lookup table is defined as follows (assuming that K (K>=1)) motion candidates are allowed to be checked):

[0320] a. Look up the last K motion candidates in the table, eg, in descending order of the entry index of the LUT.

[0321] b. Top K%L candidates, where L is the size of the lookup table, when K>=L, e.g., in descending order of the entry index of the LUT.

[0322] c. When K>=L, look up all candidates (L candidates) in the table, based on order. In one example, check the first K%L candidates in the table in descending order of the entry index of the LUT, and then check the last (L-K%L) candidates in descending order of the entry index.

[0323] d. Alternatively, further, based on descending order of motion candidate index.

[0324] e. Alternatively, further, based on ascending order of motion candidate index.

[0325] f. Alternatively, K motion candidates are selected based on candidate information, such as the position associated with the motion candidate and the distance of the current block.

[0326] i. In one example, select the K closest motion candidates.

[0327] ii. In one example, when calculating the distance, the candidate information may also take into account the block shape.

[0328] g. In one example, the order of checking K motion candidates from a table of L candidates can be defined as: sequentially selecting the candidates whose index is equal to a0, a0+T0, a0+T0+T1, a0+T0+T1+T2, ... a0+T0+T1+T2+...+T K-1 Those candidates where a0 and T i (i is 0...K-1) is an integer value.

[0329] i. In one example, a0 is set to 0 (i.e., the first entry in the table for the motion candidate). Alternatively, a0 is set to (K - L / K). The arithmetic operation " / " is defined as integer division with the result truncated towards zero. Alternatively, a0 is set to any integer between 0 and L / K.

[0330] 1. Alternatively, the value of a0 may depend on the coding information of the current block and the neighboring blocks.

[0331] ii. In one example, all intervals T i (i is 0...K-1) are the same, such as

[0332] L / K. The arithmetic operation " / " is defined as integer division with truncation toward zero.

[0333] iii. In one example, (K, L, a0, T i ) is set to (4,16,0,4) or (4,12,0,3) or (4,8,0,1) or (4,16,3,4) or (4,12,2,3) or (4,8,1,2). For all i, T i All the same.

[0334] iv. This method can only be applied when K is less than L.

[0335] v. Alternatively, in addition, when K is greater than or equal to a threshold, item 7.c may be applied. The threshold may be defined as L, or it may depend on K or vary adaptively from block to block. In one example, the threshold may depend on the number of available motion candidates in the list before adding new motion candidates from the lookup table.

[0336] h. In one example, the order of checking K motion candidates from a table of L candidates can be defined as: sequentially selecting the candidates whose index is equal to a0, a0-T0, a0-T0-T1, a0-T0-T1-T2, ... a0-T0-T1-T2- ... -T K-1 Those candidates where a0 and T i(i is 0...K-1) is an integer value.

[0337] i. In one example, a0 is set to L-1 (ie, the last entry of the motion candidate in the table). Alternatively, a0 is set to any integer between L-1-L / K and L-1.

[0338] ii. In one example, all intervals T i (i is 0...K-1) are all the same, such as L / K.

[0339] iii. In one example, (K, L, a0, T i ) is set to (4, 16, L-1, 4) or (4, 12, L-1, 3) or (4, 8, L-1, 1) or (4, 16, L-4, 4) or (4, 12, L-3, 3) or (4, 8, L-2, 2). For all i, T i All the same.

[0340] iv. This method can only be applied when K is less than L.

[0341] Alternatively, further, when K is greater than or equal to a threshold, item 7.c may be applied. The threshold may be defined as L, or it may depend on K or vary adaptively from block to block. In one example, the threshold may depend on the number of available motion candidates in the list before adding new motion candidates from the lookup table.

[0342] i. How many and / or how to select motion candidates from the lookup table may depend on codec information such as block size / block shape.

[0343] i. In one example, for smaller block sizes, instead of selecting the last K motion candidates, other K motion candidates (not starting from the last one) may be selected.

[0344] ii. In one example, the codec information may be AMVP or Merge mode.

[0345] iii. In an example, the codec information may be an affine mode, a non-affine AMVP mode, or a non-affine Merge mode.

[0346] iv. In an example, the codec information may be an affine AMVP (inter) mode, an affine Merge mode, or a non-affine AMVP mode or a non-affine Merge mode.

[0347] v. In one example, the codec information may be a current picture reference (CPR) mode or not a CPR mode.

[0348] vi. Alternatively, how to select a motion candidate from the lookup table may also depend on the number of motion candidates in the lookup table, and / or the number of available motion candidates in the list before adding a new motion candidate from the lookup table.

[0349] j. In one example, the maximum number of motion candidates to be checked in the lookup table (ie, which can be added to the Merge / amvp candidate list) can depend on the number of available motion candidates in the lookup table (given by N avaiMCinLUT ), and / or the maximum allowed motion candidates to be added (denoted by NUM maxMC ), and / or the number of available candidates in the candidate list (denoted by N) before testing candidates from the lookup table avaiC express).

[0350] i. In one example, the maximum number of motion candidates to be checked in the lookup table is set to (N avaiMCinLUT , NUM maxMC , N avaiC ) is the minimum value among .

[0351] ii. Alternatively, the maximum number of motion candidates to be checked in the lookup table is set to (N avaiMCinLUT , NUM maxMC -N avaiC ) is the minimum value among .

[0352] iii. In one example, N avaiC N represents the number of insertion candidates derived from neighboring blocks in the spatial or temporal domain (adjacent and / or non-adjacent). Alternatively, in addition, the number of sub-block candidates (such as AMTVP, STMVP) is not counted in N avaiC middle.

[0353] iv.NUM maxMC It may depend on the codec mode. For example, for Merge mode and AMVP mode, NUM maxMC Can be set to different values. In one example, for Merge mode, NUM maxMC Can be set to 4, 6, 8, 10, etc. For AMVP mode, NUM maxMC Can be set to 1, 2, 4, etc.

[0354] v.Alternatively, NUM maxMC It may depend on other codec information, such as block size, block shape, slice type, etc.

[0355] k. The order in which the different lookup tables are checked is defined in the use of lookup tables in the next subsection.

[0356] l. Once the Merge / AMVP candidate list reaches the maximum allowed number of candidates, the checking process will terminate.

[0357] m. Once the Merge / AMVP candidate list reaches the maximum allowed number of candidates minus a threshold (Th), the checking process will terminate. In one example, Th may be predefined as a positive integer value, such as 1, or 2, or 3. Alternatively, Th may be adaptively changed from block to block. Alternatively, Th may be signaled in the SPS / PPS / slice header, etc. Alternatively, Th may also depend on block shape / block size / codec mode, etc. Alternatively, Th may depend on how many available candidates are before adding motion candidates from the LUT.

[0358] n. Alternatively, it will terminate once the number of added motion candidates reaches the maximum allowed number of motion candidates. The maximum allowed number of motion candidates can be signaled or predefined. Alternatively, the maximum allowed number of motion candidates can also depend on block shape / block size / coding mode, etc.

[0359] o. One syntax element indicating the table size and the number of motion candidates allowed to be checked (ie, K=L) may be signaled in the SPS, PPS, slice header, slice header.

[0360] Example C5: The use of the lookup table for motion information coding of a block may be enabled / disabled by signaling in an SPS, PPS, slice header, slice header, CTU, CTB, CU or PU, or in an area covering multiple CTUs / CTBs / CUs / PUs.

[0361] Example C6: Whether prediction from a lookup table is applied may also depend on codec information. When inference is not applied to a block, additional signaling of a prediction indication is skipped. Alternatively, when inference is not applied to a block, no motion candidates of a lookup table need be accessed, and checking of relevant motion candidates is omitted.

[0362] a. Whether to apply prediction from a lookup table may depend on the block size / block shape. In one example, for smaller blocks, such as 4x4, 8x4 or 4x8 blocks, prediction from a lookup table is not allowed to be performed.

[0363] b. Whether to apply prediction from a lookup table may depend on whether the block is encoded in AMVP mode or Merge mode. In one example, for AMVP mode, prediction from a lookup table is not allowed to be performed.

[0364] c. Whether to apply prediction from a lookup table may depend on whether the block is coded with affine motion or other kind of motion (such as translational motion). In one example, for affine mode, prediction from a lookup table is not allowed to be performed.

[0365] Example C7: Motion candidates from a lookup table in a previously coded frame / slice / slice may be used to predict motion information of a block in a different frame / slice / slice.

[0366] a. In one example, only the lookup table associated with the reference picture of the current block can be used to encode the current block.

[0367] b. In one example, only the lookup tables associated with pictures having the same slice type and / or the same quantization parameter as the current block may be used to encode the current block.

[0368] Update of lookup table

[0369] Example D1: After encoding or decoding a block with motion information (ie, IntraBC mode, inter coding mode), one or more lookup tables may be updated.

[0370] a. In one example, whether to update the lookup table may reuse the rule for selecting the lookup table, for example, when a lookup table may be selected for encoding / decoding the current block, the selected lookup table may also be updated after encoding / decoding the block.

[0371] b. The lookup table to be updated may be selected based on codec information and / or the location of the block / LCU.

[0372] c. If the block is encoded and decoded with directly signaled motion information (such as AMVP mode, MMVD mode for normal / affine inter mode, AMVR mode for normal / affine inter mode), the motion information of the block can be added to the lookup table.

[0373] i. Alternatively, if a block is encoded with motion information directly inherited from spatially neighboring blocks without any refinement (eg, a spatial Merge candidate without refinement), the motion information of the block should not be added to the lookup table.

[0374] ii. Alternatively, if a block is encoded with motion information inherited directly from spatially neighboring blocks with refinement (such as DMVR, FRUC), the motion information of the block should not be added to any lookup table.

[0375] iii. Alternatively, if a block is encoded with motion information directly inherited from a motion candidate stored in a lookup table, the motion information of the block should not be added to any lookup table.

[0376] iv. In one example, such motion information may be added directly to the lookup table, such as to the last entry of the table, or to the entry storing the next available motion candidate.

[0377] v. Alternatively, such motion information may be added directly to the lookup table without pruning, e.g. without any pruning.

[0378] vi. Alternatively, such motion information can be used to reorder the lookup table.

[0379] vii. Alternatively, such motion information can be used to update the lookup table with limited pruning (eg, comparison with the latest one in the lookup table).

[0380] d. Select M (M>=1) representative positions within the block and update the lookup table using the motion information associated with the representative positions.

[0381] i. In one example, a representative position is defined as one of four corner positions within a block (e.g., Fig.26 C0-C3 in ).

[0382] ii. In one example, the representative location is defined as the center location within the block (e.g., Fig.26 of Ca-Cd).

[0383] iii. When sub-block prediction is not allowed for a block, M is set to 1.

[0384] iv. When sub-block prediction is allowed for a block, M can be set to 1 or the total number of sub-blocks or any other value between [1, the number of sub-blocks] (excluding the endpoints).

[0385] v. Alternatively, when sub-block prediction is allowed for a block, M can be set to 1 and the representative sub-block is selected based on:

[0386] 1. The frequency of the motion information used,

[0387] 2. Whether it is a bidirectional prediction block,

[0388] 3. Based on reference picture index / reference picture,

[0389] 4. The motion vector difference compared to other motion vectors (e.g., select the largest MV difference),

[0390] 5. Other codec information.

[0391] e. When selecting M (M>=1) sets of representative positions to update the lookup table, other conditions may be checked before adding them as additional motion candidates to the lookup table.

[0392] i. Pruning a new set of motion information can be applied to existing motion candidates in the lookup table.

[0393] ii. In one example, the new motion information set should not be identical to any or part of the existing motion candidates in the lookup table.

[0394] iii. Alternatively, for the same reference picture from a new motion information set and an existing motion candidate, the MV difference should be no less than one / multiple thresholds. For example, the horizontal and / or vertical components of the MV difference should be greater than 1 pixel distance.

[0395] iv. Alternatively, the new motion information set is pruned with only the last K candidates or, when K>L, the first K%L existing motion candidates to allow reactivation of old motion candidates.

[0396] v. Alternatively, no pruning is applied.

[0397] f. If M motion information sets are used to update the lookup table, the corresponding counter should be increased by M.

[0398] g. Assuming that the counter of the lookup table to be updated is denoted by K before encoding the current block, after encoding the block, for a selected motion information set (using the above method), it is added as an additional motion candidate with an index equal to K%L (where L is the lookup table size). Fig.27A and Fig.27B An example is shown in .

[0399] i. Alternatively, it is added as an additional motion candidate with index equal to min(K+1, L-1). Also, alternatively, if K>=L, the first motion candidate (index equal to 0) is removed from the lookup table and the subsequent K candidate indices are reduced by 1.

[0400] ii. For the above two methods (adding the new motion candidate to the entry with index equal to K%L, or adding it with index equal to min(K+1, L-1)), they both try to keep the most recent few motion information sets from previously encoded blocks, regardless of whether there are the same / similar motion candidates.

[0401] iii. Alternatively, when adding a new motion information set as a motion candidate to the LUT, a redundancy check is first applied. In this case, the LUT retains the most recent few motion information sets from the previously encoded and decoded blocks, however, the redundant motion information sets can be removed from the lookup table. Such a method is called a LUT update method based on redundancy removal.

[0402] 1. If there are redundant motion candidates in the LUT, the counter associated with the LUT may not increase or decrease.

[0403] 2. Redundancy check can be defined as a pruning process during the Merge candidate list construction process, for example, checking whether the reference pictures / reference picture indexes are the same, and whether the motion vector differences are within the range or the same.

[0404] 3. If a redundant motion candidate is found in the LUT, the redundant motion candidate is moved from its current position to the last position of the LUT.

[0405] a. Similarly, if a redundant motion candidate is found in the LUT, it is removed from the LUT. In addition, all motion candidates inserted into the LUT after the redundant motion candidate are shifted forward to refill the entry of the removed redundant motion candidate. After the shift, the new motion candidate is added to the LUT.

[0406] b. In this case, the counter remains unchanged.

[0407] c. Once redundant motion candidates are identified in the LUT, the redundancy checking process terminates.

[0408] 4. Multiple redundant motion candidates may be identified. In this case, they are all removed from the LUT. In addition, all remaining motion candidates may be moved forward in sequence.

[0409] a. In this case, the counter is decremented (the number of redundant motion candidates minus 1).

[0410] b. After maxR redundant motion candidates are identified (maxR is a positive integer variable), the redundancy check process terminates.

[0411] 5. The redundancy checking process may start from the first motion candidate to the last motion candidate (ie, in the order of addition to the LUT, in the order of the decoding process of the blocks the motion information comes from).

[0412] 6. Alternatively, when redundant motion candidates exist in the LUT, instead of removing one or more redundant motion candidates from the LUT, virtual motion candidates may be derived from the redundant motion candidates and may be used to replace the redundant motion candidates.

[0413] a. A virtual motion candidate can be derived from a redundant motion candidate by adding offset(s) to the horizontal and / or vertical components of one or more motion vectors; or the average of two motion vectors if pointing to the same reference picture. Alternatively, a virtual motion candidate can be derived from any function that takes a motion vector from a lookup table as input. Exemplary functions are: adding two or more motion vectors; averaging two or more motion vectors. The motion vectors can be scaled before being input to the function.

[0414] b. Virtual motion candidates can be added to the same position as redundant motion candidates.

[0415] c. Virtual motion candidates may be added before all other motion candidates (eg, starting from the smallest entry index, such as zero).

[0416] d. In one example, it is only applied under certain conditions, such as when the current LUT is not full.

[0417] 7. The LUT update method based on redundancy removal can be called under certain conditions, such as:

[0418] a. The current block is encoded and decoded in Merge mode,

[0419] b. The current block is encoded and decoded in AMVP mode, but at least one component of the MV difference is non-zero;

[0420] c. The current block is encoded or not encoded using a sub-block based motion prediction / motion compensation method (eg, not encoded or decoded in affine mode);

[0421] d. The current block is encoded and decoded in Merge mode, and the motion information is associated with a certain type (for example, from a spatial neighboring block, from a left neighboring block, or from a temporal block).

[0422] h. After encoding / decoding a block, one or more lookup tables may be updated by simply inserting the M motion information sets to the end of the table, ie after all existing candidates.

[0423] i. Alternatively, furthermore, some existing motion candidates in the table may be removed.

[0424] 1. In one example, if the table is full after inserting M motion information sets, the first few entries of the motion candidates may be removed from the table.

[0425] 2. In one example, if the table is full before inserting the M motion information sets, the first few entries of the motion candidates may be removed from the table.

[0426] ii. Alternatively, furthermore, if the block is encoded with a motion candidate from the table, the motion candidates in the table may be reordered so that the selected motion candidate is placed in the last entry of the table.

[0427] i. In one example, before encoding / decoding a block, a lookup table may include a list of HMVP0, HMVP1, HMVP2, ..., HMVP K-1 、HMVP K 、HMVP K+1 , …, HMVP L-1Represents the motion candidate, where HMVP i Represents the i-th entry in the lookup table. If the block is from HMVP K predicted (K is in the range [0, L-1], inclusive), after encoding / decoding the block, the lookup table is reordered as: HMVP0, HMVP1, HMVP2, ..., HMVP K-1 , HMVP K , HMVP K+1 ,…,HMVP L-1 , HMVP K .

[0428] j. After encoding and decoding an intra-frame constrained block, the lookup table can be cleared.

[0429] k. If an entry of motion information is added to the lookup table, further entries of motion information may also be added to the table by being derived from the motion information. In this case, a counter associated with the lookup table may be incremented by more than one.

[0430] i. In one example, the MV of an entry of motion information is scaled and placed in a table;

[0431] ii. In one example, the MV of the entry of motion information is increased by (dx, dy) and placed in the table;

[0432] iii. In one example, the average of the MVs of two or more entries of motion information is calculated and put into a table.

[0433] Example D2: If a block is located at a picture / slice / slice boundary, then updates of the lookup table may never be allowed.

[0434] Example D3: The motion information on the upper side of the LCU row may be disabled to encode the current LCU row.

[0435] a. In this case, the number of available motion candidates may be reset to 0 at the beginning of a new slice / slice / LCU row.

[0436] Example D4: At the beginning of encoding a slice / slice with a new temporal layer index, the number of available motion candidates may be reset to 0.

[0437] Example D5: The lookup table may be continuously updated with one slice / slice / LCU row / slices having the same temporal layer index.

[0438] a. Alternatively, the lookup table may be updated only after encoding / decoding every S (S>=1) CTUs / CTBs / CUs / CBs or after encoding / decoding a certain region (eg, size equal to 8×8 or 16×16).

[0439] b. Alternatively, the lookup table may be updated only after every S (S>=1) blocks (e.g., CU / CB) are coded / decoded with certain modes (e.g., S inter-coded blocks). Alternatively, the lookup table may be updated only after every S (S>=1) inter-coded blocks (e.g., CU / CB) that are not coded / decoded with sub-block-based motion prediction / motion compensation methods (e.g., not coded / decoded with affine and / or ATMVP modes) are coded / decoded.

[0440] c. Alternatively, the lookup table can be updated only when the upper left coordinate of the codec / decode block meets some conditions. For example, the lookup table is updated only when (x&M==0)&&(y&M==0), where (x, y) is the upper left coordinate of the codec / decode block. M is an integer such as 2, 4, 8, 16, 32, or 64.

[0441] d. Alternatively, a lookup table may stop updating once it reaches the maximum allowed counter.

[0442] e. In one example, the counter may be predefined. Alternatively, it may be signaled in a video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), slice header, slice header, codec tree unit (CTU), codec tree block (CTB), codec unit (CU) or prediction unit (PU), a region covering multiple CTU / CTB / CU / PU. Example D6: The lookup table update process may be called within a different process.

[0443] a. In one example, for a block encoded and decoded in Merge mode, the lookup table update process may be called after decoding a Merge candidate or after constructing a Merge list or after decoding motion information with and / or without refinement.

[0444] b. In one example, for blocks encoded in AMVP mode, the lookup table update process may be invoked after decoding motion information with and / or without refinement.

[0445] c. When and / or how to update the lookup table may depend on codec mode, block size, video processing data unit, low latency check, etc.

[0446] i. In one example, when a block is encoded or decoded in AMVP mode, the lookup table can be directly updated without pruning.

[0447] ii. Alternatively, when a block is encoded or decoded in Merge mode, the lookup table may be updated with pruning performed.

[0448] iii. Alternatively, when a block is encoded in Merge mode and its motion information is derived from spatial and / or temporal blocks, the lookup table can be updated with pruning.

[0449] iv. Alternatively, when a block is encoded in Merge mode and its motion information is derived from motion candidates in the lookup table, the lookup table can be reordered without pruning.

[0450] v. Alternatively, when a block is encoded in Merge mode and its motion information is derived from virtual candidates in the lookup table (eg, combined bidirectional, paired, zero motion vector candidates), the lookup table may not be updated.

[0451] vi. Alternatively, when a block is encoded and decoded in sub-block Merge mode and / or triangle Merge mode, the lookup table may not be updated.

[0452] vii. Alternatively, when a block is coded in Merge with Motion Vector Difference (MMVD) mode and its motion information is derived from spatial and / or temporal blocks, the lookup table can be updated directly.

[0453] viii. In one example, when a block is encoded or decoded in illumination compensation (IC) mode and / or overlapped block motion compensation (OBMC) mode and / or decode-side motion vector derivation (DMVD) mode, the lookup table may not be updated. Alternatively, when a block is encoded or decoded in such a mode, the lookup table may be updated.

[0454] Additional Example Embodiments

[0455] A history-based MVP (HMVP) method is proposed, where the HMVP candidate is defined as the motion information of the previously coded block. A table with multiple HMVP candidates is maintained during the encoding / decoding process. When a new slice is encountered, the table is cleared. Whenever there is an inter-coded block, the associated motion information is added to the last entry of the table as a new HMVP candidate. The entire encoding and decoding process is Fig.31 Depicted in.

[0456] In one example, the table size is set to L (eg, L=16 or 6, or 44), where L indicates that at most L HMVP candidates can be added to the table.

[0457] In one embodiment (corresponding to Example 11.gi), if there are more than L HMVP candidates from previously encoded blocks, a First-In-First-Out (FIFO) rule is applied so that the table always contains the most recent L previously encoded motion candidates. Fig.32 An example of applying the FIFO rule to remove an HMVP candidate and add a new one to the table used in the proposed method is depicted.

[0458] In another embodiment (corresponding to invention 11.g.iii), whenever a new motion candidate is added (such as the current block is inter-coded and in non-affine mode), a redundancy check process is first applied to identify whether the same or similar motion candidate exists in the LUT.

[0459] Some examples are described below:

[0460] Fig.33A An example is shown where the LUT is full before a new motion candidate is added.

[0461] Fig.33B An example is shown where the LUT is not full before a new motion candidate is added.

[0462] Fig.33A and Fig.33B An example of a LUT updating method based on redundancy removal is shown together (one redundant motion candidate is removed).

[0463] Fig.34A and Fig.34B Example implementations for two cases of the LUT updating method based on redundancy removal are shown (multiple redundant motion candidates are removed, there are 2 candidates in the figure).

[0464] Fig.34A An example case is shown where the LUT is full before adding a new motion candidate.

[0465] Fig.34B An example case is shown where the LUT is not full before adding a new motion candidate.

[0466] HMVP candidates can be used in the Merge candidate list construction process. All HMVP candidates from the last entry to the first entry in the table (or the last K0 HMVPs, for example, K0 is equal to 16 or 6) are inserted after the TMVP candidate. Pruning is applied to HMVP candidates. Once the total number of available Merge candidates reaches the maximum allowed Merge candidate notified by signaling, the Merge candidate list construction process is terminated. Alternatively, once the total number of added motion candidates reaches a given value, the motion candidate is terminated from the LUT.

[0467] Similarly, HMVP candidates can also be used in the AMVP candidate list construction process. The motion vectors of the last K1 HMVP candidates in the table are inserted after the TMVP candidate. Only HMVP candidates with the same reference picture as the AMVP target reference picture are used to construct the AMVP candidate list. Pruning is applied to HMVP candidates. In one example, K1 is set to 4.

[0468] References

[0469] [1] "Overview of the High Efficiency Video Coding (HEVC) Standard", GaryJ.Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand, IEEE Transactionson Circuits and Systems for Video Technology, Vol.22, No.12, December 2012.

[0470] [2] "Overview of the H.264 / AVC video coding standard", Ajay Luthra, Pankaj Topiwala, Proceedings of SPIE Vol.5203Applications of Digital ImageProcessing XXVI.

[0471] [3] J.Chen, E.Alshina, GJSullivan, J.-R.Ohm, J.Boyce, "Algorithmdescription of Joint Exploration Test Model 7(JEM7)," JVET-G1001, Aug.2017.

[0472] [4]JEM-7.0: https: / / jvet.hhi.fraunhofer.de / svn / svn_HMJEMSoftware / tags / HM-16.6-JEM-7.0.

[0473] [5]H.265 / HEVC,https: / / www.itu.int / rec / T-REC-H.265

[0474] [6]"Multi-Type-Tree",JVET-D0117

[0475] [7]International Patent Application WO2016 / 091161

[0476] [8] "Description of SDR, HDR and 360° video coding technology proposal by Qualcomm and Technicolor–low and high complexity versions", JVET-J0021.

[0477] Fig.28A is a block diagram of a video processing device 2800. The device 2800 can be used to implement one or more of the methods described herein. The device 2800 can be embodied in a smart phone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. The device 2800 may include one or more processors 2802, one or more memories 2804, and video processing hardware 2806. The processor(s) 2802 may be configured to implement one or more methods described in this document. The memory(s) may be used to store data and code for implementing the methods and techniques described herein. The video processing hardware 2806 may be used to implement some of the techniques described in this document in hardware circuits.

[0478] Fig.28B is another example of a block diagram of a video processing system in which the disclosed technology may be implemented. Fig.28B is a block diagram illustrating an example video processing system 3100 in which various techniques disclosed herein may be implemented. Various embodiments may include some or all of the components of system 3100. System 3100 may include an input 3102 for receiving video content. The video content may be received in a raw or uncompressed format, such as 8 or 10 bit multi-component pixel values, or may be received in a compressed or encoded format. Input 3102 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, a passive optical network (PON), etc., and wireless interfaces such as Wi-Fi or cellular interfaces.

[0479] System 3100 may include a codec component 3104, which may implement various codecs or encoding methods described in this document. Codec component 3104 may reduce the average bit rate of the video from input 3102 to the output of codec component 3104 to generate a codec representation of the video. Therefore, codec technology is sometimes referred to as video compression or video transcoding technology. The output of codec component 3104 may be stored, or transmitted via a connected communication, as shown in component 3106. Component 3108 may use a bitstream (or codec) representation of the storage or communication of the video received at input 3102 to generate pixel values ​​or displayable video sent to display interface 3110. The process of generating a video that a user can watch from a bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "codec" operations or tools, it should be understood that codec tools or operations are used at encoders, and the corresponding decoding tools or operations of the inverse codec results will be performed by decoders.

[0480] Examples of peripheral bus interfaces or display interfaces may include universal serial bus (USB) or high definition multimedia interface (HDMI) or display port, etc. Examples of storage interfaces include SATA (serial advanced technology attachments), PCI, IDE interfaces, etc. The technology described in this document may be embodied in various electronic devices, such as mobile phones, laptops, smart phones, or other devices capable of performing digital data processing and / or video display.

[0481] In some embodiments, the video encoding and decoding method can use the reference Fig.28A or Fig.28B The method is implemented by the device implemented on the hardware platform.

[0482] Fig.29 2 is a flow chart of an example of a video processing method 2900. The method 2900 includes, at step 2902, maintaining one or more motion candidate tables. The method 2900 also includes, at step 2904, using the motion candidate table to perform a conversion between a current video block of a video and a codec representation of the current video block in sequence using a plurality of video processing operations. The method 2900 also includes, at step 2906, updating the motion candidate table due to completion of a particular processing operation.

[0483] Fig.303000. The method 3000 includes, at step 3002, maintaining one or more motion candidate tables. The method 3000 also includes, at step 3004, determining i) whether to update one of the one or more motion candidate tables, or ii) how to update the motion candidate table based on characteristics of the current video block including codec mode, size, video processing data unit, or low latency check. The method 3000 also includes, at step 3006, performing a conversion between a current video block of a video and a codec representation of the video based on the determination.

[0484] With respect to method 2900 and method 3000, in some embodiments, the motion information includes at least one of the following: a prediction direction, a reference image index, a motion vector value, a light intensity compensation flag, an affine flag, a motion vector difference precision, and a motion vector difference value. In addition, the motion information may also include block position information indicating a source of the motion information. In some embodiments, the video block may be a CU or a PU, and the portion of the video may correspond to one or more video slices or one or more video pictures.

[0485] In some embodiments, each LUT includes an associated counter, wherein the counter is initialized to a zero value at the beginning of the video portion and incremented for each encoded video region in the video portion. The video region includes one of a codec tree unit, a codec tree block, a codec unit, a codec block, or a prediction unit. In some embodiments, for a corresponding LUT, the counter indicates the number of motion candidates removed from the corresponding LUT. In some embodiments, the motion candidate set may have the same size for all LUTs. In some embodiments, the video portion corresponds to a stripe of video, and wherein the number of LUTs is equal to N*P, where N is an integer representing the number of LUTs for each decoding thread, and P is an integer representing the number of maximum codec unit rows or the number of slices in a stripe of video. Additional details of methods 2900 and 3000 are described in the examples provided in Section 4 and in the examples listed below.

[0486] 1. A video processing method, comprising: maintaining one or more motion candidate tables; using the motion candidate tables to sequentially use multiple video processing operations to perform conversion between a current video block of a video and a codec representation of the current video block; and updating the motion candidate tables due to completion of a particular processing operation.

[0487] 2. The method of example 1, wherein the identification of the specific processing operation depends on codec characteristics of the current video block.

[0488] 3. The method of example 1, wherein the current video block is encoded and decoded in Merge mode, and wherein the motion candidate table is updated after at least one of the following operations as part of the conversion:

[0489] (i) Decode Merge candidates,

[0490] (ii) Construct a Merge list including Merge candidates,

[0491] (iii) decoding motion information without refinement, or

[0492] (iv) Decoding has motion information for refinement.

[0493] 4. The method of example 3, wherein the current video block is encoded and decoded in Merge mode, and wherein the decoded motion information is used to update the motion candidate table before the decoded motion information is refined using decoding-side motion refinement (DMVR).

[0494] 5. The method according to Example 3, wherein the current video block is encoded and decoded in Merge mode, and wherein after the decoded motion information is refined using Merge with Motion Vector Difference (MMVD), the decoded motion information is used to update the motion candidate table.

[0495] 6. The method of example 1, wherein the current video block is encoded and decoded in an advanced motion vector prediction (AMVP) mode, and wherein the motion candidate table is updated after decoding the motion information with or without refinement.

[0496] 7. A video processing method, comprising: maintaining one or more motion candidate tables; determining at least one of the following based on characteristics of a current video block including a codec mode, size, video processing data unit, or low latency check: i) whether to update a motion candidate table in the one or more motion candidate tables, or ii) how to update the motion candidate table; and performing conversion between the current video block of a video and a codec representation of the video based on the determination.

[0497] 8. The method of example 7, wherein the current video block is encoded in an advanced motion vector prediction (AMVP) mode, and wherein the motion candidate table is updated without pruning.

[0498] 9. The method according to example 7, wherein the current video block is encoded and decoded in Merge mode, and wherein the motion candidate table is updated when pruning is performed.

[0499] 10. The method of example 8 or 9, wherein pruning comprises comparing motion information associated with the current video block with at least one motion candidate in a motion candidate table.

[0500] 11. The method of example 9, wherein the motion information of the current video block is derived from a spatial domain block or a temporal domain block.

[0501] 12. The method according to Example 7, wherein the current video block is encoded and decoded in Merge mode, and the motion information of the current video block is derived from motion candidates in a motion candidate table, and the motion candidate table is reordered without pruning.

[0502] 13. The method according to Example 7, wherein the current video block is encoded and decoded in Merge mode, and the motion information of the current video block is derived from a virtual candidate in a motion candidate table, and the motion candidate table is not updated.

[0503] 14. The method according to Example 7, wherein the current video block is encoded and decoded in a sub-block Merge mode and / or a triangle Merge mode, and the motion candidate table is not updated.

[0504] 15. The method according to Example 7, wherein the current video block is encoded and decoded in Merge with Motion Vector Difference (MMVD) mode, and the motion information of the current video block is derived from a spatial domain block or a temporal domain block, and the motion candidate table is updated.

[0505] 16. The method of example 7, wherein the current video block is encoded and decoded in one or more of an illumination compensation (IC) mode, an overlapped block motion compensation (OBMC) mode, or a decoding side motion vector derivation (DMVD) mode, and the motion candidate table is not updated.

[0506] 17. The method of Example 7, wherein the current video block is coded in one or more of an illumination compensation (IC) mode, an overlapped block motion compensation (OBMC) mode, or a decoding side motion vector derivation (DMVD) mode to update the motion candidate table.

[0507] 18. A method according to any one of Examples 1 to 17, wherein the motion candidate table is a historical motion vector prediction (HMVP) table.

[0508] 19. A method according to any one of Examples 1 to 18, wherein performing the conversion includes generating a codec representation from the current video block.

[0509] 20. A method according to any one of Examples 1 to 18, wherein performing the conversion includes generating the current video block from a codec representation.

[0510] 21. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method of one or more of Examples 1 to 20.

[0511] 22. A computer program product stored on a non-transitory computer readable medium, the computer program product comprising program code for executing the method of one or more of Examples 1 to 20.

[0512] From the foregoing, it will be appreciated that specific embodiments of the disclosed technology have been described herein for purposes of illustration, but that various modifications may be made without departing from the scope of the invention. Accordingly, the disclosed technology is not to be limited, except by the appended claims.

[0513] The disclosed and other embodiments, modules and functional operations described in this document may be implemented in digital electronic circuit systems, or in computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or in a combination of one or more of them. The disclosed and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium, for operation by a data processing device or controlling the operation of the data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances that affect machine-readable propagation signals, or a combination of one or more of them. The term "data processing device" encompasses all devices, equipment, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the device may include code that creates an operating environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal, such as a machine-generated electrical signal, an optical signal, or an electromagnetic signal, that is generated to encode information for transmission to a suitable receiver device.

[0514] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a document that stores other programs or data (e.g., one or more scripts stored in a markup language document), in a single document dedicated to the program in question, or in multiple coordination documents (e.g., a document storing one or more modules, subroutines, or portions of code). A computer program can be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0515] The processes and logic flows described in this document can be performed by one or more programmable processors running one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, and the apparatus can also be implemented as special purpose logic circuits, such as FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit).

[0516] Processors suitable for the operation of computer programs include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include or be operably coupled to one or more large-capacity storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, to receive data from it or to transmit data to it or to receive data and transmit data. However, the computer does not need to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

[0517] Although this patent document contains many details, these details should not be interpreted as limitations on any invention or the scope that can be claimed, but as descriptions of features that can be specific to specific embodiments of specific inventions. Certain features described in the context of separate embodiments in this patent document may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable sub-combination. In addition, although the above features may be described as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be cut out from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0518] Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all of the illustrated operations be performed, in order to achieve the desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0519] Only a few implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A video processing method, comprising: maintaining one or more motion candidate tables, wherein each motion candidate table includes one or more motion candidates, at least one of the one or more motion candidates is derived from at least one previous video block, and an arrangement of the motion candidates in the motion candidate table is based on an order in which the motion candidates are added to the motion candidate table; performing a motion candidate list derivation process to derive a motion candidate list for the first video block; deriving motion information of the first video block based on the motion candidate list; Refining the motion information using a codec tool; performing conversion between the first video block and a bitstream of a video including the first video block by applying refined motion information; and updating a motion candidate table in the one or more motion candidate tables, wherein whether to update the motion candidate table using the refined motion information or using the motion information without refinement is based on whether a codec tool selected for encoding and decoding the first video block is a first refinement tool or a merge tool with motion vector difference; wherein, in response to the codec tool being the first refinement tool, the motion candidate table is updated by applying the motion information without refinement, and wherein, in the first refinement tool, the refined motion information is derived based on calculation of a sum of absolute differences (SAD) of reference samples corresponding to the first video block; In which, in response to the encoding and decoding tool being the Merge tool with motion vector difference, the motion candidate table is updated by applying the refined motion information.

2. The method according to claim 1, wherein: The first video block is encoded and decoded in Merge mode.

3. The method according to claim 1, wherein: Based on the coding mode of the current block belonging to at least one specific mode, it is determined that the motion candidate table is to be updated.

4. The method according to claim 3, wherein: The at least one specific mode includes at least one of a motion vector prediction mode or a Merge mode.

5. The method according to claim 3, wherein: The at least one specific mode excludes at least one of a sub-block Merge mode or a triangle Merge mode, wherein the motion candidate table is updated without using motion information derived from a block using the sub-block Merge mode or the triangle Merge mode.

6. The method according to claim 1, wherein: The motion candidate table is a historical motion vector prediction HMVP table.

7. The method according to claim 1, wherein: The updates include: The motion candidate table is updated using a pruning operation, wherein the pruning operation includes comparing a new motion candidate to be added to the motion candidate table with at least one motion candidate in the motion candidate table to determine whether to delete at least one motion candidate from the motion candidate table.

8. The method according to claim 1, wherein: If the codec tool is an illumination compensation tool or an overlapped block motion compensation tool, the motion candidate table is updated using the unrefined motion information.

9. The method according to claim 1, wherein: If a new candidate associated with motion information derived from the motion candidate list is a motion candidate added from the motion candidate table to the motion candidate list, the motion candidate list is reordered without a pruning operation.

10. The method according to claim 1, wherein: If a new candidate associated with the motion information derived from the motion candidate list is a motion candidate related to a spatially neighboring block in a current picture including the first video block or a temporal block in a picture different from the current picture, the motion candidate list is updated with a pruning operation.

11. The method according to claim 1, wherein: The method further comprises: performing a motion candidate list derivation process to derive a motion candidate list for a second video block encoded and decoded in a motion vector prediction mode; deriving motion information for the second video block based on the motion candidate list derived for the second video block; performing conversion between the second video block and a bitstream of a video including the second video block based on the motion information derived for the second video block; and The motion candidate table is updated using the motion information derived for the second video block without refinement.

12. The method according to claim 1, wherein: Performing the conversion includes generating a bitstream based on the first video block.

13. The method according to claim 1, wherein: Performing the conversion includes generating the first video block from a bitstream.

14. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: Maintain one or more motion candidate tables, where Each motion candidate table includes one or more motion candidates, at least one of the one or more motion candidates is derived from at least one previous video block, and an arrangement of the motion candidates in the motion candidate table is based on an order in which the motion candidates are added to the motion candidate table; performing a motion candidate list derivation process to derive a motion candidate list for the first video block; deriving motion information of the first video block based on the motion candidate list; Refining the motion information using a codec tool; performing conversion between the first video block and a bitstream of a video including the first video block by applying refined motion information; as well as updating a motion candidate table in the one or more motion candidate tables, wherein whether to update the motion candidate table using the refined motion information or using the motion information without refinement is based on whether a codec tool selected for encoding and decoding the first video block is a first refinement tool or a merge tool with motion vector difference; wherein, in response to the codec tool being the first refinement tool, the motion candidate table is updated by applying the motion information without refinement, and wherein, in the first refinement tool, the refined motion information is derived based on calculation of a sum of absolute differences (SAD) of reference samples corresponding to the first video block; In which, in response to the encoding and decoding tool being the Merge tool with motion vector difference, the motion candidate table is updated by applying the refined motion information.

15. A non-transitory computer-readable storage medium storing instructions that cause a processor to: Maintain one or more motion candidate tables, where Each motion candidate table includes one or more motion candidates, at least one of the one or more motion candidates is derived from at least one previous video block, and an arrangement of the motion candidates in the motion candidate table is based on an order in which the motion candidates are added to the motion candidate table; performing a motion candidate list derivation process to derive a motion candidate list for the first video block; deriving motion information of the first video block based on the motion candidate list; Refining the motion information using a codec tool; performing conversion between the first video block and a bitstream of a video including the first video block by applying refined motion information; as well as updating a motion candidate table in the one or more motion candidate tables, wherein whether to update the motion candidate table using the refined motion information or using the motion information without refinement is based on whether a codec tool selected for encoding and decoding the first video block is a first refinement tool or a merge tool with motion vector difference; wherein, in response to the codec tool being the first refinement tool, the motion candidate table is updated by applying the motion information without refinement, and wherein, in the first refinement tool, the refined motion information is derived based on calculation of a sum of absolute differences (SAD) of reference samples corresponding to the first video block; In which, in response to the encoding and decoding tool being the Merge tool with motion vector difference, the motion candidate table is updated by applying the refined motion information.

16. A method for storing a bit stream of a video, comprising: maintaining one or more motion candidate tables, wherein each motion candidate table includes one or more motion candidates, at least one of the one or more motion candidates is derived from at least one previous video block, and an arrangement of the motion candidates in the motion candidate table is based on an order in which the motion candidates are added to the motion candidate table; performing a motion candidate list derivation process to derive a motion candidate list for the first video block; deriving motion information of the first video block based on the motion candidate list; Refining the motion information using a codec tool; generating the bitstream by applying the refined motion information; storing the generated bit stream in a non-transitory computer-readable recording medium; and updating a motion candidate table in the one or more motion candidate tables, wherein whether to update the motion candidate table using the refined motion information or using the motion information without refinement is based on whether a codec tool selected for encoding and decoding the first video block is a first refinement tool or a merge tool with motion vector difference; wherein, in response to the codec tool being the first refinement tool, the motion candidate table is updated by applying the motion information without refinement, and wherein, in the first refinement tool, the refined motion information is derived based on calculation of a sum of absolute differences (SAD) of reference samples corresponding to the first video block; In which, in response to the encoding and decoding tool being the Merge tool with motion vector difference, the motion candidate table is updated by applying the refined motion information.

17. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein: The instructions, when executed by a processor, cause the processor to implement the method according to any one of claims 2 to 13 and 16.

18. A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 2 to 13, 16.

19. A computer program product stored on a non-transitory computer readable medium, the computer program product comprising program code for executing the method of any one of claims 1 to 13, 16.

Images