The order in which motion candidates are checked in the LUT

Abstract

A method for video processing is provided for checking motion candidates in a LUT, comprising: maintaining tables, wherein each table includes a set of motion candidates, and each motion candidate is associated with corresponding motion information; and performing conversion between a first video block and a bitstream representation of a video including the first video block based on at least one of the tables, the performing comprising checking the motion candidates in the tables in an order defined in a rule until a checking termination criterion is met.

Description

[0001] This application is a divisional application of the invention patent application with the application date of July 1, 2019, application number 201910586791.3, and invention name “Checking order of motion candidates in LUT”. Technical Field

[0002] This patent document relates to video encoding and decoding technologies, devices, and systems. Background Art

[0003] Despite advances in video compression, digital video still accounts for the largest use of bandwidth on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth required for digital video usage is expected to continue to grow. Summary of the Invention

[0004] This document discloses methods, systems, and devices for encoding and decoding digital video.

[0005] In one example aspect, a video processing method is disclosed. The method includes maintaining tables, wherein each table includes a set of motion candidates and each motion candidate is associated with corresponding motion information; and performing conversion between a first video block and a bitstream representation of a video including the first video block based at least on the one or more tables, the performing comprising checking the motion candidates in the tables in an order defined in a rule until a checking termination criterion is satisfied.

[0006] In yet another example aspect, a video encoder device implementing the video encoding method described herein is disclosed.

[0007] In another representative aspect, the various techniques described herein can be embodied as a computer program product stored on a non-transitory computer-readable medium. The computer program product includes program code for executing the methods described herein.

[0008] In yet another representative aspect, a video decoder device may implement a method as described herein.

[0009] The details of one or more implementations are set forth in the accompanying drawings, the accompanying figures, and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram illustrating an example of a video encoder implementation

[0011] Figure 2 The macroblock partitioning in the H.264 video coding standard is shown.

[0012] Figure 3An example of dividing a coding block into prediction blocks is shown.

[0013] Figure 4 An example embodiment of subdividing a CTB (coding tree block) into CBs and transform blocks (TBs) is shown. Solid lines indicate CB boundaries and dashed lines indicate TB boundaries, including an example CTB with its partitioning and the corresponding quadtree.

[0014] Figure 5 An example of a Quad Tree Binary Tree (QTBT) structure for segmenting video data is shown.

[0015] Figure 6 An example of video block segmentation is shown.

[0016] Figure 7 An example of quadtree partitioning is shown.

[0017] Figure 8 An example of tree type signaling is shown.

[0018] Figure 9 An example of the derivation process for Merge candidate list construction is shown.

[0019] Figure 10 Example locations of spatial merge candidates are shown.

[0020] Figure 11 An example of candidate pairs for redundancy checking considering spatial Merge candidates is shown.

[0021] Figure 12 Examples of the location of the second PU for Nx2N and 2NxN partitions are shown.

[0022] Figure 13 Motion vector scaling of temporal merge candidates is shown.

[0023] Figure 14 A picture showing candidate positions of temporal merge candidates and their co-location.

[0024] Figure 15 An example of a combined bi-predictive Merge candidate is shown.

[0025] Figure 16 An example of a process of deriving motion vector prediction candidates is shown.

[0026] Figure 17 An example of motion vector scaling of spatial motion vector candidates is shown.

[0027] Figure 18 An example Alternative Temporal Motion Vector Prediction (ATMVP) for motion prediction of a CU (coding unit) is shown.

[0028] Figure 19 An example of identification of source blocks and source pictures is pictorially depicted.

[0029] Figure 20 An example of one CU having four subblocks and neighboring blocks is shown.

[0030] Figure 21 An example of bilateral matching is shown.

[0031] Figure 22 An example of template matching is shown.

[0032] Figure 23 Depicts an example of unilateral motion estimation (ME) in frame rate up conversion (FRUC).

[0033] Figure 24 An example of DMVR based on bilateral template matching is shown.

[0034] Figure 25 An example of spatial neighboring blocks used to derive spatial Merge candidates is shown.

[0035] Figure 26 Depicted is an example of how the selection of a representative location of a lookup table is updated.

[0036] Figure 27A and Figure 27B An example of updating a lookup table with a new set of motion information is shown.

[0037] Figure 28 is a block diagram of an example of a hardware platform for implementing the visual media decoding or visual media encoding techniques described in this document.

[0038] Figure 29 is a flow chart of an example method of video bitstream processing.

[0039] Figure 30 An example of a decoding flow chart with the proposed HMVP method is shown.

[0040] Figure 31 An example of updating a table using the proposed HMVP method is shown.

[0041] Figure 32A and Figure 32B An example of a LUT (Look Up Table) updating method based on redundancy removal (one redundant motion candidate is removed) is shown.

[0042] Figure 33A-Figure 33B An example of a table updating method based on redundancy removal (a plurality of redundant motion candidates are removed) is shown. DETAILED DESCRIPTION

[0043] To improve the compression ratio of video, researchers are continually searching for new techniques by which to encode video.

[0044] 1. Introduction

[0045] This document relates to video coding techniques. Specifically, it relates to motion information coding in video coding (such as Merge mode and AMVP mode). This document can be applied to existing video coding standards such as HEVC, or to standards to be finalized (Versatile Video Coding). This document can also be applied to future video coding standards or video codecs.

[0046] Brief Discussion

[0047] Video coding standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T produced H.261 and H.263, ISO / IEC produced MPEG-1 and MPEG-4 Video (Visual), and the two organizations jointly produced the H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Coding (AVC) and H.265 / HEVC standards. Since H.262, video coding standards have been based on a hybrid video coding structure that utilizes temporal prediction plus transform coding. Figure 1 An example of a typical HEVC encoder framework is depicted in .

[0048] 2.1 Segmentation Structure

[0049] 2.1.1 Segmentation Tree Structure in H.264 / AVC

[0050] The core of the coding layer in previous standards is the macroblock, which contains a 16×16 block of luminance samples and, in the common case of 4:2:0 color sampling, two corresponding 8×8 blocks of chrominance samples.

[0051] Intra-coded blocks use spatial prediction to exploit spatial correlation among pixels. Two partitions are defined: 16x16 and 4x4.

[0052] Inter-coded blocks use temporal prediction instead of spatial prediction by estimating the motion within the picture. Can be partitioned for 16x16 macroblocks or any of their sub-macroblocks: 16x8, 8x16, 8x8, 8x4, 4x8, 4x4 (see Figure 2 ) estimates motion independently. Only one motion vector (MV) is allowed per sub-macroblock partition.

[0053] 2.1.2 Segmentation Tree Structure in HEVC

[0054] In HEVC, a CTU is divided into CUs using a quadtree structure denoted as a coding tree to accommodate various local characteristics. The decision whether to use inter-picture (temporal) or intra-picture (spatial) prediction to encode a picture area is made at the CU level. Depending on the PU partition type, each CU can be further divided into one, two, or four PUs. Within a PU, the same prediction process is applied, and relevant information is sent to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU partition type, the CU can be partitioned into transform units (TUs) according to another quadtree structure similar to the coding tree of the CU. One of the key features of the HEVC structure is that it has multiple partitioning concepts including CU, PU, and TU.

[0055] In the following, various features involved in hybrid video coding using HEVC are highlighted as follows.

[0056] 1) Coding Tree Unit and Coding Tree Block (CTB) Structure: A similar structure in HEVC is the Coding Tree Unit (CTU), which has a size chosen by the encoder and can be larger than a traditional macroblock. A CTU consists of a luma CTB and corresponding chroma CTBs and syntax elements. The size L×L of the luma CTB can be chosen to be L=16, 32, or 64 samples, with larger sizes typically enabling better compression. HEVC then supports splitting the CTB into smaller blocks using a tree structure and quadtree-like signaling.

[0057] 2) Coding Unit (CU) and Coding Block (CB): The quadtree syntax of a CTU specifies the size and position of its luma and chroma CBs. The root of the quadtree is associated with the CTU. Therefore, the size of the luma CTB is the maximum supported size of the luma CB. The partitioning of the CTU into luma and chroma CBs is signaled jointly. One luma CB and typically two chroma CBs together with the associated syntax form a coding unit (CU). A CTB may contain only one CU or may be partitioned to form multiple CUs, and each CU has an associated partitioning into prediction units (PUs) and a tree of transform units (TUs).

[0058] 3) Prediction Unit and Prediction Block (PB): The decision whether to use inter or intra prediction to encode a picture region is made at the CU level. The PU partitioning structure has its roots at the CU level. Depending on the basic prediction type decision, the luma and chroma CBs can then be further split into sizes and predicted from luma and chroma prediction blocks (PBs). HEVC supports variable PB sizes from 64×64 down to 4×4 samples. Figure 3 An example of allowed PBs for an MxM CU is shown.

[0059] 4) TU and transform block: The prediction residual is coded using block transform. The TU tree structure has its root at the CU level. The luma CB residual can be the same as the luma transform block (TB) or can be further divided into smaller luma TBs. The same applies to chroma TBs. Integer basis functions similar to those of the discrete cosine transform (DCT) define square TB sizes 4×4, 8×8, 16×16, and 32×32. For the 4×4 transform of the luma intra prediction residual, an integer transform derived from the form of the discrete sine transform (DST) is specified instead.

[0060] Figure 4 An example of subdividing a CTB into CBs and transform blocks (TBs) is shown. Solid lines indicate CB boundaries and dashed lines indicate TB boundaries. (a) CTB and its partitions. (b) Corresponding quadtree.

[0061] 2.1.2.1 Tree-structured partitioning into transform blocks and units

[0062] For residual coding, the CB can be recursively partitioned into transform blocks (TBs). The partitioning is signaled by the residual quadtree. Only square CB and TB partitioning is specified, where blocks can be recursively divided into quadrants, such as Figure 4As shown in . For a given luma CB of size M×M, a flag signals whether it is divided into four blocks of size M / 2×M / 2. If further division is possible, as signaled by the maximum depth of the residual quadtree indicated in the SPS, each quadrant is assigned a flag indicating whether it is divided into four quadrants. The leaf node blocks arising from the residual quadtree are transform blocks that are further processed by transform coding. The encoder indicates the maximum and minimum luma TB sizes that it will use. When the CB size is larger than the maximum TB size, the division is implicit. When the division would cause the luma TB size to be smaller than the indicated minimum, no division is implicit. Except when the luma TB size is 4×4, the chroma TB size is half the luma TB size in each dimension, in which case a single 4×4 chroma TB is used for the area covered by four 4×4 luma TBs. In the case of an intra-picture predicted CU, the decoded samples of the nearest neighboring TB (inside or outside the CB) are used as reference data for intra prediction.

[0063] In contrast to previous standards, the HEVC design allows a TB to span multiple PBs of a CU for inter-picture prediction to maximize the potential coding efficiency benefits of quadtree-structured TB partitioning.

[0064] 2.1.2.2 Parent and Child Nodes

[0065] The CTB is divided according to a quadtree structure, where the nodes of the quadtree structure are coding units. The multiple nodes in the quadtree structure include leaf nodes and non-leaf nodes. A leaf node does not have child nodes in the tree structure (i.e., the leaf node is not further divided). The non-leaf nodes include the root node of the tree structure. The root node corresponds to an initial video block of video data (e.g., a CTB). For each non-root node in the multiple nodes, the non-root node corresponds to a video block that is a child block of the video block of the parent node in the tree structure corresponding to the non-root node. Each non-leaf node in the multiple non-leaf nodes has one or more child nodes in the tree structure.

[0066] 2.1.3 Quadtree plus binary tree block structure with larger CTU in JEM

[0067] To explore future video coding technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, many new methods have been adopted by JVET and put into reference software called the Joint Exploration Model (JEM).

[0068] 2.1.3.1QTBT Block Partition Structure

[0069] Different from HEVC, the QTBT structure removes the concept of multiple partition types, that is, it removes the separation of CU, PU and TU concepts and supports more flexibility for CU partition shape. In the QTBT block structure, CU can have a square or rectangular shape. Figure 5 As shown in , the coding tree unit (CTU) is first partitioned by a quadtree structure. The quadtree leaf nodes are further partitioned by a binary tree structure. There are two types of partitioning in the binary tree partitioning, symmetric horizontal partitioning and symmetric vertical partitioning. The binary tree leaf nodes are called coding units (CUs), and this partitioning is used for prediction and transform processing without any further partitioning. This means that CU, PU and TU have the same block size in the QTBT coding block structure. In JEM, a CU sometimes consists of coding blocks (CBs) of different color components, for example, in the case of P and B slices of a 4:2:0 chroma format, one CU contains one luminance CB and two chroma CBs, and sometimes consists of CBs of a single component, for example, in the case of an I slice, one CU contains only one luminance CB or only two chroma CBs.

[0070] The following parameters are defined for the QTBT segmentation scheme.

[0071] -CTU size: the root node size of the quadtree, the same concept as in HEVC

[0072] -MinQTSize: Minimum allowed quad leaf node size

[0073] -MaxBTSize: Maximum allowed binary tree root node size

[0074] -MaxBTDepth: Maximum allowed binary tree depth

[0075] -MinBTSize: minimum allowed binary tree leaf node size

[0076] In one example of a QTBT partitioning structure, the CTU size is set to 128×128 luma samples with two corresponding 64×64 chroma sample blocks, MinQTSize is set to 16×16, MaxBTSize is set to 64×64, MinBTSize (for both width and height) is set to 4×4, and MaxBTDepth is set to 4. Quadtree partitioning is first applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can have sizes from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the leaf quadtree node is 128×128, it will not be further partitioned by the binary tree because the size exceeds MaxBTSize (i.e., 64×64). Otherwise, the leaf quadtree node can be further partitioned by the binary tree. Therefore, the quadtree leaf node is also the root node of the binary tree and has a binary tree depth of 0. When the binary tree depth reaches MaxBTDepth (i.e., 4), no further partitioning is considered. When a binary tree node has a width equal to MinBTSize (i.e., 4), no further horizontal splitting is considered. Similarly, when a binary tree node has a height equal to MinBTSize, no further vertical splitting is considered. Leaf nodes of the binary tree are further processed by prediction and transform processing without any further splitting. In JEM, the maximum CTU size is 256×256 luma samples.

[0077] Figure 5 (left) shows an example of partitioning by using QTBT blocks, and Figure 5 (Right) shows the corresponding tree representation. Solid lines indicate quadtree partitioning and dashed lines indicate binary tree partitioning. In each partition (i.e., non-leaf) node of the binary tree, a flag is signaled to indicate which partition type is used (i.e., horizontal or vertical), where 0 indicates horizontal partitioning and 1 indicates vertical partitioning. For quadtree partitioning, there is no need to indicate the partition type because quadtree partitioning always partitions the block horizontally and vertically to produce 4 sub-blocks of the same size.

[0078] In addition, the QTBT scheme supports the ability to have separate QTBT structures for luma and chroma. Currently, for P and B slices, the luma and chroma CTBs in one CTU share the same QTBT structure. However, for I slices, the luma CTBs are split into CUs using the QTBT structure, and the chroma CTBs are split into chroma CUs using another QTBT structure. This means that a CU in an I slice consists of coding blocks for the luma component or coding blocks for two chroma components, and a CU in a P or B slice consists of coding blocks for all three color components.

[0079] In HEVC, inter-frame prediction for small blocks is restricted to reduce memory access for motion compensation, so that bidirectional prediction is not supported for 4×8 and 8×4 blocks, and inter-frame prediction is not supported for 4×4 blocks. In JEM's QTBT, these restrictions are removed.

[0080] 2.1.4 Ternary Tree for VVC

[0081] In some embodiments, tree types other than quadtree and binary tree are supported. In an embodiment, two more ternary tree (TT) splits are introduced, namely horizontal and vertical center-side ternary trees, such as Figure 6 The middle ones are marked with (d) and (e).

[0082] Figure 6 Shown: (a) quadtree partition (b) vertical binary tree partition (c) horizontal binary tree partition (d) vertical center-side ternary tree partition (e) horizontal center-side ternary tree partition.

[0083] In some embodiments, there are two levels of trees: a region tree (quadtree) and a prediction tree (binary or ternary tree). The CTU is first partitioned by the region tree (RT). The RT leaf can be further partitioned by the prediction tree (PT). The PT leaf can also be further partitioned by the PT until the maximum PT depth is reached. The PT leaf is the basic coding unit. For convenience, it is still referred to as a CU. The CU cannot be further partitioned. Both prediction and transform are applied to the CU in the same manner as JEM. The entire partitioning structure is referred to as a "multi-type tree."

[0084] 2.1.5 Example of Segmentation Structure

[0085] The tree structure used in this response, called Multi-Tree Type (MTT), is a generalization of QTBT. In QTBT, Figure 5 As shown in , the coding tree unit (CTU) is first divided into a quadtree structure. The quadtree leaf nodes are further divided into a binary tree structure.

[0086] The basic structure of MTT consists of two types of tree nodes: region tree (RT) and prediction tree (PT), which supports nine types of segmentation, such as Figure 7 As shown in .

[0087] Figure 7 Shown: (a) quadtree partition (b) vertical binary tree partition (c) horizontal binary tree partition (d) vertical ternary tree partition (e) horizontal ternary tree partition (f) horizontal upper asymmetric binary tree partition (g) horizontal lower asymmetric binary tree partition (h) vertical left asymmetric binary tree partition (i) vertical right asymmetric binary tree partition.

[0088] The region tree can recursively divide the CTU into square blocks, down to the region tree leaf nodes of size 4x4. At each node in the region tree, a prediction tree can be formed from one of three tree types: binary tree (BT), ternary tree (TT), and asymmetric binary tree (ABT). In PT partitioning, it is forbidden to have quadtree splits in the branches of the prediction tree. As in JEM, the luma tree and chroma tree are separated in the I slice. The signaling method for RT and PT is in Figure 8 Shown in.

[0089] 2.2 Inter-frame prediction in HEVC / H.265

[0090] Each inter-predicted PU has motion parameters for one or two reference picture lists. The motion parameters include motion vectors and reference picture indices. The use of one of the two reference picture lists can also be signaled using inter_pred_idc. The motion vector can be explicitly encoded as a delta relative to the predictor, a coding mode known as AMVP mode.

[0091] When a CU is encoded using skip mode, one PU is associated with the CU and has no significant residual coefficients, no coded motion vector increments, or reference picture indices. Merge mode is specified, whereby the motion parameters for the current PU are obtained from neighboring PUs, including spatial and temporal candidates. Merge mode can be applied to any inter-predicted PU, not only for skip mode. An alternative to Merge mode is explicit transmission of motion parameters, where for each PU, the motion vector, the corresponding reference picture index for each reference picture list, and the reference picture list used are explicitly signaled.

[0092] When signaling indicates that one of the two reference picture lists is to be used, a PU is generated from a block of samples. This is called "unidirectional prediction." Unidirectional prediction can be used for both P slices and B slices.

[0093] When signaling indicates that both of the reference picture lists are to be used, a PU is generated from both blocks of samples. This is called "bi-prediction." Bi-prediction is only available for B slices.

[0094] The following text provides details about the inter prediction modes specified in HEVC. The description will start with the Merge mode.

[0095] 2.2.1Merge Mode

[0096] 2.2.1.1 Derivation of Merge Mode Candidates

[0097] When using Merge mode to predict a PU, the index pointing to the entry in the Merge candidate list is parsed from the bitstream and used to retrieve motion information. The construction of this list is specified in the HEVC standard and can be summarized according to the following sequence of steps:

[0098] Step 1: Initial candidate derivation

[0099] oStep 1.1: Spatial Candidate Derivation

[0100] oStep 1.2: Redundancy check of spatial candidates

[0101] oStep 1.3: Temporal Candidate Derivation

[0102] Step 2: Additional candidate insertion

[0103] oStep 2.1: Creation of Bidirectional Prediction Candidates

[0104] oStep 2.2: Insertion of zero-motion candidates

[0105] These steps are also Figure 9 Schematically depicted in . For spatial Merge candidate derivation, a maximum of four Merge candidates are selected among the candidates located in five different positions. For temporal Merge candidate derivation, a maximum of one Merge candidate is selected among two candidates. Because a constant number of candidates is assumed at the decoder for each PU, additional candidates are generated when the number of candidates does not reach the maximum number of Merge candidates (Maximum Number of MergeCandidate, MaxNumMergeCand) signaled in the slice header. Because the number of candidates is constant, the index of the best Merge candidate is encoded using truncated unary binarization (TU). If the size of the CU is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

[0106] Hereinafter, operations associated with the aforementioned steps are described in detail.

[0107] 2.2.1.2 Spatial Candidate Derivation

[0108] In the derivation of spatial Merge candidates, when locating Figure 10A maximum of four Merge candidates are selected from the candidates in the positions depicted in . The order of derivation is A1, B1, B0, A0 and B2. Position B2 is considered only if any PU at position A1, B1, B0, A0 is unavailable (for example, because it belongs to another strip or slice) or is intra-coded. After adding the candidate at position A1, the addition of the remaining candidates is subject to a redundancy check that ensures that candidates with the same motion information are excluded from the list, thereby improving coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. On the contrary, only candidates are considered if the corresponding candidates for the redundancy check do not have the same motion information. Figure 11 The pairs linked by arrows in [ ] are added to the list only. Another source of copied motion information is a "second PU" associated with a partition other than 2Nx2N. As an example, Figure 12 The second PU is depicted for the N×2N and 2N×N cases, respectively. When the current PU is partitioned into N×2N, the candidate at position A1 is not considered for list construction. In fact, adding this candidate would result in two prediction units with the same motion information, which is redundant for having only one PU in the coding unit. Similarly, position B1 is not considered when the current PU is partitioned into 2N×N.

[0109] 2.2.1.3 Time Candidate Derivation

[0110] In this step, only one candidate is added to the list. Specifically, in the derivation of this temporal Merge candidate, the scaled motion vector is derived based on the co-located PU belonging to the picture with the smallest POC difference with the current picture in the given reference picture list. The reference picture list to be used for the derivation of the co-located PU is explicitly signaled in the slice header. Figure 13 The scaled motion vector for the temporal merge candidate is obtained, as shown by the dashed line, by scaling the motion vector of the collocated PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal merge candidate is set equal to zero. The actual implementation of the scaling process is described in the HEVC specification. For B slices, two motion vectors are obtained, one for reference picture list 0 and the other for reference picture list 1, and the two motion vectors are combined to obtain a bi-predictive merge candidate. An illustration of motion vector scaling for a temporal merge candidate.

[0111] like Figure 14As depicted in , in a co-located PU (Y) belonging to a reference frame, the position for the temporal candidate is selected between candidates C0 and C1. If the PU at position C0 is unavailable, intra-coded, or outside the current CTU, position C1 is used. Otherwise, position C0 is used in the derivation of the temporal merge candidate.

[0112] 2.2.1.4 Additional Candidate Insertion

[0113] In addition to the spatial-temporal Merge candidate, there are two additional types of Merge candidates: combined bi-predictive Merge candidate and zero Merge candidate. Combined bi-predictive Merge candidate is generated by utilizing the spatial-temporal Merge candidate. Combined bi-predictive Merge candidate is only used for B slices. Combined bi-predictive candidate is generated by combining the first reference picture list motion parameters of the initial candidate with the second reference picture list motion parameters of another. If these two tuples provide different motion hypotheses, they will form a new bi-predictive candidate. As an example, Figure 15 Depicted is the case when two candidates in the original list (on the left) are used to create combined bi-predictive Merge candidates that are added to the final list (on the right). There are many rules about the combinations that are considered to generate these additional Merge candidates.

[0114] Zero motion candidates are inserted to fill the remaining entries in the Merge candidate list and thus hit the MaxNumMergeCand capacity. These candidates have zero spatial displacement and reference picture index, which starts at zero and increases each time a new zero motion candidate is added to the list. The number of reference frames used by these candidates is one and two for unidirectional prediction and bidirectional prediction, respectively. Finally, no redundancy check is performed on these candidates.

[0115] 2.2.1.5 Motion Estimation Regions for Parallel Processing

[0116] To speed up the encoding process, motion estimation can be performed in parallel, thereby deriving motion vectors for all prediction units within a given region simultaneously. Deriving Merge candidates from spatial neighborhoods may interfere with parallel processing because a prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is completed. To alleviate the trade-off between coding efficiency and processing delay, HEVC defines a motion estimation region (MER), the size of which is signaled in the picture parameter set using the "log2_parallel_merge_level_minus2" syntax element. When MER is defined, Merge candidates falling in the same region are marked as unavailable and are therefore not considered in list construction.

[0117] 7.3.2.3 Picture Parameter Set RBSP Syntax

[0118] 7.3.2.3.1 General Picture Parameter Set RBSP Syntax

[0119]

[0120] log2_parallel_merge_level_minus2 plus 2 specifies the value of the variable Log2ParMrgLevel, which is used in the derivation of luma motion vectors for Merge mode as specified in 8.5.3.2.2 and in the derivation of spatial Merge candidates as specified in 8.5.3.2.3. The value of log2_parallel_merge_level_minus2 shall be in the range of 0 to CtbLog2SizeY-2, inclusive.

[0121] The variable Log2ParMrgLevel is derived as follows:

[0122] Log2ParMrgLevel=log2_parallel_merge_level_minus2+2 (7-37)

[0123] NOTE 3 - The value of Log2ParMrgLevel indicates the built-in capability of parallel derivation of Merge candidate lists. For example, when Log2ParMrgLevel is equal to 6, the Merge candidate lists of all prediction units (PUs) and coding units (CUs) contained in a 64x64 block can be derived in parallel.

[0124] 2.2.2 Motion Vector Prediction in AMVP Mode

[0125] Motion vector prediction exploits the spatial-temporal correlation of motion vectors with neighboring PUs, which is used for explicit transmission of motion parameters. It builds a motion vector candidate list by first checking the availability of PU positions that are temporally adjacent to the left and above, removing redundant candidates, and adding zero vectors to make the candidate list a constant length. The encoder can then select the best prediction value from the candidate list and transmit the corresponding index indicating the selected candidate. Similar to Merge index signaling, the index of the best motion vector candidate is encoded using truncated unary. The maximum value to be encoded in this case is 2 (e.g., Figures 2 to 8 ). In the following sections, details on the derivation process of motion vector prediction candidates are provided.

[0126] 2.2.2.1 Derivation of Motion Vector Prediction Candidates

[0127] Figure 16 The derivation process of motion vector prediction candidates is summarized.

[0128] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For spatial motion vector candidate derivation, as Figure 11 As depicted in , two motion vector candidates are ultimately derived based on the motion vector of each PU located in five different positions.

[0129] For temporal motion vector candidate derivation, a motion vector candidate is selected from two candidates derived based on two different co-located positions. After making the first list of spatio-temporal candidates, duplicate motion vector candidates are removed from the list. If the number of potential candidates is greater than two, motion vector candidates whose reference picture index within the associated reference picture list is greater than 1 are removed from the list. If the number of spatio-temporal motion vector candidates is less than two, an additional zero motion vector candidate is added to the list.

[0130] 2.2.2.2 Spatial Motion Vector Candidates

[0131] In the derivation of spatial motion vector candidates, a maximum of two candidates are considered among five potential candidates, which are located at positions such as Figure 11 The PU derivation in the positions depicted in the figure is identical to those of the motion merge. The order of derivation for the left side of the current PU is defined as A0, A1, and scaled A0, scaled A1. The order of derivation for the top side of the current PU is defined as B0, B1, B2, scaled B0, scaled B1, scaled B2. For each side, there are therefore four possible cases that can be used as motion vector candidates, two of which do not require the use of spatial scaling, and two of which use spatial scaling. The four different cases are summarized below.

[0132] No spatial scaling

[0133] -(1) Same reference picture list and same reference picture index (same POC)

[0134] -(2) Different reference picture lists, but same reference picture (same POC)

[0135] Spatial scaling

[0136] -(3) Same reference picture list, but different reference pictures (different POC)

[0137] -(4) Different reference picture lists and different reference pictures (different POC)

[0138] First, the no spatial scaling case is checked, followed by the spatial scaling case. When the POC differs between the reference picture of the neighboring PU and the reference picture of the current PU, spatial scaling is considered, regardless of the reference picture list. If all PUs of the left candidate are unavailable or intra-coded, scaling of the upper motion vector is allowed to facilitate the parallel derivation of the left and upper MV candidates. Otherwise, spatial scaling is not allowed for the upper motion vector.

[0139] During spatial scaling, Figure 17 As depicted in , the motion vectors of neighboring PUs are scaled in a similar manner as for time scaling. The main difference is that the reference picture list and the index of the current PU are given as input; the actual scaling process is the same as that for time scaling.

[0140] 2.2.2.3 Temporal Motion Vector Candidates

[0141] Except for the reference picture index derivation, all the processes for the derivation of temporal merge candidates are the same as those for the derivation of spatial motion vector candidates (see e.g. Figure 6 ). The reference picture index is signaled to the decoder.

[0142] 2.2.2.4 Signaling Notification of AMVP Information

[0143] For the AMVP mode, four parts may be signaled in the bitstream, namely, prediction direction, reference index, MVD (Motion Vector Difference) and mv prediction value candidate index.

[0144] Syntax table:

[0145]

[0146] 7.3.8.9 Motion Vector Difference Syntax

[0147]

[0148] 2.3 New Inter-frame Prediction Method in JEM (Joint Exploration Model)

[0149] 2.3.1 Motion Vector Prediction Based on Sub-CU

[0150] In JEM with QTBT, each CU can have at most one motion parameter set for each prediction direction. Two sub-CU level motion vector prediction methods are considered in the encoder by dividing the large CU into sub-CUs and deriving motion information for all sub-CUs of the large CU. The optional temporal motion vector prediction (ATMVP) method allows each CU to extract multiple sets of motion information from multiple blocks smaller than the current CU in the collocated reference picture. In the Spatial-Temporal Motion Vector Prediction (STMVP) method, the motion vector of the sub-CU is recursively derived by using the temporal motion vector predictor and the spatial neighboring motion vectors.

[0151] In order to preserve a more accurate motion field for sub-CU motion prediction, motion compression on reference frames is currently disabled.

[0152] 2.3.1.1 Optional Temporal Motion Vector Prediction

[0153] In the optional temporal motion vector prediction (ATMVP) method, the motion vector is modified by extracting multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU. Figure 18 As shown in , a sub-CU is a square N×N block (N is set to 4 by default).

[0154] ATMVP predicts the motion vectors of sub-CUs within a CU in two steps. The first step is to identify the corresponding blocks in the reference picture using the so-called temporal vector. The reference picture is called the motion source picture. The second step is to divide the current CU into sub-CUs and obtain the motion vectors and reference indexes of each sub-CU from the blocks corresponding to each sub-CU, such as Figure 18 shown.

[0155] In the first step, the reference picture and the corresponding block are determined by the motion information of the spatial neighboring blocks of the current CU. In order to avoid repeated scanning of neighboring blocks, the first Merge candidate in the Merge candidate list of the d CU is used. The first available motion vector and its associated reference index are set to the time vector and the index to the motion source picture. In this way, in ATMVP, the corresponding block can be identified more accurately than in TMVP, where the corresponding block (sometimes referred to as the co-located block) is always in the lower right or center position relative to the current CU. In one example, if the first Merge candidate comes from the left neighboring block (i.e., Figure 19 A1) in , the associated MV and reference picture are used to identify the source block and source picture.

[0156] Figure 19 An example of identification of source blocks and source pictures is shown.

[0157] In the second step, the corresponding block of the sub-CU is identified by the time vector in the motion source picture by adding the time vector to the coordinates of the current CU. For each sub-CU, the motion information of its corresponding block (the minimum motion grid covering the center sample) is used to derive the motion information of the sub-CU. After the motion information of the corresponding N×N block is identified, it is converted into the motion vector and reference index of the current sub-CU in the same way as the TMVP of HEVC, where motion scaling and other processes apply. For example, the decoder checks whether the low latency condition is met (i.e., the POC of all reference pictures of the current picture is less than the POC of the current picture) and may use the motion vector MV x (corresponding to the motion vector of reference picture list X) to predict the motion vector MV of each sub-CU y (where X is equal to 0 or 1, and Y is equal to 1-X).

[0158] 2.3.1.2 Space-time motion vector prediction

[0159] In this method, the motion vector of a sub-CU is recursively derived following the raster scan order. Figure 20 This concept is illustrated in . Let us consider an 8x8 CU containing four 4x4 sub-CUs A, B, C, and D. The neighboring 4x4 blocks in the current frame are labeled a, b, c, and d.

[0160] The motion derivation of sub-CU A starts by identifying its two spatial neighbors. The first neighbor is the N×N block (block c) on the upper side of the sub-CU A. If block c is unavailable or intra-coded, check the other N×N blocks on the upper side of sub-CU A (from left to right, starting from block c). The second neighbor is the block (block b) on the left side of sub-CU A. If block b is unavailable or intra-coded, check the other blocks on the left side of sub-CU A (from top to bottom, starting from block b). The motion information obtained from the adjacent blocks of each list is scaled to the first reference frame of the given list. Next, the temporal motion vector predictor (TMVP) of sub-block A is derived by following the same process as the TMVP derivation specified in HEVC. The motion information of the co-located blocks at position D is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors (up to 3) are averaged for each reference list respectively. The averaged motion vector is assigned as the motion vector of the current sub-CU.

[0161] Figure 20 An example of one CU having four sub-blocks (AD) and their neighboring blocks (ad) is shown.

[0162] 2.3.1.3 Sub-CU Motion Prediction Mode Signaling

[0163] The sub-CU mode is enabled as an additional Merge candidate, and no additional syntax elements are required to signal the mode. Two additional Merge candidates are added to the Merge candidate list of each CU to represent ATMVP mode and STMVP mode. If the sequence parameter set indicates that ATMVP and STMVP are enabled, up to seven Merge candidates are used. The encoding logic of the additional Merge candidates is the same as that used for the Merge candidates in HM, which means that for each CU in a P or B slice, two more RD checks are required for the two additional Merge candidates.

[0164] In JEM, all bins of the Merge index are CABAC context coded, whereas in HEVC, only the first bin is context coded and the remaining bins are context bypass coded.

[0165] 2.3.2 Adaptive Motion Vector Difference Resolution

[0166] In HEVC, when use_integer_mv_flag in the slice header is equal to 0, the motion vector difference (MVD) (between the PU's motion vector and the predicted motion vector) is signaled in units of quarter luma samples. In JEM, Locally Adaptive Motion Vector Resolution (LAMVR) is introduced. In JEM, MVD can be coded in units of quarter luma samples, integer luma samples, or four luma samples. The MVD resolution is controlled at the coding unit (CU) level, and the MVD resolution flag is conditionally signaled for each CU with at least one non-zero MVD component.

[0167] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter luma sample MV precision is not used, another flag is signaled to indicate whether integer luma sample MV precision or four luma sample MV precision is used.

[0168] When the first MVD resolution flag of a CU is zero or not encoded for the CU (meaning all MVDs in the CU are zero), a quarter luma sample MV resolution is used for the CU. When the CU uses integer luma sample MV precision or four luma sample MV precision, the MVP in the AMVP candidate list for the CU is rounded to the corresponding precision.

[0169] In the encoder, CU-level RD check is used to determine which MVD resolution to use for a CU. That is, three CU-level RD checks are performed for each MVD resolution. To speed up the encoder, the following encoding scheme is applied in JEM.

[0170] During RD check of a CU with normal quarter luma sample MVD resolution, the motion information of the current CU (integer luma sample precision) is stored. The stored (after rounding) motion information is used as a starting point for further small-scale motion vector refinement during RD check of the same CU with integer luma samples and 4 luma sample MVD resolution, so that the time-consuming motion estimation process is not repeated three times.

[0171] Conditionally call RD check for CUs with 4 luma sample MVD resolution. For a CU, skip RD check for 4 luma sample MVD resolution of the CU when the RD cost integer luma sample MVD resolution is much larger than the resolution of quarter luma sample MVD.

[0172] 2.3.3 Motion Vector Derivation by Pattern Matching

[0173] The Pattern Matched Motion Vector Derivation (PMMVD) mode is a special Merge mode based on the Frame-Rate Up Conversion (FRUC) technology. In this mode, the motion information of the block is not signaled but derived at the decoder side.

[0174] When the Merge flag of a CU is true, the FRUC flag is signaled for the CU. When the FRUC flag is false, the Merge index is signaled and the normal Merge mode is used. When the FRUC flag is true, an additional FRUC mode flag is signaled to indicate which method (bilateral matching or template matching) to use to derive the motion information of the block.

[0175] At the encoder side, the decision on whether to use FRUC Merge mode for a CU is based on RD cost selection, just like for normal merge candidates. That is, two matching modes (bilateral matching and template matching) are checked for the CU using RD cost selection. The mode that results in the minimum cost is further compared with the other CU modes. If the FRUC matching mode is the most efficient mode, the FRUC flag is set to true for the CU, and the relevant matching mode is used.

[0176] The motion derivation process in FRUC Merge mode has two steps. First, a CU-level motion search is performed, followed by sub-CU-level motion refinement. At the CU level, an initial motion vector is derived for the entire CU based on bilateral matching or template matching. First, a list of MV candidates is generated, and the candidate that results in the minimum matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bilateral matching or template matching is performed around the starting point, and the MV resulting in the minimum matching cost is adopted as the MV for the entire CU. Subsequently, the motion information is further refined at the sub-CU level, using the derived CU motion vector as the starting point.

[0177] For example, the following derivation process is performed for the W×H CU motion information derivation. In the first stage, the MV of the entire W×H CU is derived. In the second stage, the CU is further divided into M×M sub-CUs. The value of M is calculated as in (16), where D is the predefined partition depth, which is set to 3 by default in JEM. The MV of each sub-CU is then derived.

[0178]

[0179] like Figure 21As shown in , bilateral matching is used to derive the motion information of the current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. Under the assumption of continuous motion trajectory, the motion vectors MV0 and MV1 pointing to the two reference blocks should be proportional to the temporal distance between the current picture and the two reference pictures (i.e., TD0 and TD1). As a special case, when the current picture is temporally between the two reference pictures and the temporal distance from the current picture to the two reference pictures is the same, bilateral matching becomes mirror-based bidirectional MV.

[0180] like Figure 22 As shown in , template matching is used to derive the motion information of the current CU by finding the closest match between the template in the current picture (the top and / or left neighboring blocks of the current CU) and the block (of the same size as the template) in the reference picture. In addition to the aforementioned FRUC Merge mode, template matching is also applicable to AMVP mode. In JEM, as done in HEVC, AMVP has two candidates. Using the template matching method, a new candidate is derived. If the candidate newly derived by template matching is different from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and then the list size is set to 2 (meaning removing the second existing AMVP candidate). When applied to AMVP mode, only CU level search is applied.

[0181] 2.3.3.1 CU-level MV candidate set

[0182] The MV candidate set at the CU level consists of the following:

[0183] (i) If the current CU is in AMVP mode, the original AMVP candidate

[0184] (ii) All Merge candidates,

[0185] (iii) Several MVs in the interpolated MV field.

[0186] (iv) Top and left neighboring motion vectors

[0187] When bilateral matching is used, each valid MV of the Merge candidate is used as input to generate an MV pair under the assumption of bilateral matching. For example, one valid MV of the Merge candidate is (MVa, refa) at reference list A. Then, a reference picture refb of its paired bilateral MV is found in other reference list B, so that refa and refb are on different sides of the current picture in time. If such refb is not available in reference list B, refb is determined as a reference different from refa, and its temporal distance to the current picture is the smallest one in list B. After determining refb, MVb is derived by scaling MVa based on the temporal distance between the current picture and refa and refb.

[0188] The four MVs from the interpolated MV field are also added to the CU level candidate list. More specifically, the interpolated MVs at positions (0, 0), (W / 2, 0), (0, H / 2), and (W / 2, H / 2) of the current CU are added.

[0189] When FRUC is applied in AMVP mode, the original AMVP candidates are also added to the CU-level MV candidate set.

[0190] At the CU level, up to 15 MVs of an AMVP CU and up to 13 MVs of a Merge CU are added to the candidate list.

[0191] 2.3.3.2 Sub-CU Level MV Candidate Set

[0192] The MV candidate set at the sub-CU level consists of the following:

[0193] (i) Search the determined MV from the CU level,

[0194] (ii) top, left, upper-left, and upper-right adjacent MVs,

[0195] (iii) a scaled version of the co-located MV from the reference picture,

[0196] (iv) Up to 4 ATMVP candidates,

[0197] (v) Up to 4 STMVP candidates

[0198] The scaled MV from the reference picture is derived as follows: Traverse all reference pictures in both lists. The MV at the collocated position of the sub-CU in the reference picture is scaled to the reference of the starting CU level MV.

[0199] ATMVP and STMVP candidates are limited to the top four.

[0200] At the sub-CU level, up to 17 MVs are added to the candidate list.

[0201] 2.3.3.3 Generation of interpolated MV fields

[0202] Before encoding a frame, an interpolated motion field is generated for the entire picture based on unilateral ME. The motion field can then be used later as a CU-level or sub-CU-level MV candidate.

[0203] First, the motion fields of each reference picture in both reference lists are traversed at the 4×4 block level. For each 4×4 block, if the motion associated with the block crosses a 4×4 block in the current picture (e.g. Figure 23 ) and the block has not been assigned any interpolated motion, the motion of the reference block is scaled to the current picture according to the temporal distances TD0 and TD1 (in the same way as the MV scaling of TMVP in HEVC), and the scaled motion is assigned to the block in the current frame. If the scaled MV is not assigned to the 4×4 block, the motion of the block is marked as unavailable in the interpolated motion field.

[0204] 2.3.3.4 Interpolation and Matching Costs

[0205] When motion vectors point to fractional sample locations, motion compensated interpolation is required. To reduce complexity, bilinear interpolation is used instead of conventional 8-tap HEVC interpolation for bilateral matching and template matching.

[0206] The calculation of the matching cost is slightly different at different steps. When selecting a candidate from the candidate set at the CU level, the matching cost is the absolute sum difference (SAD) of bilateral matching or template matching. After determining the starting MV, the matching cost C of bilateral matching at the sub-CU level search is calculated as follows.

[0207]

[0208] where w is a weighting factor empirically set to 4, MV and MV s Indicates the current MV and the start MV respectively. SAD is still used as the matching cost of template matching at the sub-CU level search.

[0209] In FRUC mode, MV is derived by using only luma samples. The derived motion will be used for both luma and chroma for MC inter-frame prediction. After deciding the MV, the final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma.

[0210] 2.3.3.5MV Refinement

[0211] MV refinement is a pattern-based MV search with criteria of bilateral matching cost or template matching cost. In JEM, two search modes are supported - respectively, Unrestricted Center-Biased Diamond Search (UCBDS) and Adaptive Cross Search for MV refinement at CU level and sub-CU level. For both CU and sub-CU level MV refinement, MVs are searched directly with quarter luma sample MV precision, followed by eighth luma sample MV refinement. The search range of MV refinement for CU and sub-CU steps is set to equal 8 luma samples.

[0212] 2.3.3.6 Prediction Direction Selection in Template Matching FRUC Merge Mode

[0213] In bilateral matching Merge mode, bidirectional prediction is always applied because the motion information of the CU is derived based on the closest match between two blocks in two different reference pictures along the motion trajectory of the current CU. There is no such restriction for template matching Merge mode. In template matching Merge mode, the encoder can choose between unidirectional prediction according to list0, unidirectional prediction according to list1, or bidirectional prediction of the CU. The selection is based on the template matching cost as follows:

[0214] If costBi<=factor*min(cost0,cost1)

[0215] Use bidirectional prediction;

[0216] Otherwise, if cost0 <= cost1

[0217] Use one-way prediction based on list0;

[0218] otherwise,

[0219] Use one-way prediction based on list1;

[0220] Where cost0 is the SAD of list0 template matching, cost1 is the SAD of list1 template matching, and costBi is the SAD of bidirectional prediction template matching. The value of factor is equal to 1.25, which means that the selection process is biased towards bidirectional prediction.

[0221] Inter prediction direction selection is only applied to the CU level template matching process.

[0222] 2.3.4 Decoder-side Motion Vector Refinement

[0223] In bidirectional prediction, for the prediction of a block region, two prediction blocks, formed using motion vectors (MVs) from list0 and list1, are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of the bidirectional prediction are further refined through a bilateral template matching process. Bilateral template matching is applied in the decoder to perform a distortion-based search between the bilateral template and the reconstructed samples in the reference picture to obtain the refined MVs without transmitting additional motion information.

[0224] In DMVR, such as Figure 23 As shown in , a bilateral template is generated from the initial MV0 of list0 and MV1 of list1 as a weighted combination (i.e., average) of the two prediction blocks, respectively. The template matching operation consists of calculating a cost measure between the generated template and the sample area in the reference picture (around the initial prediction block). For each of the two reference pictures, the MV that induces the minimum template cost is considered as the updated MV of the list to replace the original MV. In JEM, nine MV candidates are searched for each list. The nine MV candidates include the original MV and 8 surrounding MVs that are offset by one luminance sample in the horizontal or vertical direction, or in both. Finally, the two new MVs, i.e. Figure 24 MV0' and MV1' shown in are used to generate the final bidirectional prediction result. The sum of absolute differences (SAD) is used as the cost metric.

[0225] DMVR is applied to the Merge mode of bi-prediction, where one MV comes from a past reference picture and the other MV comes from a future reference picture, without the transmission of additional syntax elements. In JEM, DMVR is not applied when LIC, affine motion, FRUC, or sub-CU Merge candidate is enabled for a CU.

[0226] 2.3.5 Merge / Skip Mode with Bilateral Match Refinement

[0227] The Merge candidate list is first constructed by inserting the motion vectors and reference indices of spatially and temporally adjacent blocks into the candidate list using redundancy check until the number of available candidates reaches the maximum candidate size of 19. The spatial candidates are inserted according to a predefined insertion order ( Figure 11 ), temporal candidates, affine candidates, advanced temporal MVP (ATMVP) candidates, spatial temporal MVP (STMVP) candidates, and additional candidates (combined candidates and zero candidates) as used in HEVC to construct the candidate list for Merge / Skip mode:

[0228] - Spatial candidates for blocks 1-4.

[0229] - Presumed affine candidates for blocks 1-4.

[0230] -ATMVP.

[0231] -STMVP.

[0232] -Virtual affine candidate.

[0233] - Spatial candidates (block 5) (used only when the number of available candidates is less than 6).

[0234] - Inferred affine candidates (block 5).

[0235] - Temporal candidates (as derived in HEVC).

[0236] - non-adjacent spatial candidates, followed by the inferred affine candidates (blocks 6 to 49, e.g. Figure 25 as depicted in ).

[0237] -Candidates for combination.

[0238] -Zero Candidates

[0239] Note that in addition to STMVP and Affine, the IC flag is also inherited from the Merge candidate. In addition, for the first four spatial candidates, bidirectional prediction candidates are inserted before candidates with unidirectional prediction.

[0240] In some embodiments, blocks that are not connected to the current block may be accessed.If non-neighboring blocks are coded in non-intra mode, the associated motion information may be added as additional Merge candidates.

[0241] 3. Examples of Problems Solved by the Embodiments Disclosed herein

[0242] The current HEVC design can better encode motion information by using the correlation between the current block and its neighboring blocks (immediately adjacent to the current block). However, it is possible that the neighboring blocks correspond to different objects with different motion trajectories. In this case, the prediction from its neighboring blocks is not effective.

[0243] Prediction of motion information from non-neighboring blocks can bring additional coding gain, which has the cost of storing all motion information (typically on a 4x4 level) into cache, which significantly increases the complexity of hardware implementation.

[0244] 4. Some Examples

[0245] To overcome the shortcomings of existing implementations, a LUT-based motion vector prediction technique that uses one or more tables (e.g., lookup tables) with at least one stored motion candidate to predict motion information of a block can be implemented in various embodiments to provide video encoding with higher coding efficiency. A lookup table is an example of a table that can be used to include motion candidates to predict motion information of a block, and other implementations are also possible. Each LUT can include one or more motion candidates, each motion candidate being associated with corresponding motion information. The motion information of a motion candidate can include some or all of a prediction direction, a reference index / picture, a motion vector, a LIC flag, an affine flag, a motion vector derivation (MVD) accuracy, and / or an MVD value. The motion information can also include block position information to indicate where the motion information is coming from.

[0246] LUT-based motion vector prediction based on the disclosed technology, which can enhance both existing and future video coding standards, is illustrated in the following examples described with respect to various embodiments. Because the LUT allows the encoding / decoding process to be performed based on historical data (e.g., blocks that have already been processed), the LUT-based motion vector prediction can also be referred to as a history-based motion vector prediction (HMVP) method. In the LUT-based motion vector prediction method, one or more tables with motion information from previously encoded blocks are maintained during the encoding / decoding process. These motion candidates stored in the LUT are named HMVP candidates. During the encoding / decoding of a block, the associated motion information in the LUT can be added to a motion candidate list (e.g., a Merge / AMVP candidate list), and after encoding / decoding a block, the LUT can be updated. The updated LUT is then used to encode subsequent blocks. That is, the updating of the motion candidates in the LUT is based on the encoding / decoding order of the block.

[0247] The following examples should be considered as examples used to explain the general concepts. They should not be interpreted in a narrow sense. Furthermore, these examples can be combined in any way.

[0248] Some embodiments may use one or more lookup tables with at least one stored motion candidate to predict motion information for a block. Embodiments may use motion candidates to indicate a set of motion information stored in a lookup table. For traditional AMVP or Merge modes, embodiments may use AMVP or Merge candidates to store motion information.

[0249] The following example explains the general concept.

[0250] Example of a lookup table

[0251] Example A1: Each lookup table may contain one or more motion candidates, where each candidate is associated with its motion information.

[0252] a. The motion information of the motion candidate here may include part or all of the prediction direction, reference index / picture, motion vector, LIC flag, affine flag, MVD accuracy, and MVD value.

[0253] b. Motion information may also include block position information and / or block shape to indicate where the motion information is coming from.

[0254] LUT Selection

[0255] Example B1: For coding a block, some or all motion candidates from a lookup table may be checked in order. When a motion candidate is checked during coding a block, it may be added to a motion candidate list (eg, AMVP, Merge candidate list).

[0256] Example B2: The choice of the lookup table may depend on the location of the block.

[0257] Use of lookup tables

[0258] Example C1: The total number of motion candidates to be checked in the lookup table may be predefined.

[0259] Example C2: The motion candidate(s) included in the lookup table can be directly inherited by the block.

[0260] a. They can be used for Merge mode coding, i.e., motion candidates can be checked during Merge candidate list derivation.

[0261] b. They can be used for affine Merge mode encoding.

[0262] i. If its affine flag is 1, the motion candidate in the lookup table can be added as an affine Merge candidate.

[0263] c. Checking for motion candidates in the lookup table can be enabled when:

[0264] i. After the TMVP candidate is inserted, the Merge candidate list is not full;

[0265] ii. After checking a certain spatial neighboring block for spatial merge candidate derivation, the merge candidate list is not full;

[0266] iii. After all spatial merge candidates are considered, the merge candidate list is not full;

[0267] iv. After the combined bidirectional prediction Merge candidate, the Merge candidate list is not full;

[0268] v. When the number of spatial or temporal Merge candidates (e.g., including adjacent spaces and non-adjacent spaces, TMVP, STMVP, ATMVP, etc.) that have been placed into the Merge candidate list according to other encoding methods (e.g., the Merge derivation process of HEVC design, or JEM design) is less than the maximum allowed Merge candidates minus a given threshold.

[0269] 1. In one example, the threshold is set to 1 or 0.

[0270] 2. Alternatively, the threshold may be signaled or predefined in the SPS / PPS / sequence, picture, slice header / slice.

[0271] 3. Alternatively, the threshold can be adaptively changed from block to block. For example, it can depend on the coded block information, such as block size / block shape / slice type, and / or on the number of available spatial or temporal merge candidates.

[0272] 4. In another example, when the number of certain Merge candidates already placed in the Merge candidate list is less than the maximum allowed Merge candidate minus a given threshold, "certain Merge candidates" may be spatial candidates or non-neighboring Merge candidates as in HEVC.

[0273] vi. Pruning can be applied before adding motion candidates to the Merge candidate list. In various embodiments in this example and other examples in this patent document, pruning can include at least one of: a) comparing the motion information with existing entries for uniqueness, or b) adding the motion information to the list if it is unique, or c) if it is not unique, c1) not adding it, or c2) adding the motion information and deleting the matching existing entry.

[0274] 1. In one example, the motion candidate can be pruned from all or part of the available spatial or temporal (eg, including adjacent and non-adjacent spaces, TMVP, STMVP, ATMVP, etc.) Merge candidates in the Merge candidate list from other encoding methods.

[0275] 2. Motion candidates may not be pruned with sub-block based motion candidates, such as ATMVP, STMVP.

[0276] 3. In one example, the current motion candidate and all or part of the available motion candidates in the Merge candidate list (inserted before the current motion candidate) may be pruned.

[0277] 4. The number of pruning operations related to motion candidates (e.g., how many times a motion candidate needs to be compared with other candidates in the merge list) can depend on the number of available spatial or temporal merge candidates. For example, when checking for a new motion candidate, if there are M candidates available in the merge list, the new motion candidate can be compared only with the first K (K <= M) candidates. If the pruning function returns false (e.g., not equal to any of the first K candidates), the new motion candidate is considered different from all M candidates and can be added to the merge candidate list.

[0278] 5. In one example, the newly appended motion candidate is only compared with the first N candidates in the Merge candidate list. For example, N = 3, 4 or 5. N can be signaled from the encoder to the decoder.

[0279] 6. In one example, only new motion candidates to be checked are compared with the last N candidates in the Merge candidate list. For example, N = 3, 4 or 5. N can be signaled from the encoder to the decoder.

[0280] 7. In one example, how to select the candidate previously added in the list to be compared with the new motion candidate from the table may depend on where the previously added candidate was derived from.

[0281] a. In one example, motion candidates in a lookup table can be compared with candidates derived from given temporal and / or spatial neighboring blocks.

[0282] b. In one example, different entries of motion candidates in the lookup table may be compared with different previously added candidates (ie, derived from different positions).

[0283] Example C3: The motion candidate(s) included in the lookup table may be used as prediction values for the motion information of the coding block.

[0284] a. They can be used for AMVP mode coding, i.e., motion candidates can be checked during AMVP candidate list derivation.

[0285] b. Checking for motion candidates in the lookup table can be enabled when:

[0286] i. After inserting the TMVP candidate, the AMVP candidate list is not full;

[0287] ii. Just before inserting the TMVP candidate, after selecting and pruning from spatial neighbors, the AMVP candidate list is not full;

[0288] iii. When there is no AMVP candidate from the upper neighboring block that does not need to be scaled and / or when there is no AMVP candidate from the left neighboring block that does not need to be scaled

[0289] iv. Pruning can be applied before adding the motion candidate to the AMVP candidate list.

[0290] v. Rules similar to those in Example C2 may be applied to the AMVP mode.

[0291] c. Check motion candidates that have the same reference picture as the current reference picture.

[0292] i. Alternatively, additionally, motion candidates with reference pictures different from the current reference picture are also checked (in case of scaled MV).

[0293] ii. Alternatively, first check all motion candidates that have the same reference picture as the current reference picture, and then check motion candidates that have a reference picture different from the current reference picture.

[0294] iii. Alternatively, checking for motion candidates is the same as Merge.

[0295] Example C4: The order of checking motion candidates in the lookup table is defined as follows (assuming that K (K>=1) motion candidates are allowed to be checked):

[0296] a. Look up the last K motion candidates in the table, eg, in descending order of the entry index of the LUT.

[0297] b. Top K%L candidates, where L is the lookup table size, K>=L, eg, in descending order of entry index of the LUT.

[0298] c. Based on the order, when K>=L, look up all candidates in the table (L candidates). In one example, the first K%L candidates in the table are checked in descending order of the entry index of the LUT, and then the last (LK%L) candidates are checked in descending order of the entry index.

[0299] d. Alternatively, further, based on the descending order of motion candidate index (ie, entry index of the LUT).

[0300] e. Alternatively, further, based on ascending order of motion candidate indices.

[0301] f. Alternatively, select K motion candidates based on candidate information, such as distances of locations associated with the motion candidates and the current block.

[0302] i. In one example, select the K closest motion candidates.

[0303] ii. In one example, the candidate information may also take block shape into consideration when calculating distance.

[0304] g. In one example, the order of checking K motion candidates from a table of L candidates can be defined as: selecting in order the candidate with index equal to a0, a0+T0, a0+T0+T1, a0+T0+T1+T2....a0+T0+T1+T2+..+T K-1 Those candidates, where a0 and T i (i is 0...K-1) is an integer value.

[0305] In one example, a0 is set to 0 (i.e., the first entry of the motion candidate in the table). Alternatively, a0 is set to (KL / K). The arithmetic operation " / " is defined as integer division with truncation of the result towards zero. Alternatively, a0 is set to any integer between 0 and L / K.

[0306] 1. Alternatively, the value of a0 may depend on the coding information of the current block and the neighboring blocks.

[0307] ii. In one example, all intervals T i (i is 0...K-1) are the same, such as L / K. The arithmetic operation " / " is defined as integer division with truncation of the result towards zero.

[0308] iii. In one example, (K, L, a0, T i ) is set to (4, 16, 0, 4), or (4, 12, 0, 3), or (4, 8, 0, 1), or (4, 16, 3, 4), or (4, 12, 2, 3), or (4, 8, 1, 2). Ti is the same for all i.

[0309] iv. Such an approach may only be applicable when K is less than L.

[0310] v. Alternatively, further, when K is greater than or equal to a threshold, subsection c of embodiment C4 may apply. The threshold may be defined as L, or it may depend on K or vary adaptively from block to block. In one example, the threshold may depend on the number of available motion candidates in the list before adding a new motion candidate from the lookup table.

[0311] h. In one example, the order of checking K of the motion candidates from the table of L candidates can be defined as: selecting in order the candidate with index equal to a0, a0-T0, a0-T0-T1, a0-T0-T1-T2...a0-T0-T1-T2-..-T K-1 Those candidates, where a0 and T i (i is 0...K-1) is an integer value.

[0312] i. In one example, a0 is set to L-1 (ie, the last entry of the motion candidate in the table). Alternatively, a0 is set to any integer between L-1-L / K and L-1.

[0313] ii. In one example, all intervals T i (i is 0...K-1) are the same, such as L / K.

[0314] iii. In one example, (K, L, a0, T i ) is set to (4, 16, L-1, 4), or (4, 12, L-1, 3), or (4, 8, L-1, 1), or (4, 16, L-4, 4), or (4, 12, L-3, 3), or (4, 8, L-2, 2). T i Same for all i.

[0315] iv. Such an approach may only be applicable when K is less than L.

[0316] Alternatively, further, when K is greater than or equal to a threshold, subsection c of embodiment C4 may apply. The threshold may be defined as L, or it may depend on K or vary adaptively from block to block. In one example, the threshold may depend on the number of available motion candidates in the list before adding a new motion candidate from the lookup table.

[0317] i. How many and / or how motion candidates are selected from the lookup table may depend on encoding information such as block size / block shape.

[0318] i. In one example, for smaller block sizes, instead of selecting the last K motion candidates, other K motion candidates (not starting from the last one) may be selected.

[0319] ii. In one example, the encoding information may be AMVP or Merge mode.

[0320] iii. In one example, the encoding information may be an affine mode, a non-affine AMVP mode, or a non-affine Merge mode.

[0321] iv. In one example, the encoding information may be an affine AMVP (inter) mode or an affine Merge mode or a non-affine AMVP mode or a non-affine Merge mode.

[0322] v. In one example, the coding information may be a Current Picture Reference (CPR) mode or not a CPR mode.

[0323] vi. Alternatively, how to select a motion candidate from the lookup table may also depend on the number of motion candidates in the lookup table, and / or the number of available motion candidates in the list before adding a new motion candidate from the lookup table.

[0324] j. In one example, the maximum number of motion candidates to be checked in the lookup table (ie, the number of motion candidates that can be added to the Merge / amvp candidate list) can depend on the number of available motion candidates in the lookup table (given by N avaiMCinLUT ), and / or the maximum allowed motion candidate to be added from the lookup table (denoted by NUM maxMC flag) (which may be predefined or signaled), and / or the number of available candidates in the candidate list before checking candidates from the lookup table (denoted by N avaiC mark).

[0325] i. In one example, the maximum number of motion candidates to be checked in the lookup table is set to (N avaiMCinLUT 、NUM maxMC 、N avaiC ) is the minimum value among .

[0326] ii. Alternatively, the maximum number of motion candidates to be checked in the lookup table is set to (N avaiMCinLUT 、NUM maxMC -N avaiC ) is the minimum value among .

[0327] iii. In one example, N avaiC Indicates the number of insertion candidates derived from spatial or temporal (adjacent and / or non-adjacent) neighboring blocks. Alternatively, in addition, avaiC The number of sub-block candidates (like AMTVP, STMVP) is not counted.

[0328] iv.NUM maxMC It may depend on the encoding mode, for example, for Merge mode and AMVP mode, NUM maxMC Can be set to different values. In one example, for Merge mode, NUM maxMC It can be set to 4, 6, 8, 10, etc. For AMVP mode, NUM maxMC Can be set to 1, 2, 4, etc.

[0329] v.Alternatively, NUM maxMC It may depend on other coding information like block size, block shape, slice type etc.

[0330] k. The order in which the different lookup tables are checked is defined in the next subsection on the use of lookup tables.

[0331] l. Once the Merge / AMVP candidate list reaches the maximum number of allowed candidates, the checking process will terminate.

[0332] m. Once the Merge / AMVP candidate list reaches the maximum allowed number of candidates minus a threshold (Th), the checking process will terminate. In one example, Th can be predefined as a positive integer value, such as 1, 2, or 3. Alternatively, Th can be adaptively changed from block to block. Alternatively, Th can be signaled in the SPS / PPS / slice header, etc. Alternatively, Th can also depend on block shape / block size / coding mode, etc. Alternatively, Th can depend on how many available candidates are before adding motion candidates from the LUT.

[0333] Alternatively, once the number of added motion candidates reaches the maximum allowed number of motion candidates, the checking process will terminate. The maximum allowed number of motion candidates can be signaled or predefined. Alternatively, the maximum allowed number of motion candidates can also depend on block shape / block size / coding mode, etc.

[0334] o. One syntax element indicating the table size and the number of motion candidates allowed to be checked (ie, K=L) may be signaled in the SPS, PPS, slice header, or slice header.

[0335] Example C5: Enabling / disabling the use of a lookup table for motion information coding of a block may be signaled in an SPS, PPS, slice header, slice header, CTU, CTB, CU or PU, or an area covering multiple CTUs / CTBs / CUs / PUs.

[0336] Example C6: Whether to apply prediction from a lookup table can also depend on coding information. When it is inferred that it is not applicable to the block, additional signaling of the indication of prediction is skipped. Alternatively, when it is inferred that it is not applicable to the block, the motion candidates of the lookup table do not need to be accessed, and the checking of relevant motion candidates is omitted.

[0337] Example C7: Motion candidates from a lookup table in a previously coded frame / slice / slice may be used to predict motion information for a block in a different frame / slice / slice.

[0338] Update of the lookup table

[0339] Example D1: After encoding a block with motion information (ie, IntraBC mode, Inter coding mode), one or more lookup tables may be updated.

[0340] Additional Example Embodiments

[0341] A history-based MVP (HMVP) method is proposed, where the HMVP candidate is defined as the motion information of the previously coded block. A table with multiple HMVP candidates is maintained during the encoding / decoding process. When a new slice is encountered, the table is cleared. Whenever there is an inter-coded block, the associated motion information is added to the last entry of the table as a new HMVP candidate. The entire encoding process is Figure 30 Depicted in.

[0342] In one example, the table size is set to L (eg, L=16 or 6, or 44), L indicating that up to L HMVP candidates can be added to the table.

[0343] In one embodiment (corresponding to Example 11.gi), if there are more than L HMVP candidates from previously encoded blocks, a First-In-First-Out (FIFO) rule is applied so that the table always contains the most recent L previously encoded motion candidates. Figure 31 Depicted is an example of applying the FIFO rule to remove an HMVP candidate and add a new one to the table used in the proposed method.

[0344] In another embodiment (corresponding to Example 11.g.iii), whenever a new motion candidate is added (such as when the current block is inter-coded and in non-affine mode), a redundancy check process is first applied to identify whether the same or similar motion candidate exists in a table (e.g., LUT).

[0345] Some examples are depicted below:

[0346] Figure 32A An example is shown when the table is full before adding a new motion candidate.

[0347] Figure 32B An example is shown when the table is not full before adding a new motion candidate.

[0348] Figure 32A and Figure 32B An example of a table updating method based on redundancy removal is shown together (with one redundant motion candidate removed). Figure 32A As shown in (top), there are L table entries before the update. During the redundancy check, HMVP2 is found to be redundant (e.g., duplicated) and is removed, creating L-1 entries to which the new entry can be added (bottom). Figure 32B The removal of redundant candidates is similarly shown.

[0349] Figure 33A and Figure 33B Example implementations for two cases of the redundancy removal-based table updating method are shown (with multiple redundant motion candidates removed, 2 candidates in the figure).

[0350] Figure 33A An example situation is shown when the table is full before adding new motion candidates.

[0351] Figure 33B An example case is shown when the table is not full before adding a new motion candidate.

[0352] HMVP candidates can be used in the Merge candidate list construction process. All HMVP candidates from the last entry to the first entry in the table (or the last K0 HMVPs, for example, K0 is equal to 16 or 6) are inserted after the TMVP candidate. Pruning is applied to the HMVP candidates. Once the total number of available Merge candidates reaches the maximum allowed Merge candidates signaled, the Merge candidate list construction process is terminated. Alternatively, once the total number of added motion candidates reaches a given value, the extraction of motion candidates from the LUT is terminated.

[0353] Similarly, HMVP candidates can also be used in the AMVP candidate list construction process. The motion vectors of the last K1 HMVP candidates in the table are inserted after the TMVP candidates. Only HMVP candidates with the same reference picture as the AMVP target reference picture are used to construct the AMVP candidate list. Pruning is applied to HMVP candidates. In one example, K1 is set to 4.

[0354] Figure 28 is a block diagram of a video processing device 2800. Device 2800 can be used to implement one or more of the methods described herein. Device 2800 can be embodied in a smartphone, a tablet, a computer, an Internet of Things (IoT) receiver, etc. Device 2800 may include one or more processors 2802, one or more memories 2804, and video processing hardware 2806. Processor(s) 2802 can be configured to implement one or more methods described in this document. Memory(s) 2804 can be used to store code and data for implementing the methods and techniques described herein. Video processing hardware 2806 can be used to implement some of the techniques described in this document in hardware circuitry.

[0355] Figure 29 2 is a flow chart of an example of a video processing method 2900. The method 2900 includes maintaining tables at step 2902, wherein each table includes a set of motion candidates, and each motion candidate is associated with corresponding motion information. The method 2900 also includes, at step 2904, performing a conversion between a first video block and a bitstream representation of a video including the first video block based on at least one of the one or more tables, the performing including checking the motion candidates in the tables in an order defined in a rule until a checking termination criterion is satisfied.

[0356] Features and embodiments of the above-described methods / techniques are described below.

[0357] 1. A video processing method, comprising: maintaining tables, wherein each table includes a set of motion candidates, and each motion candidate is associated with corresponding motion information; and performing a conversion between a first video block and a bitstream representation of a video including the first video block based at least on the one or more tables, the conversion comprising checking the motion candidates in the tables in an order defined in a rule until a checking termination criterion is satisfied.

[0358] 2. The method of clause 1, wherein the rule specifies the last K motion candidates in the check list, K being equal to or greater than 1.

[0359] 3. The method of clause 1, wherein the rule specifies checking the top K modulo L (K%L) candidates in the table, K being equal to or greater than 1, and L indicating the number of entries in the table.

[0360] 4. The method of clause 1, wherein the rule specifies checking all L motion candidates in a table, L indicating the number of entries in the table.

[0361] 5. The method of clause 1, wherein the rule specifies that the motion candidates in the table are checked based on descending order of motion candidate index.

[0362] 6. The method of clause 1, wherein the rule specifies that the motion candidates in the table are checked based on ascending order of motion candidate index.

[0363] 7. The method of clause 1, wherein the rule specifies selecting K motion candidates based on the candidate information, K being equal to or greater than 1.

[0364] 8. The method of clause 7, wherein the candidate information comprises a distance between the first video block and a location associated with the motion candidate.

[0365] 9. The method of clause 7, wherein the K closest motion candidates are selected.

[0366] 10. The method of clause 8, wherein the candidate information includes block shapes considered to calculate the distance.

[0367] 11. The method of clause 7, wherein the selection of the K motion candidates depends on encoding information of the first video block.

[0368] 12. The method of clause 11, wherein the encoding information comprises at least one of a block size or a block shape.

[0369] 13. A method as described in clause 11, wherein the encoding information of the first video block indicates at least one of the following: AMVP mode, Merge mode, affine mode, non-affine AMVP mode, non-affine Merge mode, affine AMVP (inter-frame) mode, affine Merge mode, non-affine AMVP mode, non-affine Merge mode, current picture reference (CPR) mode, or non-CPR mode.

[0370] 14. The method of clause 7, wherein the selection of the K motion candidates depends on at least one of i) the number of motion candidates in the table or ii) the number of available candidates in the motion candidate list before adding a new one from the table.

[0371] 15. The method of clause 11, wherein, for smaller block sizes, the selected K motion candidates do not correspond to the last K motion candidates in the table.

[0372] 16. The method of clause 1, wherein the motion candidates in another table are checked in another order.

[0373] 17. A method as recited in any of clauses 1-16, wherein the checking termination criterion is met when a motion candidate list comprising at least one of a Merge or an AMVP (Advanced Motion Vector Prediction) candidate list reaches a maximum allowed number of candidates.

[0374] 18. The method of any of clauses 1-16, wherein a checking termination criterion is met when the number of motion candidates to be checked that are added from the table reaches a maximum allowed number of motion candidates.

[0375] 19. The method of clause 18, wherein a maximum allowed number of motion candidates to be checked is predefined or signaled.

[0376] 20. The method of clause 18, wherein a maximum allowed number of motion candidates from the table to be checked depends on at least one of block shape, block size, or coding mode.

[0377] 21. The method of clause 1, further comprising signaling at least one of: i) a size of the table, or ii) a number of motion candidates from the table to be checked.

[0378] 22. A method as recited in any of clauses 1-16, wherein the checking termination criterion is satisfied when the number of candidates in the candidate list reaches a maximum allowed number of candidates for the candidate list minus a threshold value (Th).

[0379] 23. The method of clause 22, wherein the threshold value (Th) is predefined as a positive integer.

[0380] 24. The method of clause 22, wherein the threshold value (Th) varies depending on the block.

[0381] 25. The method of clause 22, wherein the threshold (Th) is signaled in one or more of a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, or a slice header.

[0382] 26. The method of clause 22, wherein the threshold (Th) depends on at least one of block shape, block size, or coding mode.

[0383] 27. The method of clause 22, wherein the threshold (Th) depends on the number of available candidates before adding motion candidates from the table.

[0384] 28. A method as described in any of clauses 1-16, wherein the checking termination criteria depends on at least one of: the number of available motion candidates in the table, the maximum allowed motion candidates to be added from the table, the number of available candidates in the motion candidate list before checking the motion candidates from the table.

[0385] 29. The method of clause 1, wherein the index is equal to a0, a0+T0, a0+T0+T1, a0+T0+T1+T2, ... a0+T0+T1+T2+..+T K-1 K motion candidates, the order includes the order of checking K motion candidates from a table including L candidates, where a0 and T i (i is 0...K-1) is an integer value.

[0386] 30. The method of clause 29, wherein a0 is set to 0 as the first entry of the motion candidate in the table.

[0387] 31. The method of clause 29, wherein a0 is set to (KL / K), the arithmetic operation " / " being defined as integer division with truncation of the result towards zero.

[0388] 32. The method of clause 29, wherein a0 is set to any integer between 0 and L / K, and the arithmetic operation " / " is defined as integer division with truncation of the result towards zero.

[0389] 33. The method of clause 29, wherein a0 depends on coding information of the first video block and neighboring blocks.

[0390] 34. The method of clause 29, wherein T i (i is an integer between 0 and K-1) have the same value.

[0391] 35. The method of clause 34, wherein the identical value is L / K, the arithmetic operation " / " being defined as integer division with truncation of the result towards zero.

[0392] 36. The method of clause 29, wherein (K, L, a0, T i ) is set to at least one of (4,16,0,4), (4,12,0,3), (4,8,0,1), (4,16,3,4), (4,12,2,3), or (4,8,1,2).

[0393] 37. The method of clause 29, wherein the checking order is applied when K is less than L.

[0394] 38. The method of clause 29, wherein the rule specifies that all L motion candidates in the table are checked when K is greater than or equal to a threshold.

[0395] 39. The method of clause 38, wherein the threshold is L.

[0396] 40. The method of clause 38, wherein the threshold value varies depending on K or depending on the block.

[0397] 41. The method of clause 1, wherein the method comprises selecting, in order, a sequence of indexes equal to a0, a0-T0, a0-T0-T1, a0-T0-T1-T2, ... a0-T0-T1-T2-..-T K-1 K motion candidates, the order includes the order of checking K motion candidates from a table including L candidates, where a0 and T i (i is 0...K-1) is an integer value.

[0398] 42. The method of clause 41, wherein a0 is set to L-1 as the last entry of the motion candidate in the table.

[0399] 43. The method of clause 41, wherein a0 is set to any integer between L-1-L / K and L-1, and the arithmetic operation " / " is defined as integer division with truncation of the result towards zero.

[0400] 44. The method of clause 41, wherein T i (i is an integer between 0 and K-1) have the same value.

[0401] 45. The method of clause 44, wherein the identical value is L / K, the arithmetic operation " / " being defined as integer division with truncation of the result towards zero.

[0402] 46. The method of clause 41, wherein (K, L, a0, T i) is set to at least one of (4, 16, L-1, 4), (4, 12, L-1, 3), (4, 8, L-1, 1), (4, 16, L-4, 4), (4, 12, L-3, 3), or (4, 8, L-2, 2).

[0403] 47. The method of clause 41, wherein the checking order is applied when K is less than L.

[0404] 48. The method of clause 41, wherein the rule specifies that all L motion candidates in the table are checked when K is greater than or equal to a threshold.

[0405] 49. The method of clause 48, wherein the threshold is L.

[0406] 50. The method of clause 48, wherein the threshold value varies depending on K or depending on the block.

[0407] 51. A method as described in any of clauses 1 to 50, wherein the motion candidate is associated with motion information, the motion information comprising at least one of: a prediction direction, a reference picture index, a motion vector value, an intensity compensation flag, an affine flag, a motion vector difference precision, or a motion vector difference value.

[0408] 52. The method of any of clauses 1 to 50, further comprising adding the checked motion candidate to a Merge candidate list.

[0409] 53. The method of clause 52, wherein the order in which motion candidates added to the Merge candidate list are checked is based on descending order of motion candidate indices.

[0410] 54. The method of any of clauses 1 to 50, further comprising adding the motion vector of the checked motion candidate to an AMVP candidate list;

[0411] 55. The method of clause 54, wherein the order in which motion candidates added to the AMVP candidate list are checked is based on ascending order of motion candidate indices.

[0412] 56. The method of clause 54, wherein the number of motion candidates to be checked is no greater than 4.

[0413] 57. A method as described in clause 1, wherein the maximum number of motion candidates to be checked in the table depends on at least one of: the number of available motion candidates in the table indexed by N, the maximum allowed motion candidates to be added from the table indexed by M, or the number of available candidates in the candidate list before checking the motion candidates from the table indexed by K, where N, M, K are integers.

[0414] 58. The method of clause 57, wherein the maximum number of motion candidates to be checked in the table is set to the minimum of (N, M, K).

[0415] 59. The method of clause 57, wherein the maximum number of motion candidates to be checked in the table is set to the minimum of (N, MK).

[0416] 60. The method of clause 57, wherein K denotes the number of candidates for insertion derived from spatial or temporal neighboring blocks.

[0417] 61. A method as described in clause 57, wherein before checking the motion candidates from the table, the available candidates in the candidate list, considered as count K, do not include sub-block candidates that include optional temporal motion vector prediction (AMTVP) and spatio-temporal motion vector prediction (STMVP).

[0418] 62. The method of clause 57, wherein M depends on at least one of the encoding modes of the video.

[0419] 63. The method of clause 62, wherein M is set to 4, 6, 8, or 10 for Merge mode and M is set to 1, 2, or 4 for Advanced Motion Vector Prediction (AMVP) mode.

[0420] 64. The method of clause 57, wherein M depends on encoding information of the video, the encoding information comprising a block size of the first video block, a block shape of the first video block, and a slice type.

[0421] 65. The method of any of clauses 1-64, wherein the table corresponds to a lookup table.

[0422] 66. The method of any of clauses 1 to 65, further comprising updating one or more tables based on the transformation.

[0423] 67. The method of clause 66, wherein the updating of the one or more tables comprises updating the one or more tables based on motion information of the first video block after performing the conversion.

[0424] 68. The method of clause 67, further comprising performing conversion between subsequent video blocks of the video and the bitstream representation of the video based on the updated table.

[0425] 69. An apparatus in a video system comprising a processor and non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of clauses 1 to 68.

[0426] 70. A computer program product stored on a non-transitory computer readable medium, the computer program product comprising program code for carrying out the method of any of clauses 1 to 68.

[0427] From the foregoing, it will be appreciated that specific embodiments of the disclosed technology have been described herein for purposes of illustration, but that various modifications may be made without departing from the scope of the invention. Accordingly, the disclosed technology is not to be limited, except as by the appended claims.

[0428] The disclosed and other embodiments, modules, and functional operations described in this document may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or in a combination of one or more thereof. The disclosed and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium, for execution by a data processing apparatus or for controlling the operation of the data processing apparatus. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances that effect a machine-readable propagated signal, or a combination of one or more thereof. The term "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, an apparatus may include code that creates an operating environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof. A propagated signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver device.

[0429] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program need not correspond to a file in a file system. A program can be stored in the portion of a document preserving other programs or data (e.g., one or more scripts stored in a markup language document), in a single document dedicated to the program in question, or in multiple coordination documents (e.g., documents storing the portion of one or more modules, subroutines, or code). A computer program can be deployed to run on a computer or to run on multiple computers located at a site or distributed across multiple sites and interconnected by a communication network.

[0430] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0431] Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from read-only memory or random access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operatively coupled to one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, to receive data from or transfer data to, or both. However, a computer need not include such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CDROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0432] Although this patent document contains many details, these details should not be interpreted as limitations on any invention or the scope that can be claimed, but rather as descriptions of features that can be specific to a particular embodiment of a particular invention. Certain features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although the above features may be described as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be cut out from the combination in some cases, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0433] Similarly, while operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown, or in sequential order, or that all illustrated operations be performed, in order to achieve desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0434] Only a few implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A video processing method, comprising: Maintain one or more tables, where Each table corresponds to a history-based motion vector prediction (HMVP) table comprising one or more candidates derived from one or more video blocks that have been coded; constructing a candidate list for the current video block, wherein during the constructing, at least one candidate in a table of the one or more tables is checked to determine whether to add motion information from a checked candidate in the table to the candidate list, wherein the checking terminates when the number of candidates in the candidate list reaches a maximum allowed number of candidates for the candidate list minus a threshold, wherein the threshold is equal to 1 and a size of the candidate list is the maximum allowed number of candidates; Determining motion information of the current video block using the candidate list; and encoding and decoding the current video block based on the determined motion information, wherein the arrangement of the candidates in the table is based on the order in which the candidates were added to the table, The order in which the candidates are added to the table is based on the order of the codec blocks from which the candidates are derived.

2. The method according to claim 1, wherein The encoding and decoding includes encoding the current video block into a bitstream.

3. The method according to claim 1, wherein The encoding and decoding includes decoding the current video block from a bitstream.

4. The method according to claim 1, wherein The candidate list is a merge candidate list.

5. The method according to claim 1, wherein The threshold value is further dependent on at least one of a codec mode of the current video block, a block shape of the current video block, a block size of the current video block, and a number of available candidates before adding a candidate from the table.

6. The method according to claim 1, wherein The threshold is signaled in the bitstream.

7. The method according to claim 1, wherein The threshold is signaled in one or more of a video parameter set VPS, a sequence parameter set SPS, a picture parameter set PPS, a slice header, or a slice header.

8. The method according to claim 1, wherein Whether to update the candidate list based on the checked candidates is based on a checking result.

9. The method according to claim 1, wherein: At least one candidate in the table is checked in order of at least one index of the at least one candidate.

10. The method according to claim 1, wherein The candidates in the table are associated with motion information, and the motion information includes at least one of the following: prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference accuracy, motion vector difference or block position information to indicate the source of the motion information.

11. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein: The instructions, when executed by a processor, cause the processor to: Maintaining one or more tables, wherein each table corresponds to a history-based motion vector prediction (HMVP) table, the HMVP table including one or more candidates derived from one or more video blocks that have been coded; constructing a candidate list for the current video block, wherein during the constructing, at least one candidate in a table of the one or more tables is checked to determine whether to add motion information from a checked candidate in the table to the candidate list, wherein the checking terminates when the number of candidates in the candidate list reaches a maximum allowed number of candidates for the candidate list minus a threshold, wherein the threshold is equal to 1 and a size of the candidate list is the maximum allowed number of candidates; Determining motion information of the current video block using the candidate list; and encoding and decoding the current video block based on the determined motion information, wherein the arrangement of the candidates in the table is based on the order in which the candidates were added to the table, The order in which the candidates are added to the table is based on the order of the codec blocks from which the candidates are derived.

12. The device according to claim 11, wherein The encoding and decoding includes encoding the current video block into a bitstream.

13. The device according to claim 11, wherein The encoding and decoding includes decoding the current video block from a bitstream.

14. The device according to claim 11, wherein The arrangement of the candidates in each table is based on the order in which the candidates were added to the table.

15. A non-transitory computer-readable storage medium storing instructions, wherein: The instructions cause the processor to: Maintaining one or more tables, wherein each table corresponds to a history-based motion vector prediction (HMVP) table, the HMVP table including one or more candidates derived from one or more video blocks that have been coded; constructing a candidate list for the current video block, wherein during the constructing, at least one candidate in a table of the one or more tables is checked to determine whether to add motion information from a checked candidate in the table to the candidate list, wherein the checking terminates when the number of candidates in the candidate list reaches a maximum allowed number of candidates for the candidate list minus a threshold, wherein the threshold is equal to 1 and a size of the candidate list is the maximum allowed number of candidates; Determining motion information of the current video block using the candidate list; and encoding and decoding the current video block based on the determined motion information, wherein the arrangement of the candidates in the table is based on the order in which the candidates were added to the table, The order in which the candidates are added to the table is based on the order of the codec blocks from which the candidates are derived.

16. A method for storing a bitstream of a video, comprising: Maintain one or more tables, where Each table corresponds to a history-based motion vector prediction (HMVP) table comprising one or more candidates derived from one or more video blocks that have been coded; constructing a candidate list for the current video block, wherein during the constructing, at least one candidate in a table of the one or more tables is checked to determine whether to add motion information from a checked candidate in the table to the candidate list, wherein the checking terminates when the number of candidates in the candidate list reaches a maximum allowed number of candidates for the candidate list minus a threshold, wherein the threshold is equal to 1 and a size of the candidate list is the maximum allowed number of candidates; determining motion information of the current video block using the candidate list; generating the bitstream from the current video block based on the determined motion information; and storing the bitstream in a non-transitory computer-readable recording medium, wherein the arrangement of the candidates in the table is based on the order in which the candidates were added to the table, The order in which the candidates are added to the table is based on the order of the codec blocks from which the candidates are derived.

Images