Updates to lookup tables: FIFO, constrained FIFO

Abstract

A method for video processing is provided, the method involving updating a lookup table: a FIFO, a constrained FIFO, the method comprising: determining motion information of a current video block of a video; encoding and decoding the current video block based on the determined motion information; and updating a table using a new motion candidate corresponding to the determined motion information; wherein the table comprises one or more motion candidates derived from one or more video blocks that have already been encoded and decoded, and at least one motion candidate is removed from the table because the at least one motion candidate is identified as redundant with the current motion candidate through a redundancy check before the new motion candidate is added to the table, wherein the at least one motion candidate in the table is checked during the determination of motion information of a subsequent video block.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a divisional application of Chinese invention patent application No. 201910586591.8, filed on July 1, 2019, and claims priority to and the benefit of International Patent Application No. PCT / CN2018 / 093663, filed on June 29, 2018, International Patent Application No. PCT / CN2018 / 094929, filed on July 7, 2018, International Patent Application No. PCT / CN2018 / 101220, filed on August 18, 2018, International Patent Application No. PCT / CN2018 / 117627, filed on November 27, 2018, and International Patent Application No. PCT / CN2019 / 071214, filed on January 10, 2019. The entire disclosures of the above applications are incorporated by reference as part of the disclosure of this application. Technical Field

[0003] This patent document relates to video encoding and decoding technologies, devices, and systems. Background Art

[0004] Despite advances in video compression, digital video still accounts for the largest use of bandwidth on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth required for digital video usage is expected to continue to grow. Summary of the Invention

[0005] This document discloses methods, systems, and devices for encoding and decoding digital video.

[0006] In one example aspect, a video processing method is provided, comprising: maintaining one or more tables, wherein each table includes a set of motion candidates and each motion candidate is associated with corresponding motion information; performing a conversion between a current block and a bitstream representation of a video including the current block by using the motion information in the table; and updating the one or more tables based on M sets of additional motion information associated with the current block after the conversion is performed, where M is an integer.

[0007] In another example aspect, another video processing method is provided, comprising: performing a conversion between a current block and a bitstream representation of a video including the current block by using one or more tables, wherein each table includes one or more motion candidates and each motion candidate is associated with corresponding motion information; based on the conversion, updating the one or more tables based on M sets of additional motion information associated with the current block, where M is an integer.

[0008] In yet another example aspect, a video encoder device implementing the video encoding method described herein is disclosed.

[0009] In another representative aspect, the various techniques described herein can be embodied as a computer program product stored on a non-transitory computer-readable medium. The computer program product includes program code for executing the methods described herein.

[0010] In yet another representative aspect, a video decoder device may implement a method as described herein.

[0011] In another example aspect, a method for processing video data is provided, comprising: determining motion information of a current video block of a video; encoding and decoding the current video block based on the determined motion information; and updating a table using a new motion candidate corresponding to the determined motion information; wherein the table includes one or more motion candidates derived from one or more video blocks that have been encoded and decoded, and at least one motion candidate is removed from the table because the at least one motion candidate is identified as redundant with the current motion candidate through a redundancy check before the new motion candidate is added to the table, wherein the at least one motion candidate in the table is checked during determination of motion information for a subsequent video block.

[0012] In another example aspect, a video processing device is provided that includes a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: determine motion information of a current video block of a video; encode and decode the current video block based on the determined motion information; and update a table using a new motion candidate corresponding to the determined motion information; wherein the table includes one or more motion candidates derived from one or more video blocks that have been encoded and decoded, and because at least one motion candidate is identified as redundant with the current motion candidate through a redundancy check before the new motion candidate is added to the table, the at least one motion candidate is removed from the table, wherein the at least one motion candidate in the table is checked during determination of motion information for a subsequent video block.

[0013] In yet another example aspect, a non-transitory computer-readable storage medium is provided, storing instructions that cause a processor to: determine motion information of a current video block of a video; encode and decode the current video block based on the determined motion information; and update a table using a new motion candidate corresponding to the determined motion information; wherein the table includes one or more motion candidates derived from one or more video blocks that have been encoded and decoded, and at least one motion candidate is removed from the table because the at least one motion candidate is identified as redundant with the current motion candidate through a redundancy check before the new motion candidate is added to the table, wherein the at least one motion candidate in the table is checked during determination of motion information for a subsequent video block.

[0014] In yet another example aspect, a non-transitory computer-readable recording medium is provided that stores a bitstream generated by a method performed by a video processing device, wherein the method includes: determining motion information of a current video block of a video; generating the bitstream from the current video block based on the determined motion information; and updating a table using a new motion candidate corresponding to the determined motion information; wherein the table includes one or more motion candidates derived from one or more video blocks that have been encoded and decoded, and because at least one motion candidate is identified as redundant with the current motion candidate through a redundancy check before the new motion candidate is added to the table, the at least one motion candidate is removed from the table, wherein at least one motion candidate in the table is checked during determination of motion information of a subsequent video block.

[0015] The details of one or more implementations are set forth in the accompanying drawings, the accompanying figures, and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a block diagram illustrating an example of a video encoder implementation

[0017] Figure 2 The macroblock partitioning in the H.264 video coding standard is shown.

[0018] Figure 3 An example of dividing a coding block (CB) into prediction blocks is shown.

[0019] Figure 4 An example embodiment of subdividing a CTB (coding tree block) into CBs and transform blocks (TBs) is shown. Solid lines indicate CB boundaries and dashed lines indicate TB boundaries, including an example CTB with its partitioning and the corresponding quadtree.

[0020] Figure 5 An example of a Quad Tree Binary Tree (QTBT) structure for segmenting video data is shown.

[0021] Figure 6 An example of video block segmentation is shown.

[0022] Figure 7 An example of quadtree partitioning is shown.

[0023] Figure 8 An example of tree type signaling is shown.

[0024] Figure 9 An example of the derivation process for Merge candidate list construction is shown.

[0025] Figure 10 Example locations of spatial merge candidates are shown.

[0026] Figure 11 An example of candidate pairs for redundancy checking considering spatial Merge candidates is shown.

[0027] Figure 12 Examples of the location of the second PU for Nx2N and 2NxN partitions are shown.

[0028] Figure 13 Motion vector scaling of temporal merge candidates is shown.

[0029] Figure 14 A picture showing candidate positions of temporal merge candidates and their co-location.

[0030] Figure 15 An example of a combined bi-predictive Merge candidate is shown.

[0031] Figure 16 An example of a process of deriving motion vector prediction candidates is shown.

[0032] Figure 17 An example of motion vector scaling of spatial motion vector candidates is shown.

[0033] Figure 18 An example Alternative Temporal Motion Vector Prediction (ATMVP) for motion prediction of a CU (coding unit) is shown.

[0034] Figure 19 An example of identification of source blocks and source pictures is pictorially depicted.

[0035] Figure 20 An example of one CU having four subblocks and neighboring blocks is shown.

[0036] Figure 21 An example of bilateral matching is shown.

[0037] Figure 22 An example of template matching is shown.

[0038] Figure 23Depicts an example of unilateral motion estimation (ME) in frame rate up conversion (FRUC).

[0039] Figure 24 An example of DMVR based on bilateral template matching is shown.

[0040] Figure 25 An example of spatial neighboring blocks used to derive spatial Merge candidates is shown.

[0041] Figure 26 Depicted is an example of how the selection of a representative location of a lookup table is updated.

[0042] Figure 27A and Figure 27B An example of updating a lookup table with a new set of motion information is shown.

[0043] Figure 28 is a block diagram of an example of a hardware platform for implementing the visual media decoding or visual media encoding techniques described in this document.

[0044] Figure 29 is a flow chart of an example method of video bitstream processing.

[0045] Figure 30 is a flow chart of another example method of video bitstream processing.

[0046] Figure 31 An example of a decoding flow chart using the proposed HMVP method is shown

[0047] Figure 32 An example of updating a table using the proposed HMVP method is shown.

[0048] Figure 33A and Figure 33B An example of a LUT (Look Up Table) updating method based on redundancy removal (one redundant motion candidate is removed) is shown.

[0049] Figure 34A and Figure 34B An example of a redundancy removal-based LUT updating method (a plurality of redundant motion candidates are removed) is shown. DETAILED DESCRIPTION

[0050] To improve the compression ratio of video, researchers are continually searching for new techniques by which to encode video.

[0051] 1. Introduction

[0052] This document relates to video codec technology. Specifically, it relates to motion information coding (such as Merge mode and AMVP mode) in video codecs. This document can be applied to existing video codec standards such as HEVC, or to standards to be finalized (Multi-Function Video Codec). This document can also be applied to future video codec standards or video codecs.

[0053] Brief Discussion

[0054] Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T produced H.261 and H.263, ISO / IEC produced MPEG-1 and MPEG-4 Video (Visual), and the two organizations jointly produced H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Coding (AVC) and H.265 / HEVC standards. Since H.262, video codec standards have been based on a hybrid video codec structure that utilizes temporal prediction plus transform coding. Figure 1 An example of a typical HEVC encoder framework is depicted in .

[0055] 2.1 Segmentation Structure

[0056] 2.1.1 Segmentation Tree Structure in H.264 / AVC

[0057] The core of the codec layer in previous standards is the macroblock, which contains a 16×16 block of luma samples and, in the common case of 4:2:0 color sampling, two corresponding 8×8 blocks of chroma samples.

[0058] Intra-coded blocks use spatial prediction to exploit spatial correlation among pixels. Two partitions are defined: 16x16 and 4x4.

[0059] Inter-frame coding blocks use temporal prediction instead of spatial prediction by estimating the motion within the picture. Can be split for 16x16 macroblock or any of its sub-macroblocks: 16x8, 8x16, 8x8, 8x4, 4x8, 4x4 (see Figure 2 ) estimates motion independently. Only one motion vector (MV) is allowed per sub-macroblock partition.

[0060] 2.1.2 Segmentation Tree Structure in HEVC

[0061] In HEVC, CTUs are divided into CUs using a quadtree structure denoted as a codec tree to accommodate various local characteristics. The decision on whether to use inter-picture (temporal) or intra-picture (spatial) prediction to encode and decode a picture area is made at the CU level. Depending on the PU partition type, each CU can be further divided into one, two, or four PUs. Within a PU, the same prediction process is applied, and relevant information is sent to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU partition type, the CU can be partitioned into transform units (TUs) according to another quadtree structure similar to the codec tree of the CU. One of the key features of the HEVC structure is that it has multiple partitioning concepts including CU, PU, and TU.

[0062] In the following, various features involved in hybrid video codec using HEVC are highlighted as follows.

[0063] 1) Codec Tree Unit and Codec Tree Block (CTB) Structure: A similar structure in HEVC is the Codec Tree Unit (CTU), which has a size chosen by the encoder and can be larger than a traditional macroblock. A CTU consists of a luma CTB and corresponding chroma CTBs and syntax elements. The size L×L of the luma CTB can be chosen to be L=16, 32, or 64 samples, with larger sizes typically enabling better compression. HEVC then supports splitting the CTB into smaller blocks using a tree structure and quadtree-like signaling.

[0064] 2) Codec Unit (CU) and Codec Block (CB): The quadtree syntax of a CTU specifies the size and position of its luma and chroma CBs. The root of the quadtree is associated with the CTU. Therefore, the size of the luma CTB is the maximum supported size of the luma CB. The partitioning of the CTU into luma and chroma CBs is signaled jointly. One luma CB and typically two chroma CBs together with the associated syntax form a codec unit (CU). A CTB can contain only one CU or can be partitioned to form multiple CUs, and each CU has an associated partitioning into prediction units (PUs) and a tree of transform units (TUs).

[0065] 3) Prediction Unit and Prediction Block (PB): The decision on whether to use inter or intra prediction to code a picture region is made at the CU level. The PU partitioning structure has its roots at the CU level. Depending on the basic prediction type decision, the luma and chroma CBs can then be further split into different sizes and predicted from luma and chroma prediction blocks (PBs). HEVC supports variable PB sizes from 64×64 down to 4×4 samples. Figure 3 An example of allowed PBs for an MxM CU is shown.

[0066] 4) TU and transform block: The prediction residual is coded using block transform. The TU tree structure has its root at the CU level. The luma CB residual can be the same as the luma transform block (TB) or can be further divided into smaller luma TBs. The same applies to chroma TBs. Integer basis functions similar to those of the discrete cosine transform (DCT) define square TB sizes 4×4, 8×8, 16×16, and 32×32. For the 4×4 transform of the luma intra prediction residual, an integer transform derived from the form of the discrete sine transform (DST) is specified instead.

[0067] Figure 4 An example of subdividing a CTB into CBs and transform blocks (TBs) is shown. Solid lines indicate CB boundaries and dashed lines indicate TB boundaries. (a) CTB and its partitions. (b) Corresponding quadtree.

[0068] 2.1.2.1 Tree-structured partitioning into transform blocks and units

[0069] For residual codecs, the CB can be recursively partitioned into transform blocks (TBs). The partitioning is signaled by the residual quadtree. Only square CB and TB partitioning is specified, where blocks can be recursively divided into quadrants, such as Figure 4 As shown in . For a given luma CB of size M×M, a flag signals whether it is divided into four blocks of size M / 2×M / 2. If further division is possible, as signaled by the maximum depth of the residual quadtree indicated in the SPS, each quadrant is assigned a flag indicating whether it is divided into four quadrants. The leaf node blocks arising from the residual quadtree are transform blocks that are further processed by transform codecs. The encoder indicates the maximum and minimum luma TB sizes that it will use. When the CB size is larger than the maximum TB size, the division is implicit. When the division would cause the luma TB size to be smaller than the indicated minimum, no division is implicit. The chroma TB size is half the luma TB size in each dimension, except when the luma TB size is 4×4, in which case a single 4×4 chroma TB is used for the area covered by four 4×4 luma TBs. In the case of an intra-picture predicted CU, the decoded samples of the nearest neighboring TB (inside or outside the CB) are used as reference data for intra prediction.

[0070] In contrast to previous standards, the HEVC design allows a TB to span multiple PBs of a CU for inter-picture prediction to maximize the potential codec efficiency benefits of quadtree-structured TB partitioning.

[0071] 2.1.2.2 Parent and Child Nodes

[0072] The CTB is divided according to a quadtree structure, and the nodes of the quadtree structure are codec units. The multiple nodes in the quadtree structure include leaf nodes and non-leaf nodes. The leaf node does not have child nodes in the tree structure (that is, the leaf node is not further divided). The non-leaf node includes the root node of the tree structure. The root node corresponds to the initial video block of the video data (for example, CTB). For each non-root node in the multiple nodes, the non-root node corresponds to a video block, which is a child block of the video block of the parent node in the tree structure corresponding to the non-root node. Each non-leaf node in the multiple non-leaf nodes has one or more child nodes in the tree structure.

[0073] 2.1.3 Quadtree plus binary tree block structure with larger CTU in JEM

[0074] To explore future video codec technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, many new methods have been adopted by JVET and put into reference software called the Joint Exploration Model (JEM).

[0075] 2.1.3.1QTBT Block Partition Structure

[0076] Different from HEVC, the QTBT structure removes the concept of multiple partition types, that is, it removes the separation of CU, PU and TU concepts and supports more flexibility for CU partition shape. In the QTBT block structure, CU can have a square or rectangular shape. Figure 5 As shown in , the codec tree unit (CTU) is first partitioned by a quadtree structure. The quadtree leaf nodes are further partitioned by a binary tree structure. There are two types of partitioning in the binary tree partitioning, symmetric horizontal partitioning and symmetric vertical partitioning. The binary tree leaf nodes are called codec units (CUs), and this partitioning is used for prediction and transform processing without any further partitioning. This means that CU, PU and TU have the same block size in the QTBT codec block structure. In JEM, a CU sometimes consists of codec blocks (CBs) of different color components, for example, in the case of P and B slices of a 4:2:0 chroma format, one CU contains one luminance CB and two chroma CBs, and sometimes consists of CBs of a single component, for example, in the case of an I slice, one CU contains only one luminance CB or only two chroma CBs.

[0077] The following parameters are defined for the QTBT segmentation scheme.

[0078] -CTU size: the root node size of the quadtree, the same concept as in HEVC

[0079] -MinQTSize: Minimum allowed quad leaf node size

[0080] -MaxBTSize: Maximum allowed binary tree root node size

[0081] -MaxBTDepth: Maximum allowed binary tree depth

[0082] -MinBTSize: minimum allowed binary tree leaf node size

[0083] In one example of a QTBT partitioning structure, the CTU size is set to 128×128 luma samples with two corresponding 64×64 chroma sample blocks, MinQTSize is set to 16×16, MaxBTSize is set to 64×64, MinBTSize (for both width and height) is set to 4×4, and MaxBTDepth is set to 4. Quadtree partitioning is first applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can have sizes from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the leaf quadtree node is 128×128, it will not be further partitioned by the binary tree because the size exceeds MaxBTSize (i.e., 64×64). Otherwise, the leaf quadtree node can be further partitioned by the binary tree. Therefore, the quadtree leaf node is also the root node of the binary tree and has a binary tree depth of 0. When the binary tree depth reaches MaxBTDepth (i.e., 4), no further partitioning is considered. When a binary tree node has a width equal to MinBTSize (i.e., 4), no further horizontal splitting is considered. Similarly, when a binary tree node has a height equal to MinBTSize, no further vertical splitting is considered. Leaf nodes of the binary tree are further processed by prediction and transform processing without any further splitting. In JEM, the maximum CTU size is 256×256 luma samples.

[0084] Figure 5 (left) shows an example of partitioning by using QTBT blocks, and Figure 5 (Right) shows the corresponding tree representation. Solid lines indicate quadtree partitioning and dashed lines indicate binary tree partitioning. In each partition (i.e., non-leaf) node of the binary tree, a flag is signaled to indicate which partition type is used (i.e., horizontal or vertical), where 0 indicates horizontal partitioning and 1 indicates vertical partitioning. For quadtree partitioning, there is no need to indicate the partition type because quadtree partitioning always partitions the block horizontally and vertically to produce 4 sub-blocks of the same size.

[0085] In addition, the QTBT scheme supports the ability to have separate QTBT structures for luma and chroma. Currently, for P and B slices, the luma and chroma CTBs in one CTU share the same QTBT structure. However, for I slices, the luma CTB is split into CUs using the QTBT structure, and the chroma CTB is split into chroma CUs using another QTBT structure. This means that a CU in an I slice consists of either a codec block for the luma component or a codec block for two chroma components, and a CU in a P or B slice consists of codec blocks for all three color components.

[0086] In HEVC, inter-frame prediction for small blocks is restricted to reduce memory access for motion compensation, so that bidirectional prediction is not supported for 4×8 and 8×4 blocks, and inter-frame prediction is not supported for 4×4 blocks. In JEM's QTBT, these restrictions are removed.

[0087] 2.1.4 Ternary Tree for VVC

[0088] As proposed in [6], tree types other than quadtree and binary tree are supported. In the implementation, two more ternary tree (TT) partitions are introduced, namely horizontal and vertical center-side ternary trees, such as Figure 6 As shown in (d) and (e).

[0089] Figure 6 Shown: (a) quadtree partition (b) vertical binary tree partition (c) horizontal binary tree partition (d) vertical center-side ternary tree partition (e) horizontal center-side ternary tree partition.

[0090] In [6], there are two levels of trees, the region tree (quadtree) and the prediction tree (binary tree or ternary tree). The CTU is first partitioned by the region tree (RT). The RT leaf can be further partitioned by the prediction tree (PT). The PT leaf can also be further partitioned by the PT until the maximum PT depth is reached. The PT leaf is the basic coding unit. For convenience, it is still called CU. The CU cannot be further partitioned. Both prediction and transform are applied to the CU in the same way as JEM. The entire partitioning structure is called a "multi-type tree".

[0091] 2.1.5 Example of Segmentation Structure

[0092] The tree structure used in this response, called Multi-Tree Type (MTT), is a generalization of QTBT. In QTBT, Figure 5 As shown in , the codec tree unit (CTU) is first divided into a quadtree structure. The quadtree leaf nodes are further divided into a binary tree structure.

[0093] The basic structure of MTT consists of two types of tree nodes: region tree (RT) and prediction tree (PT), which supports nine types of segmentation, such as Figure 7 As shown in .

[0094] Figure 7 Shown: (a) quadtree partition (b) vertical binary tree partition (c) horizontal binary tree partition (d) vertical ternary tree partition (e) horizontal ternary tree partition (f) horizontal upper asymmetric binary tree partition (g) horizontal lower asymmetric binary tree partition (h) vertical left asymmetric binary tree partition (i) vertical right asymmetric binary tree partition.

[0095] The region tree can recursively divide the CTU into square blocks, down to the region tree leaf nodes of size 4x4. At each node in the region tree, a prediction tree can be formed from one of three tree types: binary tree (BT), ternary tree (TT), and asymmetric binary tree (ABT). In PT partitioning, it is forbidden to have quadtree splits in the branches of the prediction tree. As in JEM, the luma tree and chroma tree are separated in the I slice. The signaling method for RT and PT is in Figure 8 Shown in.

[0096] 2.2 Inter-frame prediction in HEVC / H.265

[0097] Each inter-predicted PU has motion parameters for one or two reference picture lists. The motion parameters include motion vectors and reference picture indices. The use of one of the two reference picture lists can also be signaled using inter_pred_idc. The motion vector can be explicitly encoded and decoded as a delta relative to the predictor. This encoding and decoding mode is called AMVP mode.

[0098] When a CU is coded using skip mode, a PU is associated with the CU and has no significant residual coefficients, no coded motion vector increments, or reference picture indices. Merge mode is specified whereby the motion parameters for the current PU are obtained from neighboring PUs, including spatial and temporal candidates. Merge mode can be applied to any inter-predicted PU, not only for skip mode. An alternative to Merge mode is explicit transmission of motion parameters, where for each PU, the motion vector, the corresponding reference picture index for each reference picture list, and the reference picture list used are explicitly signaled.

[0099] When signaling indicates that one of the two reference picture lists is to be used, a PU is generated from a block of samples. This is called "unidirectional prediction." Unidirectional prediction can be used for both P slices and B slices.

[0100] When signaling indicates that both of the reference picture lists are to be used, a PU is generated from both blocks of samples. This is called "bi-prediction." Bi-prediction is only available for B slices.

[0101] The following text provides details about the inter prediction modes specified in HEVC. The description will start with the Merge mode.

[0102] 2.2.1Merge Mode

[0103] 2.2.1.1 Derivation of Merge Mode Candidates

[0104] When using Merge mode to predict a PU, the index pointing to the entry in the Merge candidate list is parsed from the bitstream and used to retrieve motion information. The construction of this list is specified in the HEVC standard and can be summarized according to the following sequence of steps:

[0105] Step 1: Initial candidate derivation

[0106] oStep 1.1: Spatial Candidate Derivation

[0107] oStep 1.2: Redundancy check of spatial candidates

[0108] oStep 1.3: Temporal Candidate Derivation

[0109] Step 2: Additional candidate insertion

[0110] oStep 2.1: Creation of Bidirectional Prediction Candidates

[0111] oStep 2.2: Insertion of zero-motion candidates

[0112] These steps are also Figure 9 Schematically depicted in . For spatial Merge candidate derivation, a maximum of four Merge candidates are selected among the candidates located in five different positions. For temporal Merge candidate derivation, a maximum of one Merge candidate is selected among two candidates. Because a constant number of candidates is assumed at the decoder for each PU, additional candidates are generated when the number of candidates does not reach the maximum number of Merge candidates (Maximum Number of MergeCandidate, MaxNumMergeCand) signaled in the slice header. Because the number of candidates is constant, the index of the best Merge candidate is encoded using truncated unary binarization (TU). If the size of the CU is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

[0113] Hereinafter, operations associated with the aforementioned steps are described in detail.

[0114] 2.2.1.2 Spatial Candidate Derivation

[0115] In the derivation of spatial Merge candidates, when locating Figure 10 A maximum of four Merge candidates are selected from the candidates in the positions depicted in . The order of derivation is A1, B1, B0, A0 and B2. Position B2 is considered only if any PU at position A1, B1, B0, A0 is unavailable (for example, because it belongs to another strip or slice) or is intra-coded. After adding the candidate at position A1, the addition of the remaining candidates is subject to a redundancy check that ensures that candidates with the same motion information are excluded from the list, thereby improving the coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. On the contrary, only candidates are considered if the corresponding candidates for the redundancy check do not have the same motion information. Figure 11 The pairs linked by arrows in [ ] are added to the list only. Another source of copied motion information is a "second PU" associated with a partition other than 2Nx2N. As an example, Figure 12 The second PU is depicted for the N×2N and 2N×N cases, respectively. When the current PU is partitioned into N×2N, the candidate at position A1 is not considered for list construction. In fact, adding this candidate would result in two prediction units with the same motion information, which is redundant for having only one PU in the codec unit. Similarly, position B1 is not considered when the current PU is partitioned into 2N×N.

[0116] 2.2.1.3 Time Candidate Derivation

[0117] In this step, only one candidate is added to the list. Specifically, in the derivation of this temporal Merge candidate, the scaled motion vector is derived based on the co-located PU belonging to the picture with the smallest POC difference with the current picture in the given reference picture list. The reference picture list to be used for the derivation of the co-located PU is explicitly signaled in the slice header. Figure 13The scaled motion vector for the temporal Merge candidate is obtained as shown by the dotted line, which is scaled from the motion vector of the co-located PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal Merge candidate is set equal to zero. The actual implementation of the scaling process is described in the HEVC specification [1]. For B slices, two motion vectors are obtained, one for reference picture list 0 and the other for reference picture list 1, and the two motion vectors are combined to obtain a bi-directional prediction Merge candidate. Illustration of motion vector scaling for temporal Merge candidates.

[0118] like Figure 14 As depicted in , in a co-located PU (Y) belonging to a reference frame, the position for the temporal candidate is selected between candidates C0 and C1. If the PU at position C0 is unavailable, intra-coded, or outside the current CTU, position C1 is used. Otherwise, position C0 is used in the derivation of the temporal merge candidate.

[0119] 2.2.1.4 Additional Candidate Insertion

[0120] In addition to the spatial-temporal Merge candidate, there are two additional types of Merge candidates: combined bi-predictive Merge candidate and zero Merge candidate. Combined bi-predictive Merge candidate is generated by utilizing the spatial-temporal Merge candidate. Combined bi-predictive Merge candidate is only used for B slices. Combined bi-predictive candidate is generated by combining the first reference picture list motion parameters of the initial candidate with the second reference picture list motion parameters of another. If these two tuples provide different motion hypotheses, they will form a new bi-predictive candidate. As an example, Figure 15 Depicted is the case when two candidates in the original list (on the left) are used to create combined bi-predictive Merge candidates that are added to the final list (on the right). There are many rules about the combinations that are considered to generate these additional Merge candidates.

[0121] Zero motion candidates are inserted to fill the remaining entries in the Merge candidate list and thus hit the MaxNumMergeCand capacity. These candidates have zero spatial displacement and reference picture index, which starts at zero and increases each time a new zero motion candidate is added to the list. The number of reference frames used by these candidates is one and two for unidirectional prediction and bidirectional prediction, respectively. Finally, no redundancy check is performed on these candidates.

[0122] 2.2.1.5 Motion Estimation Regions for Parallel Processing

[0123] To speed up the encoding process, motion estimation can be performed in parallel, thereby deriving motion vectors for all prediction units within a given region simultaneously. Deriving Merge candidates from spatial neighborhoods may interfere with parallel processing because a prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is completed. To alleviate the trade-off between coding efficiency and processing delay, HEVC defines a motion estimation region (MER), the size of which is signaled in the picture parameter set using the "log2_parallel_merge_level_minus2" syntax element. When MER is defined, Merge candidates falling in the same region are marked as unavailable and are therefore not considered in list construction.

[0124] 7.3.2.3 Picture Parameter Set RBSP Syntax

[0125] 7.3.2.3.1 General Picture Parameter Set RBSP Syntax

[0126]

[0127] log2_parallel_merge_level_minus2 plus 2 specifies the value of the variable Log2ParMrgLevel, which is used in the derivation of luma motion vectors for Merge mode as specified in 8.5.3.2.2 and in the derivation of spatial Merge candidates as specified in 8.5.3.2.3. The value of log2_parallel_merge_level_minus2 shall be in the range of 0 to CtbLog2SizeY-2, inclusive.

[0128] The variable Log2ParMrgLevel is derived as follows:

[0129] Log2ParMrgLevel=log2_parallel_merge_level_minus2+2 (7-37)

[0130] NOTE 3 - The value of Log2ParMrgLevel indicates the built-in capability of parallel derivation of Merge candidate lists. For example, when Log2ParMrgLevel is equal to 6, the Merge candidate lists of all prediction units (PUs) and coding units (CUs) contained in a 64x64 block can be derived in parallel.

[0131] 2.2.2 Motion Vector Prediction in AMVP Mode

[0132] Motion vector prediction exploits the spatial-temporal correlation of motion vectors with neighboring PUs, which is used for explicit transmission of motion parameters. It builds a motion vector candidate list by first checking the availability of PU positions that are temporally adjacent to the left and above, removing redundant candidates, and adding zero vectors to make the candidate list a constant length. The encoder can then select the best prediction value from the candidate list and transmit the corresponding index indicating the selected candidate. Similar to Merge index signaling, the index of the best motion vector candidate is encoded using truncated unary. The maximum value to be encoded in this case is 2 (e.g., Figures 2 to 8 ). In the following sections, details on the derivation process of motion vector prediction candidates are provided.

[0133] 2.2.2.1 Derivation of Motion Vector Prediction Candidates

[0134] Figure 16 The derivation process of motion vector prediction candidates is summarized.

[0135] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For spatial motion vector candidate derivation, as Figure 11 As depicted in , two motion vector candidates are ultimately derived based on the motion vector of each PU located in five different positions.

[0136] For temporal motion vector candidate derivation, a motion vector candidate is selected from two candidates derived based on two different co-located positions. After making the first list of spatio-temporal candidates, duplicate motion vector candidates are removed from the list. If the number of potential candidates is greater than two, motion vector candidates whose reference picture index within the associated reference picture list is greater than 1 are removed from the list. If the number of spatio-temporal motion vector candidates is less than two, an additional zero motion vector candidate is added to the list.

[0137] 2.2.2.2 Spatial Motion Vector Candidates

[0138] In the derivation of spatial motion vector candidates, a maximum of two candidates are considered among five potential candidates, which are located at positions such as Figure 11The PU derivation in the positions depicted in the figure is identical to those of the motion merge. The order of derivation for the left side of the current PU is defined as A0, A1, and scaled A0, scaled A1. The order of derivation for the top side of the current PU is defined as B0, B1, B2, scaled B0, scaled B1, scaled B2. For each side, there are therefore four possible cases that can be used as motion vector candidates, two of which do not require the use of spatial scaling, and two of which use spatial scaling. The four different cases are summarized below.

[0139] No spatial scaling

[0140] -(1) Same reference picture list and same reference picture index (same POC)

[0141] -(2) Different reference picture lists, but same reference picture (same POC)

[0142] Spatial scaling

[0143] -(3) Same reference picture list, but different reference pictures (different POC)

[0144] -(4) Different reference picture lists and different reference pictures (different POC)

[0145] The no spatial scaling case is checked first, followed by spatial scaling. When the POC is different between the reference pictures of the neighboring PU and the reference picture of the current PU, spatial scaling is considered regardless of the reference picture list.

[0146] If all PUs of the left candidate are unavailable or intra-coded, the upper motion is allowed.

[0147] Scaling of the vectors is to help parallel derivation of left and top MV candidates. Otherwise, spatial scaling is not allowed for the top motion vector.

[0148] During spatial scaling, Figure 17 As depicted in , the motion vectors of neighboring PUs are scaled in a similar manner as for time scaling. The main difference is that the reference picture list and the index of the current PU are given as input; the actual scaling process is the same as that for time scaling.

[0149] 2.2.2.3 Temporal Motion Vector Candidates

[0150] Except for the reference picture index derivation, all the processes for the derivation of temporal merge candidates are the same as those for the derivation of spatial motion vector candidates (see e.g. Figure 6 ). The reference picture index is signaled to the decoder.

[0151] 2.2.2.4 Signaling Notification of AMVP Information

[0152] For the AMVP mode, four parts may be signaled in the bitstream, namely, prediction direction, reference index, MVD (Motion Vector Difference) and mv prediction value candidate index.

[0153] Syntax table:

[0154]

[0155] 7.3.8.9 Motion Vector Difference Syntax

[0156]

[0157] 2.3 New Inter-frame Prediction Method in JEM (Joint Exploration Model)

[0158] 2.3.1 Motion Vector Prediction Based on Sub-CU

[0159] In JEM with QTBT, each CU can have at most one motion parameter set for each prediction direction. Two sub-CU level motion vector prediction methods are considered in the encoder by dividing the large CU into sub-CUs and deriving motion information for all sub-CUs of the large CU. The optional temporal motion vector prediction (ATMVP) method allows each CU to extract multiple sets of motion information from multiple blocks smaller than the current CU in the collocated reference picture. In the Spatial-Temporal Motion Vector Prediction (STMVP) method, the motion vector of the sub-CU is recursively derived by using the temporal motion vector predictor and the spatial neighboring motion vectors.

[0160] In order to preserve a more accurate motion field for sub-CU motion prediction, motion compression on reference frames is currently disabled.

[0161] 2.3.1.1 Optional Temporal Motion Vector Prediction

[0162] In the optional temporal motion vector prediction (ATMVP) method, the motion vector is modified by extracting multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU. Figure 18 As shown in , a sub-CU is a square N×N block (N is set to 4 by default).

[0163] ATMVP predicts the motion vectors of sub-CUs within a CU in two steps. The first step is to identify the corresponding blocks in the reference picture using the so-called temporal vector. The reference picture is called the motion source picture. The second step is to divide the current CU into sub-CUs and obtain the motion vectors and reference indexes of each sub-CU from the blocks corresponding to each sub-CU, such as Figure 18 shown.

[0164] In the first step, the reference picture and the corresponding block are determined by the motion information of the spatial neighboring blocks of the current CU. In order to avoid repeated scanning of neighboring blocks, the first Merge candidate in the Merge candidate list of the current CU is used. The first available motion vector and its associated reference index are set to the time vector and the index to the motion source picture. In this way, in ATMVP, the corresponding blocks can be identified more accurately than in TMVP, where the corresponding blocks (sometimes referred to as co-located blocks) are always in the lower right or center position relative to the current CU. In one example, if the first Merge candidate comes from the left neighboring block (i.e., Figure 19 A1) in , the associated MV and reference picture are used to identify the source block and source picture.

[0165] Figure 19 An example of identification of source blocks and source pictures is shown.

[0166] In the second step, the corresponding block of the sub-CU is identified by the time vector in the motion source picture by adding the time vector to the coordinates of the current CU. For each sub-CU, the motion information of its corresponding block (the minimum motion grid covering the center sample) is used to derive the motion information of the sub-CU. After the motion information of the corresponding N×N block is identified, it is converted into the motion vector and reference index of the current sub-CU in the same way as the TMVP of HEVC, where motion scaling and other processes apply. For example, the decoder checks whether the low latency condition is met (i.e., the POC of all reference pictures of the current picture is less than the POC of the current picture) and may use the motion vector MV x (corresponding to the motion vector of reference picture list X) to predict the motion vector MV of each sub-CU y (where X is equal to 0 or 1, and Y is equal to 1-X).

[0167] 2.3.1.2 Space-time motion vector prediction

[0168] In this method, the motion vector of a sub-CU is recursively derived following the raster scan order. Figure 20 This concept is illustrated in . Let us consider an 8x8 CU containing four 4x4 sub-CUs A, B, C, and D. The neighboring 4x4 blocks in the current frame are labeled a, b, c, and d.

[0169] The motion derivation of sub-CU A starts by identifying its two spatial neighbors. The first neighbor is the N×N block (block c) on the upper side of the sub-CU A. If block c is unavailable or intra-coded, check the other N×N blocks on the upper side of the sub-CU A (from left to right, starting from block c). The second neighbor is the block (block b) on the left side of the sub-CU A. If block b is unavailable or intra-coded, check the other blocks on the left side of the sub-CU A (from top to bottom, starting from block b). The motion information obtained from the adjacent blocks of each list is scaled to the first reference frame of the given list. Next, the temporal motion vector predictor (TMVP) of sub-block A is derived by following the same process as the TMVP derivation specified in HEVC. The motion information of the co-located blocks at position D is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors (up to 3) are averaged for each reference list respectively. The averaged motion vector is assigned as the motion vector of the current sub-CU.

[0170] Figure 20 An example of one CU having four sub-blocks (AD) and their neighboring blocks (ad) is shown.

[0171] 2.3.1.3 Sub-CU Motion Prediction Mode Signaling

[0172] The sub-CU mode is enabled as an additional Merge candidate, and no additional syntax elements are required to signal the mode. Two additional Merge candidates are added to the Merge candidate list of each CU to represent ATMVP mode and STMVP mode. If the sequence parameter set indicates that ATMVP and STMVP are enabled, up to seven Merge candidates are used. The encoding logic of the additional Merge candidates is the same as that used for the Merge candidates in HM, which means that for each CU in a P or B slice, two more RD checks are required for the two additional Merge candidates.

[0173] In JEM, all bins of the Merge index are coded by CABAC context, whereas in HEVC, only the first bin is context coded and the remaining bins are context bypass coded.

[0174] 2.3.2 Adaptive Motion Vector Difference Resolution

[0175] In HEVC, when use_integer_mv_flag in the slice header is equal to 0, the motion vector difference (MVD) (between the PU's motion vector and the predicted motion vector) is signaled in units of quarter luma samples. In JEM, Locally Adaptive Motion Vector Resolution (LAMVR) is introduced. In JEM, MVD can be encoded or decoded in units of quarter luma samples, integer luma samples, or four luma samples. The MVD resolution is controlled at the codec unit (CU) level, and the MVD resolution flag is conditionally signaled for each CU with at least one non-zero MVD component.

[0176] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter luma sample MV precision is not used, another flag is signaled to indicate whether integer luma sample MV precision or four luma sample MV precision is used.

[0177] When the first MVD resolution flag of a CU is zero or not coded for the CU (meaning all MVDs in the CU are zero), a quarter luma sample MV resolution is used for the CU. When the CU uses integer luma sample MV precision or four luma sample MV precision, the MVP in the AMVP candidate list for the CU is rounded to the corresponding precision.

[0178] In the encoder, CU-level RD check is used to determine which MVD resolution to use for a CU. That is, three CU-level RD checks are performed for each MVD resolution. To speed up the encoder, the following encoding scheme is applied in JEM.

[0179] During RD check of a CU with normal quarter luma sample MVD resolution, the motion information of the current CU (integer luma sample precision) is stored. The stored (after rounding) motion information is used as a starting point for further small-scale motion vector refinement during RD check of the same CU with integer luma samples and 4 luma sample MVD resolution, so that the time-consuming motion estimation process is not repeated three times.

[0180] Conditionally call RD check for CUs with 4 luma sample MVD resolution. For a CU, skip RD check for 4 luma sample MVD resolution of the CU when the RD cost integer luma sample MVD resolution is much larger than the resolution of quarter luma sample MVD.

[0181] 2.3.3 Motion Vector Derivation by Pattern Matching

[0182] The Pattern Matched Motion Vector Derivation (PMMVD) mode is a special Merge mode based on the Frame-Rate Up Conversion (FRUC) technology. In this mode, the motion information of the block is not signaled but derived at the decoder side.

[0183] When the Merge flag of a CU is true, the FRUC flag is signaled for the CU. When the FRUC flag is false, the Merge index is signaled and the normal Merge mode is used. When the FRUC flag is true, an additional FRUC mode flag is signaled to indicate which method (bilateral matching or template matching) to use to derive the motion information of the block.

[0184] At the encoder side, the decision on whether to use FRUC Merge mode for a CU is based on RD cost selection, just like for normal merge candidates. That is, two matching modes (bilateral matching and template matching) are checked for the CU using RD cost selection. The mode that results in the minimum cost is further compared with the other CU modes. If the FRUC matching mode is the most efficient mode, the FRUC flag is set to true for the CU, and the relevant matching mode is used.

[0185] The motion derivation process in FRUC Merge mode has two steps. First, a CU-level motion search is performed, followed by sub-CU-level motion refinement. At the CU level, an initial motion vector is derived for the entire CU based on bilateral matching or template matching. First, a list of MV candidates is generated, and the candidate that results in the minimum matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bilateral matching or template matching is performed around the starting point, and the MV resulting in the minimum matching cost is adopted as the MV for the entire CU. Subsequently, the motion information is further refined at the sub-CU level, using the derived CU motion vector as the starting point.

[0186] For example, the following derivation process is performed for the W×H CU motion information derivation. In the first stage, the MV of the entire W×H CU is derived. In the second stage, the CU is further divided into M×M sub-CUs. The value of M is calculated as in (16), where D is the predefined partition depth, which is set to 3 by default in JEM. The MV of each sub-CU is then derived.

[0187]

[0188] like Figure 21As shown in , bilateral matching is used to derive the motion information of the current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. Under the assumption of continuous motion trajectory, the motion vectors MV0 and MV1 pointing to the two reference blocks should be proportional to the temporal distance between the current picture and the two reference pictures (i.e., TD0 and TD1). As a special case, when the current picture is temporally between the two reference pictures and the temporal distance from the current picture to the two reference pictures is the same, bilateral matching becomes mirror-based bidirectional MV.

[0189] like Figure 22 As shown in , template matching is used to derive the motion information of the current CU by finding the closest match between the template in the current picture (the top and / or left neighboring blocks of the current CU) and the block (of the same size as the template) in the reference picture. In addition to the aforementioned FRUC Merge mode, template matching is also applicable to AMVP mode. In JEM, as done in HEVC, AMVP has two candidates. Using the template matching method, a new candidate is derived. If the candidate newly derived by template matching is different from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and then the list size is set to 2 (meaning removing the second existing AMVP candidate). When applied to AMVP mode, only CU level search is applied.

[0190] 2.3.3.1 CU-level MV candidate set

[0191] The MV candidate set at the CU level consists of the following:

[0192] (i) If the current CU is in AMVP mode, the original AMVP candidate

[0193] (ii) All Merge candidates,

[0194] (iii) Several MVs in the interpolated MV field.

[0195] (iv) Top and left neighboring motion vectors

[0196] When bilateral matching is used, each valid MV of the Merge candidate is used as input to generate an MV pair under the assumption of bilateral matching. For example, one valid MV of the Merge candidate is (MVa, refa) at reference list A. Then, a reference picture refb of its paired bilateral MV is found in other reference list B, so that refa and refb are on different sides of the current picture in time. If such refb is not available in reference list B, refb is determined as a reference different from refa, and its temporal distance to the current picture is the smallest one in list B. After determining refb, MVb is derived by scaling MVa based on the temporal distance between the current picture and refa and refb.

[0197] The four MVs from the interpolated MV field are also added to the CU level candidate list. More specifically, the interpolated MVs at positions (0, 0), (W / 2, 0), (0, H / 2), and (W / 2, H / 2) of the current CU are added.

[0198] When FRUC is applied in AMVP mode, the original AMVP candidates are also added to the CU-level MV candidate set.

[0199] At the CU level, up to 15 MVs of an AMVP CU and up to 13 MVs of a Merge CU are added to the candidate list.

[0200] 2.3.3.2 Sub-CU Level MV Candidate Set

[0201] The MV candidate set at the sub-CU level consists of the following:

[0202] (i) Search the determined MV from the CU level,

[0203] (ii) top, left, upper-left, and upper-right adjacent MVs,

[0204] (iii) a scaled version of the co-located MV from the reference picture,

[0205] (iv) Up to 4 ATMVP candidates,

[0206] (v) Up to 4 STMVP candidates

[0207] The scaled MV from the reference picture is derived as follows: Traverse all reference pictures in both lists. The MV at the collocated position of the sub-CU in the reference picture is scaled to the reference of the starting CU level MV.

[0208] ATMVP and STMVP candidates are limited to the top four.

[0209] At the sub-CU level, up to 17 MVs are added to the candidate list.

[0210] 2.3.3.3 Generation of interpolated MV fields

[0211] Before encoding or decoding a frame, an interpolated motion field is generated for the entire picture based on unilateral ME. The motion field can then be used later as a CU-level or sub-CU-level MV candidate.

[0212] First, the motion fields of each reference picture in both reference lists are traversed at the 4×4 block level. For each 4×4 block, if the motion associated with the block crosses a 4×4 block in the current picture (e.g. Figure 23 ) and the block has not been assigned any interpolated motion, the motion of the reference block is scaled to the current picture according to the temporal distances TD0 and TD1 (in the same way as the MV scaling of TMVP in HEVC), and the scaled motion is assigned to the block in the current frame. If the scaled MV is not assigned to the 4×4 block, the motion of the block is marked as unavailable in the interpolated motion field.

[0213] 2.3.3.4 Interpolation and Matching Costs

[0214] When motion vectors point to fractional sample locations, motion compensated interpolation is required. To reduce complexity, bilinear interpolation is used instead of conventional 8-tap HEVC interpolation for bilateral matching and template matching.

[0215] The calculation of the matching cost is slightly different at different steps. When selecting a candidate from the candidate set at the CU level, the matching cost is the absolute sum difference (SAD) of bilateral matching or template matching. After determining the starting MV, the matching cost C of bilateral matching at the sub-CU level search is calculated as follows.

[0216]

[0217] where w is a weighting factor empirically set to 4, MV and MV s Indicates the current MV and the start MV respectively. SAD is still used as the matching cost of template matching at the sub-CU level search.

[0218] In FRUC mode, MV is derived by using only luma samples. The derived motion will be used for both luma and chroma for MC inter-frame prediction. After deciding the MV, the final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma.

[0219] 2.3.3.5MV Refinement

[0220] MV refinement is a pattern-based MV search with criteria of bilateral matching cost or template matching cost. In JEM, two search modes are supported - respectively, Unrestricted Center-Biased Diamond Search (UCBDS) and Adaptive Cross Search for MV refinement at CU level and sub-CU level. For both CU and sub-CU level MV refinement, the MV is searched directly with quarter luma sample MV precision, followed by eighth luma sample MV refinement. The search range of MV refinement for CU and sub-CU steps is set to equal 8 luma samples.

[0221] 2.3.3.6 Prediction Direction Selection in Template Matching FRUC Merge Mode

[0222] In bilateral matching Merge mode, bidirectional prediction is always applied because the motion information of the CU is derived based on the closest match between two blocks in two different reference pictures along the motion trajectory of the current CU. There is no such restriction for template matching Merge mode. In template matching Merge mode, the encoder can choose between unidirectional prediction according to list0, unidirectional prediction according to list1, or bidirectional prediction of the CU. The selection is based on the template matching cost as follows:

[0223] If costBi<=factor*min(cost0,cost1)

[0224] Use bidirectional prediction;

[0225] Otherwise, if cost0 <= cost1

[0226] Use one-way prediction based on list0;

[0227] otherwise,

[0228] Use one-way prediction based on list1;

[0229] Where cost0 is the SAD of list0 template matching, cost1 is the SAD of list1 template matching, and costBi is the SAD of bidirectional prediction template matching. The value of factor is equal to 1.25, which means that the selection process is biased towards bidirectional prediction.

[0230] Inter prediction direction selection is only applied to the CU level template matching process.

[0231] 2.3.4 Decoder-side Motion Vector Refinement

[0232] In bidirectional prediction, for the prediction of a block region, two prediction blocks, formed using motion vectors (MVs) from list0 and list1, are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of the bidirectional prediction are further refined through a bilateral template matching process. Bilateral template matching is applied in the decoder to perform a distortion-based search between the bilateral template and the reconstructed samples in the reference picture to obtain the refined MVs without transmitting additional motion information.

[0233] In DMVR, such as Figure 23 As shown in , a bilateral template is generated from the initial MV0 of list0 and MV1 of list1 as a weighted combination (i.e., average) of the two prediction blocks, respectively. The template matching operation consists of calculating a cost measure between the generated template and the sample area in the reference picture (around the initial prediction block). For each of the two reference pictures, the MV that induces the minimum template cost is considered as the updated MV of the list to replace the original MV. In JEM, nine MV candidates are searched for each list. The nine MV candidates include the original MV and 8 surrounding MVs that are offset by one luminance sample in the horizontal or vertical direction, or in both. Finally, the two new MVs, i.e. Figure 24 MV0' and MV1' shown in are used to generate the final bidirectional prediction result. The sum of absolute differences (SAD) is used as the cost metric.

[0234] DMVR is applied to the Merge mode of bi-prediction, where one MV comes from a past reference picture and the other MV comes from a future reference picture, without the transmission of additional syntax elements. In JEM, DMVR is not applied when LIC, affine motion, FRUC, or sub-CU Merge candidate is enabled for a CU.

[0235] 2.3.5 Merge / Skip Mode with Bilateral Match Refinement

[0236] The Merge candidate list is first constructed by inserting the motion vectors and reference indices of spatially and temporally adjacent blocks into the candidate list using redundancy check until the number of available candidates reaches the maximum candidate size of 19. The spatial candidates are inserted according to a predefined insertion order ( Figure 11 ), temporal candidates, affine candidates, advanced temporal MVP (ATMVP) candidates, spatial temporal MVP (STMVP) candidates, and additional candidates (combined candidates and zero candidates) as used in HEVC to construct the candidate list for Merge / Skip mode:

[0237] - Spatial candidates for blocks 1-4.

[0238] - Presumed affine candidates for blocks 1-4.

[0239] -ATMVP.

[0240] -STMVP.

[0241] -Virtual affine candidate.

[0242] - Spatial candidates (block 5) (used only when the number of available candidates is less than 6).

[0243] - Inferred affine candidates (block 5).

[0244] - Temporal candidates (as derived in HEVC).

[0245] - non-adjacent spatial candidates, followed by the inferred affine candidates (blocks 6 to 49, e.g. Figure 25 as depicted in ).

[0246] -Candidates for combination.

[0247] -Zero Candidates

[0248] Note that in addition to STMVP and Affine, the IC flag is also inherited from the Merge candidate. In addition, for the first four spatial candidates, bidirectional prediction candidates are inserted before candidates with unidirectional prediction.

[0249] In [8], blocks that are not connected to the current block can be accessed. If non-neighboring blocks are encoded in non-intra mode, the associated motion information can be added as additional Merge candidates.

[0250] 3. Examples of Problems Solved by the Embodiments Disclosed herein

[0251] The current HEVC design can exploit the correlation between the current block and its neighboring blocks (immediately adjacent to the current block) to better encode and decode motion information. However, it is possible that the neighboring blocks correspond to different objects with different motion trajectories. In this case, the prediction from its neighboring blocks is not effective.

[0252] Prediction of motion information from non-neighboring blocks can bring additional codec gains, but this comes at the cost of storing all motion information (typically on a 4x4 level) into cache, which significantly increases the complexity of hardware implementation.

[0253] 4. Some Examples

[0254] Embodiments of the presently disclosed technology overcome the shortcomings of existing implementations, thereby providing video codecs with higher codec efficiency.

[0255] To overcome the shortcomings of existing implementations, a LUT-based motion vector prediction technique that uses one or more tables (e.g., lookup tables) with at least one stored motion candidate to predict motion information of a block can be implemented in various embodiments to provide video codecs with higher codec efficiency. A lookup table is an example of a table that can be used to include motion candidates to predict motion information of a block, and other implementations are also possible. Each LUT can include one or more motion candidates, each motion candidate being associated with corresponding motion information. The motion information of a motion candidate can include some or all of a prediction direction, a reference index / picture, a motion vector, a LIC flag, an affine flag, a motion vector derivation (MVD) accuracy, and / or an MVD value. The motion information can also include block position information to indicate where the motion information is coming from.

[0256] LUT-based motion vector prediction based on the disclosed technology, which can enhance both existing and future video codec standards, is illustrated in the following examples described in relation to various embodiments. Because the LUT allows the encoding / decoding process to be performed based on historical data (e.g., blocks that have already been processed), LUT-based motion vector prediction can also be referred to as a history-based motion vector prediction (HMVP) method. In the LUT-based motion vector prediction method, one or more tables containing motion information from previously encoded and decoded blocks are maintained during the encoding / decoding process. These motion candidates stored in the LUT are named HMVP candidates. During the encoding / decoding of a block, the associated motion information in the LUT can be added to a motion candidate list (e.g., a Merge / AMVP candidate list), and after encoding / decoding a block, the LUT can be updated. The updated LUT is then used to encode and decode subsequent blocks. In other words, the updating of motion candidates in the LUT is based on the encoding / decoding order of the blocks. The following examples should be considered as examples for explaining general concepts. These examples should not be interpreted in a narrow sense. In addition, these examples can be combined in any way.

[0257] The following examples should be considered as examples used to explain the general concepts. They should not be interpreted in a narrow sense. Furthermore, these examples can be combined in any way.

[0258] Some embodiments may use one or more lookup tables with at least one stored motion candidate to predict motion information for a block. Embodiments may use motion candidates to indicate a set of motion information stored in a lookup table. For traditional AMVP or Merge modes, embodiments may use AMVP or Merge candidates to store motion information.

[0259] The following example explains the general concept.

[0260] Example of a table

[0261] Example A1: Each lookup table may contain one or more motion candidates, where each candidate is associated with its motion information.

[0262] a. The motion information of the motion candidate here may include part or all of the prediction direction, reference index / picture, motion vector, LIC flag, affine flag, MVD accuracy, and MVD value.

[0263] b. Motion information may also include block position information and / or block shape to indicate where the motion information is coming from.

[0264] c. A counter may also be assigned to each lookup table.

[0265] d. The table size (number of allowed motion candidates) and / or the number of tables can be fixed or adaptive. The table size can be the same for all tables or different for different tables.

[0266] LUT Selection

[0267] Example B1: For encoding a block, some or all motion candidates from a lookup table may be checked in order. When a motion candidate is checked during encoding a block, it may be added to a motion candidate list (eg, AMVP, Merge candidate list).

[0268] Use of lookup tables

[0269] Example C1: The total number of motion candidates to be checked in the lookup table may be predefined.

[0270] Example C2: The motion candidate(s) included in the lookup table can be directly inherited by the block.

[0271] a. They can be used for Merge mode encoding and decoding, that is, motion candidates can be checked during the Merge candidate list derivation process.

[0272] Example C3: The motion candidate(s) included in the lookup table may be used as a predictor for the motion information of a codec block.

[0273] a. They can be used for AMVP mode encoding and decoding, i.e., motion candidates can be checked during the AMVP candidate list derivation process.

[0274] Example C4: The order of checking motion candidates in the lookup table is defined as follows (assuming that K (K>=1) motion candidates are allowed to be checked):

[0275] a. Look up the last K motion candidates in the table, eg, in descending order of the entry index of the LUT.

[0276] Example C5: Enabling / disabling the use of a lookup table for motion information coding of a block may be signaled in an SPS, PPS, slice header, slice header, CTU, CTB, CU or PU, or an area covering multiple CTUs / CTBs / CUs / PUs.

[0277] Example C6: Whether to apply prediction based on a lookup table can also depend on codec information. When it is inferred that it is not applicable to the block, additional signaling of the prediction indication is skipped. Alternatively, when it is inferred that it is not applicable to the block, the motion candidates of the lookup table do not need to be accessed, and the checking of relevant motion candidates is omitted.

[0278] Example C7: Motion candidates from a lookup table in a previously coded frame / slice / slice may be used to predict motion information for a block in a different frame / slice / slice.

[0279] Update of the lookup table

[0280] Example D1: After encoding a block with motion information (ie, IntraBC mode, Inter coding mode), one or more lookup tables may be updated.

[0281] In one example, whether to update the lookup table can reuse the rules for selecting the lookup table. For example, when a lookup table can be selected for encoding / decoding the current block, the selected lookup table can also be updated after encoding / decoding the block.

[0282] b. The lookup table to be updated may be selected based on codec information and / or the location of the block / LCU.

[0283] c. If the block is coded with directly signaled motion information (such as AMVP mode, MMVD mode for normal / affine inter mode, AMVR mode for normal / affine inter mode), the motion information for the block may be added to the lookup table.

[0284] i. Alternatively, if the block is coded with motion information inherited directly from spatially neighboring blocks without any refinement (eg, no refined spatial Merge candidates), the motion information for the block should not be added to the lookup table.

[0285] ii. Alternatively, if the block is coded with motion information directly inherited from spatially neighboring blocks with refinement (such as DMVR, FRUC), the motion information for the block should not be added to any lookup table.

[0286] iii. Alternatively, if the block is coded directly with motion information inherited from motion candidates stored in a lookup table, the motion information for the block should not be added to any lookup table.

[0287] iv. In one example, such motion information may be added directly to the lookup table, such as to the last entry of the table or the entry storing the next available motion candidate.

[0288] v. Alternatively, such motion information may be added directly to the lookup table without pruning (eg, without any pruning).

[0289] vi. Alternatively, such motion information may be used to reorder the lookup table.

[0290] Alternatively, with limited pruning, such motion information can be used to update a lookup table. In various embodiments, pruning can include: a) comparing the motion information with existing entries for uniqueness, b) if unique, adding the motion information to the list, c) if not unique, c1) not adding, or c2) adding the motion information and deleting the matching existing entry.

[0291] d. Select M (M>=1) representative positions within the block and use the motion information associated with the representative positions to update the lookup table.

[0292] i. In one example, representative locations are defined as the four corner locations within a block (e.g., Figure 26 One of C0-C3).

[0293] ii. In one example, the representative position is defined as the center position within the block (e.g., Figure 26 Ca-Cd in the

[0294] iii. When sub-block prediction is not allowed for a block, M is set to 1.

[0295] iv. When sub-block prediction is allowed for a block, M can be set to 1 or the total number of sub-blocks or any other value between [1, number of sub-blocks].

[0296] v. Alternatively, when sub-block prediction is allowed for a block, M can be set to 1 and the selection of the representative sub-block is based on

[0297] 1. The frequency of motion information used,

[0298] 2. Is it a bidirectional prediction block?

[0299] 3. Based on reference image index / reference image

[0300] 4. Motion vector difference compared to other motion vectors (e.g., select the largest MV difference)

[0301] 5. Other codec information.

[0302] e. When selecting M (M>=1) sets of representative positions to update the lookup table, other conditions may also be checked before adding them as additional motion candidates to the lookup table.

[0303] i. The new set of motion information can be pruned against existing motion candidates in the lookup table.

[0304] ii. In one example, the new set of motion information should not be identical to any or part of the existing motion candidates in the lookup table.

[0305] iii. Alternatively, for the same reference picture from the new set of motion information and an existing motion candidate, the MV difference should be no less than one / multiple thresholds. For example, the horizontal and / or vertical components of the MV difference should be greater than 1 pixel distance.

[0306] iv. Alternatively, when K>L, the new set of motion information is pruned with only the last K candidates or the first K%L existing motion candidates to allow reactivation of old motion candidates.

[0307] v. Alternatively, no pruning is applied.

[0308] f. If M sets of motion information are used to update the lookup table, the corresponding counter should be increased by M.

[0309] g. Assume that before encoding the current block, the counter of the lookup table to be updated is marked by K. After encoding the block, for a selected set of motion information (using the above method), it is added as an additional motion candidate with an index equal to K%L (where L is the lookup table size). Figure 27A and Figure 27B Shown in.

[0310] i. Alternatively, add it as an additional motion candidate with index equal to min(K+1, L-1). Alternatively, furthermore, if K>=L, remove the first motion candidate (index equal to 0) from the lookup table and decrement the following K candidate indices by 1.

[0311] ii. For the above two methods (adding new motion candidates to entries with index equal to K%L or adding them with index equal to min(K+1, L-1)), they try to keep the most recent sets of motion information from previously coded blocks, regardless of whether the same / similar motion candidates exist.

[0312] iii. Alternatively, when a new set of motion information is added to the LUT as a motion candidate, a redundancy check is first applied. In this case, the LUT will hold the most recent sets of motion information from previously coded blocks, but any redundant ones can be removed from the LUT. This approach is known as a redundancy removal-based LUT update method.

[0313] 1. If there are redundant motion candidates in the LUT, the counter associated with the LUT may not be incremented or decremented.

[0314] 2. Redundancy check can be defined as a pruning process during the Merge candidate list construction process, for example, checking whether the reference pictures / reference picture indices are the same and whether the motion vector differences are within the range or the same.

[0315] 3. If a redundant motion candidate is found in the LUT, the redundant motion candidate is moved from its current position to the last one in the LUT.

[0316] a. Similarly, if a redundant motion candidate is found in the LUT, it is removed from the LUT. In addition, all motion candidates inserted after the redundant motion candidate in the LUT are shifted forward to refill the entry where the redundant motion candidate was removed. After the shift, the new motion candidate is added to the LUT.

[0317] b. In this case, the counter remains unchanged.

[0318] c. Once a redundant motion candidate is identified in the LUT, the redundancy checking process is terminated.

[0319] 4. Multiple redundant motion candidates may be identified. In this case, all redundant candidates are removed from the LUT. Additionally, all remaining motion candidates may be moved forward in order.

[0320] a. In this case, the counter is decremented (the number of redundant motion candidates minus 1).

[0321] b. Terminate the redundancy check process after identifying maxR redundant motion candidates (maxR is a positive integer variable).

[0322] 5. The redundancy checking process may start from the first motion candidate to the last motion candidate (ie, in the order of addition to the LUT, in the order of the decoding process of the blocks the motion information comes from).

[0323] 6. Alternatively, when redundant motion candidates exist in the LUT, instead of removing one or more redundant motion candidates from the LUT, virtual motion candidates may be derived from the redundant motion candidates and the virtual motion candidates may be used to replace the redundant motion candidates.

[0324] a. A virtual motion candidate can be derived from a redundant motion candidate by adding (multiple) offsets to the horizontal and / or vertical components of one or more motion vectors; if two motion vectors point to the same reference picture, the average of the two motion vectors. Alternatively, a virtual motion candidate can be derived from any function that takes as input a motion vector in a lookup table. Example functions are: adding two or more motion vectors together; averaging two or more motion vectors. The motion vectors can be scaled before being input to the function.

[0325] b. A virtual motion candidate can be added to the same position as the redundant motion candidate.

[0326] c. A dummy motion candidate may be added before all other motion candidates (eg, starting from the smallest entry index, such as zero).

[0327] d. In one example, it is only applied under certain conditions, such as when the current LUT is not full.

[0328] 7. The LUT update method based on redundancy removal can be called under certain conditions, such as

[0329] a. The current block is encoded and decoded using Merge mode,

[0330] b. The current block is encoded and decoded using the AMVP mode, but at least one component of the MV difference is non-zero;

[0331] c. Whether the current block is coded using a sub-block based motion prediction / motion compensation method (e.g., not coded using an affine mode)

[0332] d. The current block is encoded and decoded using Merge mode, and the motion information is associated with a certain type (e.g., from a spatial neighboring block, from a left neighboring block, from a temporal block)

[0333] h. After encoding / decoding a block, one or more lookup tables may be updated by simply inserting the M sets of motion information to the end of the table, ie after all existing candidates.

[0334] i. Alternatively, furthermore, some existing motion candidates in the table may be removed.

[0335] 1. In one example, if the table is full after inserting M sets of motion information, the first several entries of the motion candidates may be removed from the table.

[0336] 2. In one example, if the table is full before inserting the M sets of motion information, the first several entries of the motion candidates may be removed from the table.

[0337] ii. Alternatively, furthermore, if the block is coded using motion candidates from a table, the motion candidates in the table may be reordered so as to place the selected motion candidate at the last entry of the table.

[0338] 1. In one example, before encoding / decoding a block, a lookup table may include HMVP0, HMVP1, HMVP2...HMVP K-1 、HMVP K 、HMVP K+1 ...HMVP L-1 Marked motion candidates, among which HMVP i Marks the i-th entry in the lookup table. If K (K is inclusive in the range [0, L-1]) prediction block, then after encoding / decoding this block, the lookup table is reordered as: HMVP0, HMVP1, HMVP2, ..., HMVP K-1 、HMVP K 、HMVP K+1 ,...,HMVP L-1 、HMVP K .

[0339] i. After encoding and decoding an intra-constrained block, the lookup table can be cleared.

[0340] j. If an entry of motion information is added to the lookup table, further entries of motion information may also be added to the table by deduction from the motion information. In this case, the counter associated with the lookup table may be incremented by more than 1.

[0341] i. In one example, scaling the MV of an entry of motion information and placing it into a table;

[0342] ii. In one example, the MV of the entry of the motion information is added by (dx, dy) and placed into the table;

[0343] iii. In one example, the average of the MVs of two or more entries of motion information is calculated and put into a table.

[0344] Example D2: If a block is located at a picture / slice / slice boundary, then the update of the lookup table may never be allowed.

[0345] Example D3: The motion information of the upper LCU row may be disabled to encode and decode the current LCU row.

[0346] a. In this case, the number of available motion candidates may be reset to 0 at the beginning of a new slice / slice / LCU row.

[0347] Example D4: At the beginning of encoding / decoding a slice / slice with a new temporal layer index, the number of available motion candidates may be reset to 0.

[0348] Example D5: The lookup table may be continuously updated with one stripe / slice / LCU row / strip having the same temporal layer index.

[0349] a. Alternatively, the lookup table may be updated only after encoding / decoding every S (S>=1) CTUs / CTBs / CUs / CBs or after encoding / decoding a certain region (e.g., a region of size equal to 8×8 or 16×16).

[0350] b. Alternatively, the lookup table may be updated only after every S (S>=1) blocks (e.g., CU / CB) are encoded / decoded using certain modes (e.g., S inter-frame coded blocks). Alternatively, the lookup table may be updated only after every S (S>=1) inter-frame coded blocks (e.g., CU / CB) are encoded / decoded that are not coded using a sub-block-based motion prediction / motion compensation method (e.g., not coded using affine and / or ATMVP modes).

[0351] c. Alternatively, the lookup table can be updated only when the upper left coordinate of the encoded / decoded block meets some conditions. For example, the lookup table can be updated only when (x & M == 0) && (y & M == 0), where (x, y) is the upper left coordinate of the encoded / decoded block. M is an integer such as 2, 4, 8, 16, 32, or 64.

[0352] d. Alternatively, a lookup table can stop updating once the maximum allowed counter is reached.

[0353] e. In one example, a counter may be predefined. Alternatively, the counter may be signaled in a video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), slice header, slice header, codec tree unit (CTU), codec tree block (CTB), codec unit (CU), or prediction unit (PU), or in a region covering multiple CTUs / CTBs / CUs / PUs.

[0354] Example D6: A lookup table update procedure can be called from within a different procedure.

[0355] a. In one example, for a block coded in Merge mode, the lookup table update process may be called after decoding a Merge candidate, after constructing a Merge list, or after decoding motion information with / without refinement.

[0356] b. In one example, for blocks coded using AMVP mode, the lookup table update process may be invoked after decoding motion information with / without refinement.

[0357] c. When and / or how to update the lookup table may depend on the codec mode, block size, video processing data unit, low latency check, etc.

[0358] i. In one example, when a block is coded using AMVP mode, the lookup table can be directly updated without pruning.

[0359] ii. Alternatively, when a block is encoded using Merge mode, pruning can be used to update the lookup table.

[0360] iii. Alternatively, when a block is coded using Merge mode and its motion information is derived from spatial and / or temporal blocks, pruning can be used to update the lookup table.

[0361] iv. Alternatively, when a block is coded using Merge mode and its motion information is derived from motion candidates in the lookup table, the lookup table can be reordered without pruning.

[0362] v. Alternatively, when a block is coded using Merge mode and its motion information is derived from virtual candidates in the lookup table (eg, combined bidirectional, paired, zero motion vector candidates), the lookup table may not be updated.

[0363] vi. Alternatively, when a block is coded using the sub-block Merge mode and / or the triangle Merge mode, the lookup table may not be updated.

[0364] vii. Alternatively, when a block is coded using Merge mode with Motion Vector Difference (MMVD) and its motion information is derived from spatial and / or temporal blocks, the lookup table can be updated directly.

[0365] viii. In one example, when a block is coded using illumination compensation (IC) mode and / or overlapped block motion compensation (OBMC) mode and / or decoder-side motion vector derivation (DMVD) mode, the lookup table may not be updated. Alternatively, when a block is coded using such modes, the lookup table may be updated.

[0366] Additional Example Embodiments

[0367] A history-based MVP (HMVP) method is proposed, where the HMVP candidate is defined as the motion information of the previously coded block. A table with multiple HMVP candidates is maintained during the encoding / decoding process. When a new slice is encountered, the table is cleared. Whenever there is an inter-coded block, the associated motion information is added to the last entry of the table as a new HMVP candidate. The entire encoding and decoding process is Figure 31 Depicted in.

[0368] In one example, the table size is set to L (eg, L=16 or 6, or 44), L indicating that up to L HMVP candidates can be added to the table.

[0369] In one embodiment (corresponding to example D1.gi), if there are more than L HMVP candidates from previously coded blocks, a First-In-First-Out (FIFO) rule is applied so that the table always contains the most recent L motion candidates from the previous codec. Figure 32 Depicted is an example of applying the FIFO rule to remove an HMVP candidate and add a new one to the table used in the proposed method.

[0370] In another embodiment (corresponding to example D1.g.iii), whenever a new motion candidate is added (such as the current block is inter-coded and in non-affine mode), a redundancy check process is first applied to identify whether the same or similar motion candidate exists in the LUT.

[0371] Some examples are depicted below:

[0372] Figure 33A An example is shown when the LUT is full before adding a new motion candidate.

[0373] Figure 33B An example is shown when the LUT is not full before adding a new motion candidate.

[0374] Figure 33A and Figure 33B An example of a LUT updating method based on redundancy removal is shown together (with one redundant motion candidate removed).

[0375] Figure 34A and Figure 34B Example implementations for two cases of the redundancy removal-based LUT updating method are shown (with multiple redundant motion candidates removed, 2 candidates in the figure).

[0376] Figure 34A An example case is shown when the LUT is full before adding a new motion candidate.

[0377] Figure 34BAn example case is shown when the LUT is not full before adding a new motion candidate.

[0378] HMVP candidates can be used in the Merge candidate list construction process. All HMVP candidates from the last entry to the first entry in the table (or the last K0 HMVPs, for example, K0 is equal to 16 or 6) are inserted after the TMVP candidate. Pruning is applied to the HMVP candidates. Once the total number of available Merge candidates reaches the maximum allowed Merge candidates signaled, the Merge candidate list construction process is terminated. Alternatively, once the total number of added motion candidates reaches a given value, the extraction of motion candidates from the LUT is terminated.

[0379] Similarly, HMVP candidates can also be used in the AMVP candidate list construction process. The motion vectors of the last K1 HMVP candidates in the table are inserted after the TMVP candidates. Only HMVP candidates with the same reference picture as the AMVP target reference picture are used to construct the AMVP candidate list. Pruning is applied to HMVP candidates. In one example, K1 is set to 4.

[0380] Figure 28 is a block diagram of a video processing device 2800. Device 2800 can be used to implement one or more of the methods described herein. Device 2800 can be embodied in a smartphone, a tablet, a computer, an Internet of Things (IoT) receiver, etc. Device 2800 may include one or more processors 2802, one or more memories 2804, and video processing hardware 2806. Processor(s) 2802 can be configured to implement one or more methods described in this document. Memory(s) 2804 can be used to store code and data for implementing the methods and techniques described herein. Video processing hardware 2806 can be used to implement some of the techniques described in this document in hardware circuitry.

[0381] Figure 29 2 is a flow chart of an example of a video decoding method 2900. The method 2900 includes, at step 2902, maintaining one or more tables, wherein each table includes a set of motion candidates, and each motion candidate is associated with corresponding motion information. The method 2900 also includes, at step 2904, performing a conversion between a current block and a bitstream representation of a video including the current block using the motion information in the table. The method 2900 also includes, at step 2906, updating the one or more tables based on M sets of additional motion information associated with the current block after the conversion is performed, where M is an integer.

[0382] Figure 303000. The method 3000 includes, at step 3002, performing a conversion between a current block and a bitstream representation of a video including the current block using one or more tables, wherein each table includes one or more motion candidates and each motion candidate is associated with corresponding motion information. The method 3000 also includes, at step 3004, updating the one or more tables based on M sets of additional motion information associated with the current block based on the conversion, where M is an integer.

[0383] With respect to methods 2900 and 3000, in some embodiments, the motion information includes at least one of the following: a prediction direction, a reference picture index, a motion vector value, an intensity compensation flag, an affine flag, a motion vector difference precision, and a motion vector difference value. Furthermore, the motion information may also include block position information indicating a source of the motion information. In some embodiments, a video block may be a CU or a PU, and a portion of a video may correspond to one or more video slices or one or more video pictures.

[0384] In some embodiments, each LUT includes an associated counter, wherein the counter is initialized to a zero value at the beginning of the portion of the video and incremented for each encoded video region in the portion of the video. A video region includes one of a codec tree unit, a codec tree block, a codec unit, a codec block, or a prediction unit. In some embodiments, for a corresponding LUT, the counter indicates the number of motion candidates removed from the corresponding LUT. In some embodiments, the set of motion candidates can have the same size for all LUTs. In some embodiments, the portion of the video corresponds to a slice of the video, and wherein the number of LUTs is equal to N*P, where N is an integer representing the number of LUTs per decoding thread and P is an integer representing the number of maximum codec unit rows or the number of slices in a slice of the video. Additional details of methods 2900 and 3000 are described in the examples provided in Section 4 and in the examples listed below.

[0385] Additional features and embodiments of the above-described methods / techniques are described below using a clause-based description format.

[0386] 1. A video processing method, comprising: maintaining one or more tables, wherein each table includes one or more motion candidates and each motion candidate is associated with corresponding motion information; performing a conversion between a current block and a bitstream representation of a video including the current block by using the motion information in the table; and updating the one or more tables based on M sets of additional motion information associated with the current block after performing the conversion, where M is an integer.

[0387] 2. A video processing method, comprising: performing a conversion between a current block and a bitstream representation of a video including the current block by using one or more tables, wherein each table includes one or more motion candidates and each motion candidate is associated with corresponding motion information; and updating the one or more tables based on M sets of additional motion information associated with the current block based on the conversion, where M is an integer.

[0388] 3. The method of clause 1 or 2, wherein M is equal to 1.

[0389] 4. The method of clause 1 or 2, further comprising adding the M sets of motion information as M additional motion candidates to the table.

[0390] 5. The method of clause 4, wherein the updating of the table further comprises applying a comparison operation to a set of the M sets of motion information before adding the set of motion information as an additional motion candidate to the table.

[0391] 6. The method of clause 5, wherein the set of motion information is pruned with the last K candidates or the first K%L existing motion candidates, where K is an integer representing a motion candidate counter corresponding to a table and L represents the size of the table.

[0392] 7. The method of clause 4, wherein a set of the M sets of motion information is added to the table as an additional motion candidate and no comparison operation is applied.

[0393] 8. The method of clause 4, further comprising: when adding a set of the M sets of motion information to the table, the set of motion information includes motion information different from any or a portion of the current motion candidates in the table.

[0394] 9. The method as described in clause 4 also includes: when adding a set of M sets of motion information, the motion vector (MV) difference between the MV of the current motion candidate in the table and the MV of the same reference picture from the set of motion information is not less than one or more thresholds.

[0395] 10. The method of clause 1 or 2, wherein each table has a corresponding counter, and the corresponding counter is incremented by M.

[0396] 11. The method of clause 4, wherein additional motion candidates are added to the table with an index equal to K%L, K representing the motion candidate counter corresponding to the table to be updated and L representing the size of the table.

[0397] 12. The method of clause 4, wherein an additional motion candidate is added to the table with an index equal to min(K+1, L-1), K representing the motion candidate counter corresponding to the table to be updated and L representing the size of the table.

[0398] 13. The method of clause 4, wherein if K is not less than L, the first motion candidate with an index of 0 is removed from the table.

[0399] 14. A method as recited in clause 11 or 12, wherein a recent set of motion information from previously coded blocks is maintained regardless of whether identical or similar motion candidates exist.

[0400] 15. The method of clause 4, wherein the updating of the one or more tables comprises checking current motion candidates in the tables for redundancy with additional motion candidates.

[0401] 16. The method of clause 15, wherein the updating of the one or more tables comprises performing a redundancy removal based table update, wherein one or more redundant current motion candidates are removed from the table and a recent set of motion information from previously coded blocks is maintained.

[0402] 17. The method of clause 16, wherein the most recent set of motion information comprises non-copied motion information.

[0403] 18. The method of clause 16, wherein a counter associated with the table is not incremented when at least one of the current motion candidates in the table is redundant with respect to the additional motion candidate.

[0404] 19. The method of clause 16, wherein the redundancy removal based table update process comprises performing a comparison operation of the additional motion candidates to be added to the table with the current motion candidates in the table.

[0405] 20. The method of clause 19, wherein the checking for redundancy identifies a redundant current motion candidate in the table, the identified redundant current motion candidate being moved from the current position to a last position after the remaining current motion candidates in the table.

[0406] 21. A method as described in clause 19, wherein the redundancy check identifies redundant current motion candidates in the table, the identified redundant current motion candidates are removed from the table to provide empty entries in the table, and the non-redundant current motion candidates following the identified redundant current motion candidate in the table are moved forward to fill the empty entry.

[0407] 22. The method of clause 21, wherein the adding of the additional motion candidates is performed after moving the non-redundant current motion candidate in the table.

[0408] 23. The method of clause 16, wherein performing the redundancy removal based table update comprises checking the redundancy of the current motion candidate in the table to identify a plurality of redundant motion candidates, and removing all of the identified plurality of redundant motion candidates.

[0409] 24. The method of clause 23, wherein the counter corresponding to the table is decremented by a value obtained by subtracting 1 from the number of the identified plurality of redundant motion candidates.

[0410] 25. The method of clause 23, wherein the checking for redundancy is terminated after the number of identified redundant motion candidates reaches a maximum value maxR, where maxR is a positive integer variable.

[0411] 26. The method of clause 16, wherein the checking for redundancy is terminated when a redundant current motion candidate is identified.

[0412] 27. The method of clause 16, wherein the checking for redundancy is performed in order from the first current motion candidate to the last current motion candidate in the table.

[0413] 28. The method of clause 16, wherein the checking of redundancy is performed in the order in which the current motion candidates are added to the table or in the order of the decoding process of the blocks from which the current motion candidates have been obtained.

[0414] 29. The method of clause 16, wherein the checking for redundancy identifies redundant current motion candidates in a table, deriving virtual motion candidates from the identified redundant candidates, and removing the identified plurality of redundant candidates from the table.

[0415] 30. The method of clause 29, wherein the virtual motion candidate is derived by i) adding offset(s) to the horizontal or vertical component of one or more motion vectors or ii) averaging two motion vectors pointing to the same reference picture.

[0416] 31. The method of clause 29, wherein the virtual motion candidate is derived from a function of a motion vector associated with a current motion candidate in a table.

[0417] 32. The method of clause 29, wherein the dummy motion candidate is added to the table at the location of the identified redundant motion candidate.

[0418] 33. The method of clause 16, wherein a dummy motion candidate is added before a current motion candidate in the table, the current motion candidate corresponding to a motion candidate in the table other than the identified redundant motion candidate.

[0419] 34. The method of clause 16, wherein upon determining that certain conditions are satisfied, a redundancy removal based table update procedure is invoked.

[0420] 35. The method of clause 34, wherein the condition comprises: when the current block is encoded and decoded using Merge mode.

[0421] 36. The method of clause 34, wherein the condition comprises when the current block is neither coded using a sub-block based motion prediction method nor a sub-block based motion compensation method.

[0422] 37. The method of clause 1 or 2, wherein updating of the one or more tables is performed only after execution of the transformation of S blocks, S>=1.

[0423] 38. The method of clause 37, wherein the S blocks are inter-coded blocks.

[0424] 39. The method of clause 37, wherein the S blocks are neither coded using sub-block based motion prediction nor coded / decoded using sub-block based motion compensation.

[0425] 40. The method of clause 39, wherein the condition comprises: when the current block is encoded using Merge mode and the source of the motion information of the corresponding motion candidate is of a certain type.

[0426] 41. The method of clause 40, wherein the source of the motion information of the corresponding motion candidate is from a spatial neighboring block including a left neighboring block, or a temporal block.

[0427] 42. The method of clause 4, wherein M additional motion candidates are added after a current motion candidate that has been added to the table before the M additional motion candidates.

[0428] 43. The method of clause 42, further comprising removing some of the current motion candidates from the table.

[0429] 44. The method of clause 43, wherein the first one or more entries in the set of motion candidates are removed from the table when the table is full before or after inserting the M sets of motion information.

[0430] 45. The method of clause 1 or 2, wherein the motion information is added to the table when the current block is coded with directly signaled motion information.

[0431] 46. The method of clause 45, wherein the current block is encoded or decoded in AMVP mode, Merge with Motion Vector Difference (MMVD) mode for normal / affine inter mode, or Advanced Motion Vector Prediction (AMVR) mode for normal / affine inter mode.

[0432] 47. The method of clause 45, wherein the motion information is added to the last entry of the table or to the entry storing the next available motion candidate.

[0433] 48. The method of clause 1, 2, or 16, further comprising: adding additional motion candidates to the table through a motion information derivation process based on the M sets of motion information.

[0434] 49. The method of clause 48, wherein a counter corresponding to the table is incremented by a value greater than 1.

[0435] 50. The method of clause 48, wherein the motion information derivation process comprises scaling a motion vector of a set of the M sets of motion information.

[0436] 51. The method of clause 48, wherein the motion information derivation process comprises adding a motion vector offset (dx, dy) to the motion vector of a set of the M sets of motion information.

[0437] 52. The method of clause 48, wherein the motion information derivation process comprises using an average of motion vectors of two or more of the M sets of motion information.

[0438] 53. The method of any of clauses 1-52, wherein updating comprises modifying the M sets of motion information prior to adding to the table.

[0439] 54. The method of clause 1 or 2, wherein performing the conversion comprises generating a bitstream representation from the current block.

[0440] 55. The method of clause 1 or 2, wherein performing the conversion comprises generating the current block from a bitstream representation.

[0441] 56. A method as described in any of clauses 1 to 55, wherein the motion candidate is associated with motion information, the motion information comprising at least one of: a prediction direction, a reference picture index, a motion vector value, an intensity compensation flag, an affine flag, a motion vector difference precision, or a motion vector difference value.

[0442] 57. The method of any of clauses 1-55, wherein the motion candidate corresponds to a motion candidate for an intra prediction mode for intra-mode coding.

[0443] 58. The method of any of clauses 1-55, wherein the motion candidate corresponds to a motion candidate comprising illumination compensation parameters for IC parameter codec.

[0444] 59. The method of any one of clauses 1 to 58, further comprising:

[0445] Conversion between subsequent video chunks of the video and the bitstream representation of the video is performed based on the updated table.

[0446] 60. An apparatus comprising a processor and non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of clauses 1 to 59.

[0447] 61. A computer program product stored on a non-transitory computer readable medium, the computer program product comprising program code for carrying out the method of any one of clauses 1 to 59.

[0448] From the foregoing, it will be appreciated that specific embodiments of the disclosed technology have been described herein for purposes of illustration, but that various modifications may be made without departing from the scope of the invention. Accordingly, the disclosed technology is not to be limited, except as by the appended claims.

[0449] The disclosed and other embodiments, modules, and functional operations described in this document may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or in a combination of one or more thereof. The disclosed and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium, for execution by a data processing apparatus or for controlling the operation of the data processing apparatus. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances that effect a machine-readable propagated signal, or a combination of one or more thereof. The term "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, an apparatus may include code that creates an operating environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof. A propagated signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver device.

[0450] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program need not correspond to a file in a file system. A program can be stored in the portion of a document preserving other programs or data (e.g., one or more scripts stored in a markup language document), in a single document dedicated to the program in question, or in multiple coordination documents (e.g., documents storing the portion of one or more modules, subroutines, or code). A computer program can be deployed to run on a computer or to run on multiple computers located at a site or distributed across multiple sites and interconnected by a communication network.

[0451] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0452] Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from read-only memory or random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operatively coupled to one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, to receive data from or transfer data to, or both. However, a computer need not include such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0453] Although this patent document contains many details, these details should not be interpreted as limitations on any invention or the scope that can be claimed, but rather as descriptions of features that can be specific to a particular embodiment of a particular invention. Certain features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although the above features may be described as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be cut out from the combination in some cases, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0454] Similarly, while operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown, or in sequential order, or that all illustrated operations be performed, in order to achieve desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0455] Only a few implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A method for processing video data, comprising: Determining motion information of a current video block of the video; encoding and decoding the current video block based at least on the determined motion information; as well as updating the table using new motion candidates corresponding to the determined motion information; wherein the table includes one or more motion candidates derived from one or more video blocks that have been coded and decoded, and during the updating process, because at least one motion candidate currently stored in the table is identified as being identical to the new motion candidate through a redundancy check before the new motion candidate is added to the table, the at least one motion candidate identical to the new motion candidate is removed from the table, wherein during determining motion information of a subsequent video block, the at least one motion candidate in the updated table is checked; The determining of the motion information of the current video block of the video includes: maintaining said table; constructing a motion candidate list for the current video block by checking at least one motion candidate in the table to determine whether to add the at least one motion candidate to the motion candidate list for the current video block; Motion information for the current video block is determined based at least on the motion candidate list for the current video block and excluding reference to the table.

2. The method according to claim 1, wherein The encoding and decoding includes generating a bitstream from the current video block.

3. The method according to claim 1, wherein The encoding and decoding includes generating the current video block from a bitstream.

4. The method according to claim 1, wherein Determining the motion information of the subsequent video block includes: checking at least one motion candidate in the updated table during construction of a motion candidate list for the subsequent video block; and Motion information for the subsequent video block is derived using a motion candidate list for the subsequent video block.

5. The method according to claim 1, wherein The redundancy check is performed in the order in which the motion candidates in the table are added to the table or in the order of the encoding and decoding process of the blocks from which the motion candidates in the table have been obtained.

6. The method of claim 1, wherein: The redundancy check is performed in order from the first motion candidate with the smallest index in the table to the last motion candidate with the largest index in the table.

7. The method of claim 1, wherein: When a redundant motion candidate in the table is identified, the redundancy check is terminated.

8. The method of claim 1, wherein: The redundancy check includes comparing the motion candidates in the table with the new motion candidate.

9. The method of claim 1, wherein: When at least one of the motion candidates in the table is redundant with respect to the new motion candidate, a counter indicating the number of motion candidates in the table is not incremented.

10. The method of claim 1, wherein: The index of the motion candidates in the table is based on the order in which the motion candidates were added to the table.

11. The method according to claim 10, wherein: When the new motion candidate is added to the table, the new motion candidate has an index greater than other motion candidates in the table.

12. The method of claim 10, wherein: During determining the motion information of the subsequent video block, at least one motion candidate in the table is checked in an order of at least one index of the at least one motion candidate.

13. The method of claim 10, wherein: The order in which the motion candidates are added to the table is based on the order in which the blocks from which the motion candidates were derived were coded.

14. The method of claim 1, wherein: The at least one motion candidate that is identical to the new motion candidate is removed from the table to provide an empty entry in the table, and motion candidates whose index is greater than the index of the at least one motion candidate that is identical to the new motion candidate are moved forward to fill the empty entry, and after moving the motion candidates, the new motion candidate is added to the table.

15. The method of claim 1, wherein: A motion candidate in the table includes motion information that is different from any other motion candidate in the table.

16. The method of claim 1, wherein: The MV difference between the motion vector MV of the motion candidate in the table and the MV of the same reference picture of another motion candidate is not less than a threshold.

17. The method of claim 1, wherein: The updating is performed if at least one condition is satisfied, wherein the at least one condition is based on at least one of a coding mode of the current video block or a motion information source of a corresponding motion candidate.

18. The method of claim 1, wherein: The motion candidates in the table are associated with motion information, the motion information including at least one of: position information of a block indicating where the motion information comes from, a prediction direction, a reference picture index, a motion vector value, an intensity compensation flag, an affine flag, a motion vector difference accuracy, or a motion vector difference value.

19. A video processing device comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: Determining motion information of a current video block of the video; encoding and decoding the current video block based at least on the determined motion information; and updating the table using new motion candidates corresponding to the determined motion information; in, the table including one or more motion candidates derived from one or more video blocks that have been coded and decoded, wherein during the updating, at least one motion candidate currently stored in the table is identified as being identical to the new motion candidate through a redundancy check before the new motion candidate is added to the table, the at least one motion candidate identical to the new motion candidate is removed from the table, wherein during determining motion information of a subsequent video block, the at least one motion candidate in the updated table is checked; The determining of the motion information of the current video block of the video includes: maintaining said table; constructing a motion candidate list for the current video block by checking at least one motion candidate in the table to determine whether to add the at least one motion candidate to the motion candidate list for the current video block; Motion information for the current video block is determined based at least on the motion candidate list for the current video block and excluding reference to the table.

20. A non-transitory computer-readable storage medium storing instructions that cause a processor to: Determining motion information of a current video block of the video; encoding and decoding the current video block based at least on the determined motion information; and updating the table using new motion candidates corresponding to the determined motion information; in, the table including one or more motion candidates derived from one or more video blocks that have been coded and decoded, wherein during the updating, at least one motion candidate currently stored in the table is identified as being identical to the new motion candidate through a redundancy check before the new motion candidate is added to the table, the at least one motion candidate identical to the new motion candidate is removed from the table, wherein during determining motion information of a subsequent video block, the at least one motion candidate in the updated table is checked; The determining of the motion information of the current video block of the video includes: maintaining said table; constructing a motion candidate list for the current video block by checking at least one motion candidate in the table to determine whether to add the at least one motion candidate to the motion candidate list for the current video block; Motion information for the current video block is determined based at least on the motion candidate list for the current video block and excluding reference to the table.

21. A method for storing a video bitstream, comprising: Determining motion information of a current video block of the video; generating the bitstream from the current video block based at least on the determined motion information; updating the table using new motion candidates corresponding to the determined motion information; as well as storing the bitstream in a non-transitory computer-readable recording medium, wherein the table includes one or more motion candidates derived from one or more video blocks that have been coded and decoded, and during the updating process, because at least one motion candidate currently stored in the table is identified as being identical to the new motion candidate through a redundancy check before the new motion candidate is added to the table, the at least one motion candidate identical to the new motion candidate is removed from the table, wherein during determining motion information of a subsequent video block, the at least one motion candidate in the updated table is checked; The determining of the motion information of the current video block of the video includes: maintaining said table; constructing a motion candidate list for the current video block by checking at least one motion candidate in the table to determine whether to add the at least one motion candidate to the motion candidate list for the current video block; Motion information for the current video block is determined based at least on the motion candidate list for the current video block and excluding reference to the table.

Images