Insertion order of motion candidates in LUT

Abstract

Devices, systems, and methods for encoding and decoding digital video using a lookup table (LUT) containing motion candidates are described. In a representative aspect, a method of video processing includes: maintaining one or more tables, wherein at least one of the one or more tables is associated with one or more entries; deriving a second set of motion candidates based on a first set of motion candidates in the at least one table; and performing a conversion between a current block of a video and a bitstream representation of the video based on the second set of motion candidates.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is to timely claim the priority and benefits of International Patent Application No. PCT / CN2019 / 072058 filed on January 16, 2019, in accordance with applicable patent law. The entire disclosure of the foregoing application is incorporated by reference as part of the disclosure of this application. Technical Field

[0003] This patent document relates to video coding technology, equipment and systems. Background Art

[0004] Despite advances in video compression, digital video still accounts for the largest use of bandwidth on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, bandwidth demands for digital video usage are expected to continue to grow. Summary of the invention

[0005] Devices, systems and methods related to encoding and decoding digital video using a set of look-up tables (LUTs) containing codec candidates are described. The described methods can be applied to existing video codec standards (e.g., High Efficiency Video Coding (HEVC)) and future video codec standards or video codecs.

[0006] In one representative aspect, the disclosed technology can be used to provide a method of video processing. The method includes: maintaining one or more tables, wherein at least one of the one or more tables is associated with one or more entries; deriving a second motion candidate set based on a first motion candidate set in at least one table; and performing a conversion between a current block of a video and a bitstream representation of the video based on the second motion candidate set.

[0007] In another representative aspect, the disclosed technology can be used to provide a method of video processing. The method includes: maintaining one or more tables, wherein each of the one or more tables is associated with one or more entries; adaptively changing the insertion order of motion candidates from at least one of the one or more tables to a candidate list; and performing conversion between a current block of a video and a bitstream representation of the video based on the candidate list.

[0008] In yet another representative aspect, the above method is embodied in the form of processor executable code and stored in a computer readable program medium.

[0009] In yet another representative aspect, a device configured or operable to perform the above method is disclosed. The device may include a processor programmed to implement the method.

[0010] In yet another representative aspect, a video decoder device may implement a method as described herein.

[0011] The above and other aspects and features of the disclosed technology are described in more detail in the drawings, the description, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An example block diagram of a general High Efficiency Video Codec (HEVC) video encoder and decoder is shown.

[0013] Figure 2 An example of macroblock (MB) partitioning in H.264 / AVC is shown.

[0014] Figure 3 An example of dividing a coding block (Coding Block, CB) into prediction blocks (Prediction Block, PB) is shown.

[0015] Figure 4A and Figure 4B Examples of subdividing a coding tree block (CTB) into a CB and a transform block (TB), and corresponding quadtrees are shown respectively.

[0016] Figure 5A and Figure 5B An example of subdivision of the largest coding unit (LCU) and the corresponding QTBT (QuadTree plus Binary Tree) is shown.

[0017] Figure 6A-6E An example of segmenting a codec block is shown.

[0018] Figure 7 An example segmentation of a QTBT-based CB is shown.

[0019] Figure 8A-8I An example of supporting partitioning of a CB of a Multi-Tree Type (MTT) which is a generalization of QTBT is shown.

[0020] Fig. 9 An example of constructing a Merge candidate list is shown.

[0021] Fig.10Examples of locations of spatial candidates are shown.

[0022] Fig.11 An example of candidate pairs for which redundancy check of spatial merge candidates is performed is shown.

[0023] Fig. 12A and Fig. 12B An example of the position of the second prediction unit (PU) based on the size and shape of the current block is shown.

[0024] Fig.13 An example of motion vector scaling for temporal merge candidates is shown.

[0025] Fig.14 An example of candidate positions of temporal Merge candidates in collocated pictures is shown.

[0026] Fig.15 An example of generating a combined bi-predictive Merge candidate is shown.

[0027] Fig.16 An example of a derivation process of motion vector prediction candidates is shown.

[0028] Fig.17 An example of motion vector scaling for spatial motion vector candidates is shown.

[0029] Fig.18 An example of motion prediction using an Alternative Temporal Motion Vector Prediction (ATMVP) algorithm for a coding unit (CU) is shown.

[0030] Fig.19 An example of identifying source blocks and source pictures is shown.

[0031] Fig. 20 An example of a codec unit (CU) with sub-blocks and neighboring blocks used by a Spatial-Temporal Motion Vector Prediction (STMVP) algorithm is shown.

[0032] Fig.21 An example of bidirectional matching in Pattern Matched Motion Vector Derivation (PMMVD) mode as a special Merge mode of the frame rate up conversion (FRUC) algorithm is shown.

[0033] Fig. 22 An example of template matching in the FRUC algorithm is shown.

[0034] Fig.23 An example of unidirectional motion estimation in the FRUC algorithm is shown.

[0035] Fig.24 An example of a decoder-side Motion Vector Refinement (DMVR) algorithm based on bidirectional template matching is shown.

[0036] Fig.25 An example of neighboring blocks used to derive spatial Merge candidates is shown.

[0037] Fig.26 An example of the difference between Type 1 and Type 2 definitions is shown.

[0038] Fig. 27 An example of selecting a representative position for LUT updating is shown.

[0039] Fig.28A and Fig.28B An example of updating the LUT with a new set of motion information is shown.

[0040] Fig.29 An example of a decoding flow chart of the proposed history-based motion vector prediction (HMVP) method is shown.

[0041] Fig.30 An example of updating a table in the proposed HMVP method is shown.

[0042] Fig.31A and Fig.31B An example of a LUT updating method based on redundancy removal is shown.

[0043] Fig.32A and Fig.32B Additional examples of LUT update methods based on redundancy removal are shown.

[0044] Fig.33 A flow chart of an example method for video processing is shown.

[0045] Fig.34 A flow chart of another example method for video processing is shown.

[0046] Fig.35 is a block diagram of an example of a hardware platform for implementing the visual media decoding or visual media encoding techniques described in this document.

[0047] Fig.36 is a block diagram of an example video processing system in which the disclosed techniques may be implemented. DETAILED DESCRIPTION

[0048] Due to the growing demand for higher resolution video, video coding methods and techniques are ubiquitous in modern technology. Video codecs typically include electronic circuits or software that compress or decompress digital video, and are constantly being improved to provide higher coding efficiency. Video codecs convert uncompressed video to a compressed format and vice versa. Video quality, the amount of data used to represent the video (determined by the bit rate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, and end-to-end delay (latency). The compression format usually conforms to a standard video compression specification, such as the High Efficiency Video Codec (HEVC) standard (also known as H.265 or MPEG-H Part 2), the Universal Video Codec standard to be completed, or other current and / or future video codec standards.

[0049] Embodiments of the disclosed technology can be applied to existing video codec standards (e.g., HEVC, H.265) and future standards to improve compression performance. Section headings are used in this document to improve the readability of the description and do not in any way limit the discussion or embodiments (and / or implementation methods) to only the corresponding sections.

[0050] 1. Example Embodiments of Video Coding and Decoding

[0051] Figure 1 An example block diagram of a typical HEVC video encoder and decoder is shown. The encoding algorithm that produces a bitstream that complies with HEVC is generally performed as follows. Each picture is divided into block regions, where the precise block segmentation is passed to the decoder. The first picture of a video sequence (and the first picture at each clean random access point in the video sequence) is encoded and decoded using only intra-picture prediction (some data is predicted spatially from region to region within the same picture, but not depending on other pictures). For all remaining pictures in a sequence or between random access points, the inter-picture temporal prediction codec mode is generally used for most blocks. The encoding process for inter-picture prediction includes selecting motion data including a selected reference picture and a motion vector (MV) to be applied to predict the samples of each block. The encoder and decoder generate the same inter-picture prediction signal by applying motion compensation (MC) using MV and mode decision data, which is sent as side information.

[0052] The residual signal of intra-picture prediction or inter-picture prediction, which is the difference between the original block and its prediction, is transformed by a linear spatial transform. The transform coefficients are then scaled, quantized, entropy coded, and sent together with the prediction information.

[0053] The encoder replicates the decoder processing loop (see Figure 1 The quantized transform coefficients are therefore constructed by inverse scaling and then inverse transformed to replicate a decoded approximation of the residual signal. The residual is then added to the prediction, and the result of the addition can then be fed into one or two loop filters to smooth artifacts caused by the block processing and quantization. The final picture representation (i.e., a copy of the output of the decoder) is stored in the decoded picture buffer for prediction of subsequent pictures. In general, the order of the encoding or decoding process of pictures is usually different from the order in which they arrive from the source; it is necessary to distinguish between the decoding order (i.e., bitstream order) and the output order (i.e., display order) of the decoder.

[0054] It is generally expected that video material encoded by HEVC is input as a progressive scan image (because the source video originates from this format or is produced by deinterlacing before encoding). There is no explicit codec feature in the HEVC design to support the use of interlaced scanning, because interlaced scanning is no longer used for displays and has become essentially uncommon for distribution. However, metadata syntax has been provided in HEVC to allow the encoder to indicate that interlaced video has been transmitted by encoding each field of interlaced video (i.e., even or odd lines of each video frame) as a separate picture or that interlaced video has been transmitted by encoding each interlaced frame as an HEVC coded picture. This provides an efficient way to encode interlaced video without requiring the decoder to support a special decoding process for it.

[0055] 1.1. Example of segmentation tree structure in H.264 / AVC

[0056] The core of the codec layer in previous standards is the macroblock, which contains a 16×16 block of luma samples and, in the usual case of 4:2:0 color sampling, two corresponding 8×8 blocks of chroma samples.

[0057] The intra-codec block uses spatial prediction to exploit the spatial correlation among pixels. Two partitions are defined: 16×16 and 4×4.

[0058] The inter-codec block uses temporal prediction instead of spatial prediction by estimating the motion within the picture. Motion can be estimated independently for a 16×16 macroblock or any of its sub-macroblock partitions (16×8, 8×16, 8×8, 8×4, 4×8, 4×4), as Figure 2Only one motion vector (MV) is allowed per sub-macroblock segmentation.

[0059] 1.2 Example of segmentation tree structure in HEVC

[0060] In HEVC, the Coding Tree Unit (CTU) is divided into Coding Tree Units (CU) using a quadtree structure represented as a codec tree to adapt to various local characteristics. The decision whether to use inter-picture (temporal domain) or intra-picture (spatial domain) prediction to encode and decode a picture area is made at the CU level. Each CU can be further divided into one, two, or four PUs according to the prediction unit (PU) partition type. Within a PU, the same prediction process is applied, and relevant information is sent to the decoder based on the PU. After obtaining the residual block by applying a prediction process based on the PU partition type, the CU can be partitioned into transform units (TUs) according to another quadtree structure similar to the codec tree of the CU. One of the key features of the HEVC structure is that it has multiple partitioning concepts, including CU, PU, ​​and TU.

[0061] Specific features involved in hybrid video codec using HEVC include:

[0062] (1) Codec Tree Unit (CTU) and Codec Tree Block (CTB) Structure : The analog structure in HEVC is the Codec Tree Unit (CTU), which has a size chosen by the encoder and can be larger than a traditional macroblock. A CTU consists of a luma CTB and a corresponding chroma CTB, as well as syntax elements. The size L×L of the luma CTB can be chosen to be L=16, 32, or 64 samples, where larger sizes generally enable better compression. HEVC then supports splitting the CTB into smaller blocks using a tree structure and quadtree-like signaling.

[0063] (2) Codec Unit (CU) and Codec Block (CB) : The quadtree syntax of a CTU specifies the size and position of its luma CB and chroma CB. The root of the quadtree is associated with a CTU. Therefore, the size of the luma CTB is the maximum supported size of the luma CB. The partitioning of a CTU into luma CBs and chroma CBs is jointly signaled. One luma CB and typically two chroma CBs together with the associated syntax form a codec unit (CU). A CTB may contain only one CU or may be partitioned to form multiple CUs, and each CU has an associated partitioning called a prediction unit (PU) and a tree of transform units (TUs).

[0064] (3) Prediction Unit and Prediction Block (PB): The decision to use inter-picture or intra-picture prediction to encode or decode a picture region is made at the CU level. The root of the PU partitioning structure is at the CU level. Depending on the basic prediction type decision, the luma CB and chroma CB can then be further split in size and predicted based on luma and chroma prediction blocks (PBs). HEVC supports variable PB sizes from 64×64 down to 4×4 samples. Figure 3 An example of allowed PBs for an M×M CU is shown.

[0065] (4) Transform Unit (TU) and Transform Block : The prediction residuals are encoded and decoded using block transforms. The root of the TU tree structure is at the CU level. The luma CB residual can be the same as the luma transform block (TB), or can be further divided into smaller luma TBs. The same applies to the chroma TBs. Integer basis functions similar to the functions of the Discrete Cosine Transform (DCT) are defined for square TB sizes 4×4, 8×8, 16×16, and 32×32. For the 4×4 transform of the luma intra picture prediction residuals, an integer transform derived from the form of the Discrete Sine Transform (DST) may be specified alternatively.

[0066] 1.2.1. Example of tree structure partitioning into TB and TU

[0067] For residual codecs, CBs can be recursively partitioned into transform blocks (TBs). The partitioning is signaled by a residual quadtree. Only square CB and TB partitions are specified, where blocks can be recursively divided into quadrants, such as Figure 4A As shown. For a given luma CB of size M×M, a flag signals whether it is divided into four blocks of size M / 2×M / 2. If further division is possible, as signaled by the maximum depth of the residual quadtree indicated in the sequence parameter set (SPS), a flag indicating whether it is divided into four quadrants is assigned to each quadrant. The leaf node blocks generated by the residual quadtree are transform blocks that are further processed by transform coding and decoding. The encoder indicates the maximum and minimum luma TB sizes it will use. When the CB size is larger than the maximum TB size, the division is implicit. When the division will result in a luma TB size smaller than the indicated minimum value, no division is implicit. Except when the luma TB size is 4×4 (in which case a single 4×4 chroma TB is used for the area covered by four 4×4 luma TBs), the chroma TB size is half the luma TB size in each dimension. In the case of an intra picture predicted CU, the decoded samples of the nearest neighboring TB (inside or outside the CB) are used as reference data for intra picture prediction.

[0068] Contrary to previous standards, the HEVC design allows a TB to span multiple PBs of an inter-picture predicted CU to maximize the potential codec efficiency benefits of the quadtree-structured TB partitioning.

[0069] 1.2.2. Parent Node and Child Node

[0070] CTB is divided according to a quadtree structure, and the nodes of the quadtree structure are codec units. A plurality of nodes in the quadtree structure include leaf nodes and non-leaf nodes. A leaf node has no child nodes in the tree structure (i.e., the leaf node is not further divided). Non-leaf nodes include the root node of the tree structure. The root node corresponds to an initial video block (e.g., CTB) of video data. For each corresponding non-root node in a plurality of nodes, the corresponding non-root node corresponds to a video block, which is a sub-block of a video block corresponding to a parent node in the tree structure of the corresponding non-root node. Each corresponding non-leaf node in a plurality of non-leaf nodes has one or more child nodes in the tree structure.

[0071] 1.3. Example of Quadtree plus Binarytree Block Structure with Larger CTU in JEM

[0072] In some embodiments, a reference software known as the Joint Exploration Model (JEM) is used to explore future video codec technologies. In addition to the binary tree structure, JEM also describes a quadtree plus binary tree (QTBT) and a ternary tree (TT) structure.

[0073] 1.3.1. Example of QTBT block segmentation structure

[0074] In contrast to HEVC, the QTBT structure removes the concept of multiple partition types, i.e., it removes the separation of the concepts of CU, PU, ​​and TU, and supports more flexibility in the CU partition shape. In the QTBT block structure, a CU can have a square or rectangular shape. Figure 5A As shown, the codec tree unit (CTU) is first partitioned by a quadtree structure. The quadtree leaf nodes are further partitioned by a binary tree structure. There are two types of partitioning in the binary tree partitioning, symmetrical horizontal partitioning and symmetrical vertical partitioning. The binary tree leaf nodes are called codec units (CUs), and the segmentation is used for prediction and transform processing without any further partitioning. This means that CU, PU, ​​and TU have the same block size in the QTBT codec block structure. In JEM, a CU sometimes consists of codec blocks (CBs) of different color components, for example, a CU contains one luminance CB and two chrominance CBs in the case of P slices and B slices in a 4:2:0 chrominance format, and sometimes consists of CBs of a single component, for example, a CU contains only one luminance CB or only two chrominance CBs in the case of an I slice.

[0075] Define the following parameters for the QTBT segmentation scheme:

[0076] --CTU size: The root node size of the quadtree, which is the same as the concept in HEVC

[0077] --MinQTSize: Minimum allowed quadtree leaf node size

[0078] --MaxBTSize: Maximum allowed binary tree root node size

[0079] --MaxBTDepth: Maximum allowed binary tree depth

[0080] --MinBTSize: minimum allowed binary tree leaf node size

[0081] In one example of a QTBT partitioning structure, the CTU size is set to 128×128 luma samples with two corresponding 64×64 chroma sample blocks, MinQTSize is set to 16×16, MaxBTSize is set to 64×64, MinBTSize (for both width and height) is set to 4×4, and MaxBTDepth is set to 4. Quadtree partitioning is first applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can have sizes from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the leaf quadtree node is 128×128, it will not be further divided by the binary tree because its size exceeds MaxBTSize (i.e., 64×64). Otherwise, the leaf quadtree node can be further divided by the binary tree. Therefore, the quadtree leaf node is also the root node of the binary tree, and its binary tree depth is 0. When the binary tree depth reaches MaxBTDepth (i.e., 4), no further division is considered. When the width of the binary tree node is equal to MinBTSize (i.e. 4), no further horizontal partitioning is considered. Similarly, when the height of the binary tree node is equal to MinBTSize, no further vertical partitioning is considered. The leaf nodes of the binary tree are further processed through the prediction and transformation processes without any further partitioning. In JEM, the maximum CTU size is 256×256 luma samples.

[0082] Figure 5A An example of block segmentation by using QTBT is shown, and Figure 5BThe corresponding tree representation is shown. Solid lines indicate quadtree partitioning, and dashed lines indicate binary tree partitioning. In each partition (i.e., non-leaf) node of the binary tree, a flag is signaled to indicate which partition type (i.e., horizontal or vertical) is used, where 0 indicates horizontal partitioning and 1 indicates vertical partitioning. For quadtree partitioning, there is no need to indicate the partition type, because quadtree partitioning always divides the block horizontally and vertically to produce 4 sub-blocks of the same size.

[0083] In addition, the QTBT scheme supports the ability for luma and chroma to have separate QTBT structures. Currently, for P and B slices, the luma CTB and chroma CTB in one CTU share the same QTBT structure. However, for I slices, the luma CTB is split into CUs by the QTBT structure, and the chroma CTB is split into chroma CUs by another QTBT structure. This means that the CU in the I slice consists of the codec block of the luma component or the codec blocks of the two chroma components, and the CU in the P slice or B slice consists of the codec blocks of all three color components.

[0084] In HEVC, inter prediction of small blocks is restricted to reduce memory access for motion compensation, so that bidirectional prediction is not supported for 4×8 blocks and 8×4 blocks, and inter prediction is not supported for 4×4 blocks. In JEM's QTBT, these restrictions are removed.

[0085] 1.4. Versatile Video Codec (VVC) Ternary Tree (TT)

[0086] Fig. 6A An example of a quadtree (QT) partition is shown, and Figure 6B and Figure 6C Examples of vertical and horizontal binary tree (BT) partitioning are shown respectively. In some embodiments, in addition to quadtree and binary tree, ternary tree (TT) partitioning is also supported, such as horizontal and vertical center-side ternary trees (such as Fig.6D and Fig. 6E shown).

[0087] In some embodiments, two levels of trees are supported: a region tree (quadtree) and a prediction tree (binary tree or ternary tree). The CTU is first segmented by the region tree (RT). The RT leaf can be further divided by the prediction tree (PT). The PT leaf can also be further divided by the PT until the maximum PT depth is reached. The PT leaf is the basic coding unit. For convenience, it is still referred to as a CU. The CU cannot be further divided. Prediction and transformation are applied to the CU in the same way as JEM. The entire segmentation structure is named "multi-type tree".

[0088] 1.5. Examples of Segmentation Structures in Alternative Video Codec Technologies

[0089] In some embodiments, a tree structure called multi-tree type (MTT) is supported as a generalization of QTBT. Figure 7 As shown, the codec tree unit (CTU) is first divided by a quadtree structure, and the quadtree leaf nodes are further divided by a binary tree structure.

[0090] The structure of MTT consists of two types of tree nodes: region tree (RT) and prediction tree (PT), supporting nine types of segmentation, such as Figures 8A-8I As shown. The region tree can recursively divide the CTU into blocks until the region tree leaf node of size 4×4. At each node in the region tree, a prediction tree can be formed from one of three tree types (binary tree, ternary tree, and asymmetric binary tree). In PT partitioning, it is prohibited to have quadtree partitions in the branches of the prediction tree. As in JEM, the luminance tree and chrominance tree are separated in the I stripe.

[0091] 2 Example of inter-frame prediction in HEVC / H.265

[0092] Video codec standards have improved significantly over the years and now provide, in part, high codec efficiency and support for higher resolutions. The latest standards such as HEVC and H.265 are based on a hybrid video codec structure in which temporal prediction plus transform coding is utilized.

[0093] 2.1 Example of prediction model

[0094] Each inter-predicted PU (prediction unit) has motion parameters for one or two reference picture lists. In some embodiments, the motion parameters include motion vectors and reference picture indexes. In other embodiments, inter_pred_idc signaling may also be used to notify the use of one of the two reference picture lists. In other embodiments, the motion vector may be explicitly encoded as a delta relative to the predicted value.

[0095] When a CU is encoded or decoded in skip mode, one PU is associated with the CU and there are no significant residual coefficients, no coded motion vector increments, or reference picture indices. The Merge mode is specified, whereby the motion parameters of the current PU are obtained from neighboring PUs, including spatial and temporal candidates. The Merge mode can be applied to any inter-predicted PU, not just to the Skip mode. An alternative to the Merge mode is explicit transmission of motion parameters, where each PU explicitly signals the motion vectors, the corresponding reference picture indices for each reference picture list, and the reference picture list usage.

[0096] When signaling indicates that one of the two reference picture lists is to be used, a PU is generated from a block of samples. This is called "unidirectional prediction". Unidirectional prediction can be used for both P slices and B slices.

[0097] When signaling indicates that both of the reference picture lists are to be used, a PU is generated from two sample blocks. This is called "bi-prediction". Bi-prediction is only available for B slices.

[0098] 2.1.1 Example of constructing a candidate for Merge mode

[0099] When predicting a PU using Merge mode, an index pointing to an entry in the Merge candidate list is parsed from the bitstream and used to retrieve motion information. The construction of this list can be outlined according to the sequence of the following steps:

[0100] Step 1: Initial candidate derivation

[0101] Step 1.1: Spatial Candidate Derivation

[0102] Step 1.2: Redundancy check of spatial candidates

[0103] Step 1.3: Time Domain Candidate Derivation

[0104] Step 2: Additional candidate insertions

[0105] Step 2.1: Create bidirectional prediction candidates

[0106] Step 2.2: Insert zero motion candidates

[0107] Fig. 9 An example of constructing a Merge candidate list based on the sequence of steps outlined above is shown. For spatial Merge candidate derivation, up to four Merge candidates are selected from candidates located at five different positions. For temporal Merge candidate derivation, up to one Merge candidate is selected from two candidates. Since a constant number of candidates for each PU is assumed at the decoder, additional candidates are generated when the number of candidates does not reach the maximum number of Merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, truncated unary binarization (TU) is used to encode the index of the best Merge candidate. If the size of the CU is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

[0108] 2.1.2 Constructing Spatial Merge Candidates

[0109] In the derivation of spatial domain Merge candidates, Fig.10 Select up to four Merge candidates from the candidates for the positions depicted in . The order of derivation is A 1 , B 1 , B 0 , A 0 and B 2 Only when position A 1 , B 1 , B 0 , A 0 Position B is considered only when any PU in is not available (for example, because it belongs to another slice or part) or is intra coded. 2 . Add position A 1 After the candidates at the , redundancy check is performed on the addition of the remaining candidates, which ensures that the candidates with the same motion information are excluded from the list, thereby improving the encoding and decoding efficiency.

[0110] In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only pairs with Fig.11 The arrows in link pairs, and candidates are added to the list only if the corresponding candidates used for redundancy checking do not have the same motion information. Another source of duplicate motion information is a "second PU" associated with a partition other than 2N×2N. As an example, Fig. 12A and Fig. 12B The second PU in the N×2N and 2N×N cases is depicted respectively. When the current PU is partitioned into N×2N, position A 1 The candidate at position B is not considered for list construction. In some embodiments, adding this candidate may result in two prediction units having the same motion information, which is redundant for having only one PU in the codec unit. Similarly, when the current PU is split into 2N×N, position B is not considered. 1 .

[0111] 2.1.3 Constructing Time Domain Merge Candidates

[0112] In this step, only one candidate is added to the list. Specifically, in the derivation of this temporal Merge candidate, the scaled motion vector is derived based on the collocated PU belonging to the picture with the minimum POC (Picture Order Count) difference with the current picture in a given reference picture list. The reference picture list to be used for deriving the collocated PU is explicitly signaled in the slice header.

[0113] Fig.13An example of the derivation of a scaled motion vector for a temporal Merge candidate (as dashed line) is shown, which is scaled from the motion vector of the collocated PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal Merge candidate is set equal to zero. For B slices, two motion vectors (one for reference picture list 0 and the other for reference picture list 1) are obtained and combined to produce a bidirectional prediction Merge candidate.

[0114] In the collocated PU(Y) belonging to the reference frame, in the candidate C 0 and C 1 Select the position of the time domain candidate between Fig.14 As depicted in. If position C 0 If the PU at position C is unavailable, is intra-coded, or is outside the current CTU, then position C is used. 1 Otherwise, position C 0 Used in the derivation of time domain Merge candidates.

[0115] 2.1.4 Constructing Merge Candidates of Additional Types

[0116] In addition to the space-time Merge candidates, there are two additional types of Merge candidates: combined bidirectional prediction Merge candidates and zero Merge candidates. Combined bidirectional prediction Merge candidates are generated by utilizing space-time Merge candidates. Combined bidirectional prediction Merge candidates are only used for B slices. Combined bidirectional prediction candidates are generated by combining the first reference picture list motion parameters of an initial candidate with the second reference picture list motion parameters of another initial candidate. If the two tuples provide different motion hypotheses, they will form a new bidirectional prediction candidate.

[0117] Fig.15 An example of this process is shown where two candidates in the original list (1510 on the left) with mvL0 and refIdxL0 or mvL1 and refIdxL1 are used to create a combined bi-predictive Merge candidate that is added to the final list (1520 on the right).

[0118] Zero motion candidates are inserted to fill the remaining entries in the Merge candidate list and thus reach the MaxNumMergeCand capacity. These candidates have zero spatial displacement and reference picture index, which starts at zero and increases each time a new zero motion candidate is added to the list. The number of reference frames used by these candidates is 1 and 2 for unidirectional prediction and bidirectional prediction, respectively. In some embodiments, no redundancy check is performed on these candidates.

[0119] 2.1.5 Example of motion estimation regions for parallel processing

[0120] To speed up the encoding process, motion estimation can be performed in parallel, thereby deriving motion vectors for all prediction units within a given region at the same time. Deriving Merge candidates from spatial neighborhoods may interfere with parallel processing because one prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is completed. In order to ease the trade-off between coding efficiency and processing latency, a Motion Estimation Region (MER) can be defined. The size of the MER can be signaled in the picture parameter set (PPS) using the "log2_parallel_merge_level_minus2" syntax element. When a MER is defined, Merge candidates that fall into the same region are marked as unavailable and are therefore not considered in list construction.

[0121] The syntax of the Picture Parameter Set (PPS) Raw Byte Sequence Payload (RBSP) is shown in Table 1, where log2_parallel_merge_level_minus2 plus 2 specifies the value of the variable Log2ParMrgLevel, which is used in the derivation process of the luminance motion vector of the Merge mode and the derivation process of the spatial Merge candidate as specified by the existing video codec standard. The value of log2_parallel_merge_level_minus2 must be in the range of 0 to CtbLog2SizeY-2, including 0 and CtbLog2SizeY-2.

[0122] The variable Log2ParMrgLevel is derived as follows:

[0123] Log2ParMrgLevel=log2_parallel_merge_level_minus2+2

[0124] Note that the value of Log2ParMrgLevel indicates the built-in capability of parallel derivation of Merge candidate lists. For example, when Log2ParMrgLevel is equal to 6, the Merge candidate lists of all prediction units (PUs) and coding units (CUs) contained in a 64×64 block can be derived in parallel.

[0125] Table 1: General Picture Parameter Set RBSP Syntax

[0126]

[0127] 2.2 Embodiment of Motion Vector Prediction in AMVP Mode

[0128] Motion vector prediction uses the spatial-temporal correlation of motion vectors with adjacent PUs, which is used for explicit transmission of motion parameters. It builds a motion vector candidate list by first checking the availability of adjacent PU positions on the left and top time domains, removing redundant candidates and adding zero vectors to make the candidate list a constant length. The encoder can then select the best prediction value from the candidate list and send a corresponding index indicating the selected candidate. Similar to Merge index signaling, the index of the best motion vector candidate is encoded using truncated unary.

[0129] 2.2.1 Example of constructing motion vector prediction candidates

[0130] Fig.16 The derivation process of motion vector prediction candidates is outlined and can be implemented for each reference picture list with refidx as input.

[0131] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For spatial motion vector candidate derivation, based on the location as previously described in Fig.10 The motion vectors of each PU at the five different positions shown in are used to finally derive two motion vector candidates.

[0132] For temporal motion vector candidate derivation, one motion vector candidate is selected from two candidates derived based on two different collocated positions. After generating the first spatio-temporal candidate list, duplicate motion vector candidates in the list are removed. If the number of potential candidates is greater than two, motion vector candidates whose reference picture index within their associated reference picture list is greater than 1 are removed from the list. If the number of spatio-temporal motion vector candidates is less than two, an additional zero motion vector candidate is added to the list.

[0133] 2.2.2 Constructing Spatial Motion Vector Candidates

[0134] In the derivation of the spatial motion vector candidate, from the position as previously described in Fig.10 Consider up to two candidates from the five potential candidates for PU derivation at the positions shown in . Those positions are the same as the positions of the motion merge. The derivation order to the left of the current PU is defined as A 0 , A 1 , and scale A 0 , Zoom A 1 The derivation order of the upper side of the current PU is defined as B 0 , B 1 , B 2 , Zoom B 0 , Zoom B 1 , Zoom B2 For each side, there are thus four cases that can be used as motion vector candidates, two of which do not require the use of spatial scaling, and two of which use spatial scaling. The four different cases are outlined below:

[0135] --No airspace scaling

[0136] (1) Same reference picture list and same reference picture index (same POC)

[0137] (2) Different reference picture lists, but the same reference picture (same POC)

[0138] --Airspace scaling

[0139] (3) Same reference picture list, but different reference pictures (different POC)

[0140] (4) Different reference picture lists and different reference pictures (different POCs)

[0141] First check the non-spatial scaling case, then the case where spatial scaling is allowed. Spatial scaling is considered when the POC is different between the reference pictures of the neighboring PU and the reference picture of the current PU regardless of the reference picture list. If all PUs of the left candidate are unavailable or intra-coded, then scaling is allowed for the upper motion vector to help parallel derivation of the left and upper MV candidates. Otherwise, spatial scaling is not allowed for the upper motion vector.

[0142] like Fig.17 As shown in the example in , for the spatial scaling case, the motion vectors of neighboring PUs are scaled in a similar way to temporal scaling. One difference is that the reference picture list and the index of the current PU are given as input; the actual scaling process is the same as that of temporal scaling.

[0143] 2.2.3 Constructing temporal motion vector candidates

[0144] Except for the reference picture index derivation, all the processes used to derive the temporal Merge candidate are the same as those used to derive the spatial motion vector candidate (e.g. Fig.14 In some embodiments, the reference picture index is signaled to the decoder.

[0145] 2.2.4 Signaling of Merge / AMVP Information

[0146] For AMVP mode, four parts may be signaled in the bitstream, e.g., prediction direction, reference index, MVD, and mv prediction value candidate index, which are described in the context of the syntax shown in Table 2-4. For Merge mode, only the Merge index may need to be signaled.

[0147] Table 2: General Stripe Segment Header Syntax

[0148]

[0149] Table 3: Prediction unit syntax

[0150]

[0151] Table 4: Motion Vector Difference Syntax

[0152]

[0153] The corresponding semantics include:

[0154] five_minus_max_num_merge_cand specifies the maximum number of Merge MVP candidates supported in a stripe subtracted from 5. The maximum number of Merge MVP candidates, MaxNumMergeCand, is derived as follows:

[0155] MaxNumMergeCand=5-five_minus_max_num_merge_cand

[0156] The value of MaxNumMergeCand must be in the range of 1 to 5, inclusive.

[0157] merge_flag[x0][y0] specifies whether the inter prediction parameters of the current prediction unit are inferred from the adjacent inter prediction partition. Array index x0, y0 specifies the position (x0, y0) of the top left luma sample of the considered prediction block relative to the top left luma sample of the picture.

[0158] When merge_flag[x0][y0] is not present, it is inferred as follows:

[0159] --If CuPredMode[x0][y0] is equal to MODE_SKIP, merge_flag[x0][y0] is inferred to be equal to 1.

[0160] --Otherwise, merge_flag[x0][y0] is inferred to be equal to 0.

[0161] merge_idx[x0][y0] specifies the Merge candidate index of the Merge candidate list, where x0, y0 specify the position (x0, y0) of the upper left luma sample of the considered prediction block relative to the upper left luma sample of the picture.

[0162] 3. Example of Inter Prediction Method in Joint Exploration Model (JEM)

[0163] In some embodiments, a reference software known as the Joint Exploration Model (JEM) is used to explore future video codec technologies. In JEM, sub-block based prediction is employed in several codec tools, such as affine prediction, optional temporal motion vector prediction (ATMVP), spatio-temporal motion vector prediction (STMVP), bi-directional optical flow (BIO), frame rate up-conversion (FRUC), locally adaptive motion vector resolution (LAMVR), overlapped block motion compensation (OBMC), local illumination compensation (LIC), and decoder-side motion vector refinement (DMVR).

[0164] 3.1 Example of sub-CU-based motion vector prediction

[0165] In JEM with quadtree plus binary tree (QTBT), each CU can have at most one set of motion parameters for each prediction direction. In some embodiments, two sub-CU level motion vector prediction methods are considered in the encoder by dividing the large CU into sub-CUs and deriving the motion information of all sub-CUs of the large CU. The optional time domain motion vector prediction (ATMVP) method allows each CU to extract multiple sets of motion information from multiple blocks smaller than the current CU in the collocated reference picture. In the space-time motion vector prediction (STMVP) method, the motion vector of the sub-CU is recursively derived by using the time domain motion vector prediction value and the spatial domain neighboring motion vector. In some embodiments, and in order to retain a more accurate motion field for sub-CU motion prediction, motion compression of the reference frame can be disabled.

[0166] 3.1.1 Example of Alternative Temporal Motion Vector Prediction (ATMVP)

[0167] In the ATMVP method, the temporal motion vector prediction (TMVP) method is modified by extracting multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU.

[0168] Fig.18 An example of the ATMVP motion prediction process for CU 1800 is shown. The ATMVP method predicts the motion vector of sub-CU 1801 within CU 1800 in two steps. The first step is to identify the corresponding block 1851 in the reference picture 1850 with the time domain vector. The reference picture 1850 is also called the motion source picture. The second step is to divide the current CU 1800 into sub-CUs 1801 and obtain the motion vector and the reference index of each sub-CU from the block corresponding to each sub-CU.

[0169] In the first step, the reference picture 1850 and the corresponding block are determined by the motion information of the spatial neighboring blocks of the current CU 1800. In order to avoid repeated scanning of the neighboring blocks, the first Merge candidate in the Merge candidate list of the current CU 1800 is used. The first available motion vector and its associated reference index are set to the index of the temporal vector and the motion source picture. In this way, the corresponding block can be identified more accurately compared to TMVP, where the corresponding block (sometimes called a collocated block) is always at the lower right or center position relative to the current CU.

[0170] In one example, if the first Merge candidate comes from the left neighboring block (i.e. Fig.19 A in 1 ), the associated MV and reference picture are used to identify the source block and source picture.

[0171] In the second step, the corresponding blocks of the sub-CU 1851 are identified by the time domain vector in the motion source picture 1850 by adding the time domain vector to the coordinates of the current CU. For each sub-CU, the motion information of its corresponding block (e.g., the minimum motion grid covering the center sample point) is used to derive the motion information of the sub-CU. After the motion information of the corresponding N×N block is identified, it is converted into the motion vector and reference index of the current sub-CU in the same way as the TMVP of HEVC, where motion scaling and other processes apply. For example, the decoder checks whether the low latency condition is met (e.g., the POC of all reference pictures of the current picture is less than the POC of the current picture), and may use the motion vector MVx (e.g., the motion vector corresponding to the reference picture list X) to predict the motion vector MVy of each sub-CU (e.g., where X is equal to 0 or 1, and Y is equal to 1-X).

[0172] 3.1.2 Example of Space-Temporal Motion Vector Prediction (STMVP)

[0173] In the STMVP method, the motion vector of a sub-CU is recursively derived in raster scan order. Fig. 20An example of a CU with four sub-blocks and neighboring blocks is shown. Consider an 8×8 CU 2000 including four 4×4 sub-CU A (2001), sub-CU B (2002), sub-CU D (2003), and sub-CU D (2004). The neighboring 4×4 blocks in the current frame are labeled a (2011), b (2012), c (2013), and d (2014).

[0174] The motion derivation of sub-CU A starts by identifying its two spatial neighbors. The first neighbor is the N×N block above sub-CU A 1101 (block c 2013). If block c (2013) is not available or is intra-coded, check the other N×N blocks above sub-CU A (2001) (starting at block c 2013, from left to right). The second neighbor is the block to the left of sub-CU A 2001 (block b2012). If block b (2012) is not available or is intra-coded, check the other blocks to the left of sub-CU A 2001 (starting at block b 2012, from top to bottom). The motion information obtained from the adjacent blocks of each list is scaled to the first reference frame of the given list. Next, the TMVP of sub-block A 2001 is derived by following the same temporal motion vector predictor (TMVP) derivation process as specified in HEVC. The motion information of the collocated block at block D 2004 is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors are averaged for each reference list separately. The average motion vector is designated as the motion vector of the current sub-CU.

[0175] 3.1.3 Example of Sub-CU Motion Prediction Mode Signaling

[0176] In some embodiments, the sub-CU mode is enabled as an additional Merge candidate, and no additional syntax elements are required to signal the mode. Two additional Merge candidates are added to the Merge candidate list of each CU to represent the ATMVP mode and the STMVP mode. In other embodiments, if the sequence parameter set indicates that ATMVP and STMVP are enabled, up to seven Merge candidates can be used. The encoding logic of the additional Merge candidates is the same as that of the Merge candidates in the HM, which means that for each CU in a P slice or a B slice, two RD checks may be required for the two additional Merge candidates. In some embodiments, for example, JEM, all binary bits of the Merge index are context-encoded by CABAC (Context-based Adaptive Binary Arithmetic Coding). In other embodiments, for example, HEVC, only the first binary bit is context-encoded, and the remaining binary bits are context-bypassed.

[0177] 3.2 Example of Adaptive Motion Vector Difference Resolution

[0178] In some embodiments, when use_integer_mv_flag in the slice header is equal to 0, the motion vector difference (MVD) (between the motion vector of the PU and the predicted motion vector) is signaled in units of quarter luma samples. In JEM, local adaptive motion vector resolution (LAMVR) is introduced. In JEM, MVD can be encoded and decoded in units of quarter luma samples, integer luma samples, or four luma samples. The MVD resolution is controlled at the codec unit (CU) level, and the MVD resolution flag is conditionally signaled for each CU with at least one non-zero MVD component.

[0179] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter luma sample MV precision is not used, another flag is signaled to indicate whether integer luma sample MV precision or quad luma sample MV precision is used.

[0180] When the first MVD resolution flag of a CU is zero or not coded for the CU (meaning that all MVDs in the CU are zero), a quarter luma sample MV resolution is used for the CU. When the CU uses integer luma sample MV precision or quad luma sample MV precision, the MVP in the AMVP candidate list of the CU is rounded to the corresponding precision.

[0181] In the encoder, the CU level RD check is used to determine which MVD resolution is to be used for the CU. That is, for each MVD resolution, the CU level RD check is performed three times. In order to speed up the encoder, the following encoding scheme is applied in JEM:

[0182] -- During RD check of a CU with normal quarter luma sample MVD resolution, store the motion information of the current CU (integer luma sample accuracy). The stored motion information (after rounding) is used as a starting point for further small-scale motion vector refinement during RD check of the same CU with integer luma samples and 4 luma sample MVD resolution, so that the time-consuming motion estimation process is not repeated three times.

[0183] --Conditionally call RD check for CU with 4 luma sample MVD resolution. For a CU, when the RD cost of integer luma sample MVD resolution is much greater than the RD cost of quarter luma sample MVD resolution, skip the RD check for CU's 4 luma sample MVD resolution.

[0184] 3.2.1 Example of AMVP Candidate List Construction

[0185] In JEM, the process is similar to the HEVC design. However, when the current block selects a lower precision MV (e.g., integer precision), a rounding operation can be applied. In the current implementation, after selecting 2 candidates from the spatial position, if both are available, the two candidates are rounded and then pruned.

[0186] 3.3 Example of Pattern Matched Motion Vector Derivation (PMMVD)

[0187] The PMMVD mode is a special Merge mode based on the Frame Rate Up Conversion (FRUC) method. In this mode, the motion information of the block is derived at the decoder side instead of signaling the motion information of the block.

[0188] When the Merge flag of the CU is true, the FRUC flag can be signaled to the CU. When the FRUC flag is false, the Merge index can be signaled and the normal Merge mode can be used. When the FRUC flag is true, an additional FRUC mode flag can be signaled to indicate which method (e.g., bidirectional matching or template matching) to use to derive the motion information of the block.

[0189] On the encoder side, the decision on whether to use the FRUC Merge mode for a CU is based on the RD cost selection made for the normal Merge candidate. For example, multiple matching modes (e.g., bidirectional matching and template matching) are checked for the CU by using the RD cost selection. A matching mode that produces the minimum cost is further compared with other CU modes. If the FRUC matching mode is the most effective mode, the FRUC flag is set to true for the CU and the relevant matching mode is used.

[0190] Generally, the motion derivation process in FRUC Merge mode has two steps: first, a CU-level motion search is performed, and then a sub-CU-level motion refinement is performed. At the CU level, the initial motion vector of the entire CU is derived based on bidirectional matching or template matching. First, a list of MV candidates is generated, and the candidate that produces the minimum matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bidirectional matching or template matching is performed around the starting point. The MV result with the minimum matching cost is used as the MV of the entire CU. Subsequently, the motion information is further refined at the sub-CU level with the derived CU motion vector as the starting point.

[0191] For example, the following derivation process is performed for W×H CU motion information derivation. In the first stage, the MV of the entire W×HCU is derived. In the second stage, the CU is further divided into M×M sub-CUs. The value of M is calculated according to formula (3). D is a predefined partition depth, which is set to 3 by default in JEM. Then the MV of each sub-CU is derived.

[0192]

[0193] Fig.21 An example of bidirectional matching used in a frame rate up conversion (FRUC) method is shown. Bidirectional matching is used to derive motion information of a current CU by finding the closest match between two blocks along the motion trajectory of the current CU (2100) in two different reference pictures (2110, 2111). Under the assumption of a continuous motion trajectory, the motion vectors MV0 (2101) and MV1 (2102) pointing to the two reference blocks are proportional to the temporal distance (e.g., TD0 (2103) and TD1 (2104)) between the current picture and the two reference pictures. In some embodiments, when the current picture 2100 is between the two reference pictures (2110, 2111) in the temporal domain and the temporal distances from the current picture to the two reference pictures are the same, the bidirectional matching becomes a mirror-based bidirectional MV.

[0194] Fig. 22An example of template matching used in a frame rate up conversion (FRUC) method is shown. Template matching can be used to derive motion information of the current CU 2200 by finding the closest match between a template in the current picture (e.g., the top and / or left neighboring blocks of the current CU) and a block in the reference picture 2210 (e.g., the same size as the template). In addition to the above-mentioned FRUC Merge mode, template matching can also be applied to the AMVP mode. In JEM and HEVC, AMVP has two candidates. Using the template matching method, a new candidate can be derived. If the newly derived candidate by template matching is different from the first existing AMVP candidate, it is inserted at the very beginning of the AMVP candidate list, and then the list size is set to two (e.g., by removing the second existing AMVP candidate). When applied to the AMVP mode, only CU level search is applied.

[0195] The MV candidates set at the CU level may include the following: (1) the original AMVP candidates if the current CU is in AMVP mode, (2) all Merge candidates, (3) several MVs in the interpolated MV field (described later), and the top and left neighboring motion vectors.

[0196] When using bidirectional matching, each valid MV of the Merge candidate can be used as input to generate an MV pair assuming bidirectional matching. For example, at reference list A, one valid MV of the Merge candidate is (MVa, ref a ). Then, find the reference image ref of its paired bidirectional MV in other reference list B. b , so that ref a and ref b are located on different sides of the current image in the time domain. If such a ref b is not available in reference list B, then ref b is identified as ref a Different reference, and its temporal distance to the current picture is the smallest one in list B. In determining ref b Afterwards, based on the current image and refa, ref b The temporal distance between them is derived by scaling MVa to MVb.

[0197] In some embodiments, four MVs from the interpolated MV field can also be added to the CU level candidate list. More specifically, the interpolated MVs at the positions (0,0), (W / 2,0), (0,H / 2), and (W / 2,H / 2) of the current CU are added. When FRUC is applied in AMVP mode, the original AMVP candidate is also added to the CU level MV candidate set. In some embodiments, at the CU level, 15 MVs of the AMVP CU and 13 MVs of the Merge CU can be added to the candidate list.

[0198] MV candidates set at the sub-CU level include MVs determined from CU-level searches, (2) top, left, upper-left, and upper-right neighboring MVs, (3) scaled versions of collocated MVs from reference pictures, (4) one or more ATMVP candidates (e.g., up to four), and (5) one or more STMVP candidates (e.g., up to four). The scaled MVs from the reference pictures are derived as follows. The reference pictures in the two lists are traversed. The MV at the collocated position of the sub-CU in the reference picture is scaled as a reference to the starting CU-level MV. The ATMVP and STMVP candidates can be the first four. At the sub-CU level, one or more MVs (e.g., up to 17) are added to the candidate list.

[0199] Generate interpolated MV field Before encoding or decoding a frame, an interpolated motion field is generated for the entire picture based on single-sided ME. The motion field can then be used as a CU-level or sub-CU-level MV candidate later.

[0200] In some embodiments, the motion field of each reference picture in both reference lists is traversed at 4x4 block level. Fig.23 An example of unidirectional motion estimation (ME) 2300 in the FRUC method is shown. For each 4×4 block, if the motion associated with the block through the 4×4 block in the current picture and the block is not assigned any interpolated motion, the motion of the reference block is scaled to the current picture according to the current temporal distance TD0 and TD1 (in the same way as the MV scaling of TMVP in HEVC), and the scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to the 4×4 block, the motion of the block is marked as unavailable in the interpolated motion field.

[0201] Interpolation and matching costs When the motion vector points to a fractional sample position, motion compensation interpolation is required. To reduce complexity, bilinear interpolation can be used for bidirectional matching and template matching instead of conventional 8-tap HEVC interpolation.

[0202] The calculation of the matching cost is slightly different at different steps. When selecting candidates from the candidate set at the CU level, the matching cost can be the absolute sum difference (SAD) of bidirectional matching or template matching. After determining the starting MV, the matching cost C of the bidirectional matching of the sub-CU level search is calculated as follows:

[0203]

[0204] Here, w is a weighting factor. In some embodiments, w can be set to 4 based on experience. MV and MV s Indicates the current MV and the starting MV respectively. SAD can still be used as the matching cost of template matching for sub-CU level search.

[0205] In FRUC mode, MV is derived by using only luma samples. The derived motion will be used for luma and chroma for MC inter prediction. After deciding MV, final MC is performed using 8-tap interpolation filter for luma and 4-tap interpolation filter for chroma.

[0206] MV refinement is a pattern-based MV search with criteria of bidirectional matching cost or template matching cost. In JEM, two search modes are supported—Unrestricted Center-Biased Diamond Search (UCBDS) and Adaptive Cross Search, MV refinement at CU level and sub-CU level, respectively. For both CU level and sub-CU level MV refinement, MV is searched directly with quarter luma sample MV accuracy, and followed by one-eighth luma sample MV refinement. The search range of MV refinement at CU step and sub-CU step is set equal to 8 luma samples.

[0207] In the Bidirectional Matching Merge mode, bidirectional prediction is applied because the motion information of the CU is derived based on the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. In the Template Matching Merge mode, the encoder can choose between unidirectional prediction from list 0, unidirectional prediction from list 1, or bidirectional prediction of the CU. The selection can be based on the template matching cost as follows:

[0208] If costBi <= factor * min (cost0, cost1)

[0209] Use bidirectional prediction;

[0210] Otherwise, if cost0 <= cost1

[0211] Use one-way prediction from list 0;

[0212] otherwise,

[0213] Use the one-way prediction from List 1;

[0214] Here, cost0 is the SAD of list 0 template matching, cost1 is the SAD of list 1 template matching, and costBi is the SAD of bi-prediction template matching. For example, when the value of factor is equal to 1.25, it means that the selection process is biased towards bi-prediction. Inter prediction direction selection can be applied to the CU level template matching process.

[0215] 3.4 Example of Decoder-side Motion Vector Refinement (DMVR)

[0216] In a bidirectional prediction operation, to predict a block region, two prediction blocks formed using motion vectors (MVs) from list 0 and MVs from list 1, respectively, are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of the bidirectional prediction are further refined by a bidirectional template matching process. Bidirectional template matching is applied in the decoder to perform a distortion-based search between the bidirectional template and the reconstructed samples in the reference picture to obtain the refined MV without sending additional motion information.

[0217] In DMVR, a bidirectional template is generated from the initial MV0 of list 0 and MV1 of list 1 as a weighted combination (i.e., average) of two prediction blocks, as shown in Fig.24 As shown. The template matching operation includes calculating the cost metric between the generated template and the sample area (around the initial prediction block) in the reference picture. For each of the two reference pictures, the MV that produces the minimum template cost is regarded as the updated MV of the list to replace the original MV. In JEM, nine MV candidates are searched for each list. The nine MV candidates include the original MV and 8 surrounding MVs, in which one luma sample is offset to the original MV in the horizontal or vertical direction or in both directions. Finally, the two new MVs (i.e., Fig.24 MV0' and MV1' shown) are used to generate the final bidirectional prediction result. The sum of absolute differences (SAD) is used as the cost metric.

[0218] DMVR is applied to the Merge mode of bi-prediction, where one MV comes from the past reference picture and the other MV comes from the future reference picture, without sending additional syntax elements. In JEM, DMVR is not applied when LIC, affine motion, FRUC or sub-CU Merge candidate is enabled for the CU.

[0219] 3.5 Example of Merge / Skip Mode with Bidirectional Match Refinement

[0220] The Merge candidate list is first constructed by inserting the motion vectors and reference indexes of spatially adjacent and temporally adjacent blocks into the candidate list using redundancy check until the number of available candidates reaches the maximum candidate size of 19. The Merge candidate list for Merge / skip mode is constructed by inserting spatial candidates, temporal candidates, affine candidates, Advanced Temporal MVP (ATMVP) candidates, Spatial Temporal MVP (STMVP) candidates, and additional candidates used in HEVC (combined candidates and zero candidates) according to a predetermined insertion order, and Fig.25 In the context of the numbered blocks shown in:

[0221] (1) Spatial candidates for blocks 1-4

[0222] (2) Extrapolated affine candidates for blocks 1-4

[0223] (3)ATMVP

[0224] (4)STMVP

[0225] (5) Virtual Affine Candidates

[0226] (6) Spatial candidates (block 5) (used only when the number of available candidates is less than 6)

[0227] (7) Extrapolate affine candidates (block 5)

[0228] (8) Temporal candidates (as derived in HEVC)

[0229] (9) Non-adjacent spatial candidates, followed by extrapolated affine candidates (blocks 6 to 49)

[0230] (10) Candidates for combination

[0231] (11) Zero Candidate

[0232] It can be noted that the IC flag is also inherited from the Merge candidates except STMVP and Affine. In addition, for the first four spatial candidates, the bidirectional prediction candidates are inserted before the candidates with unidirectional prediction.

[0233] 3.6 Example Implementation of Shared Merge List

[0234] The same Merge candidate list of all leaf codec units (CUs) of an ancestor node in a shared CU partition tree is proposed to enable parallel processing of small skip / merge codec CUs. The ancestor node is named as a Merge shared node. Assuming that the Merge shared node is a leaf CU, a shared Merge candidate list is generated at the Merge shared node.

[0235] For type 2 definition, during the parsing phase of decoding, a Merge shared node will be determined for each CU inside a CTU; in addition, the Merge shared node is the ancestor node of a leaf CU, which must meet the following two criteria:

[0236] Merge shared node size is equal to or greater than the size threshold;

[0237] In a Merge shared node, one of the sub-CU sizes is smaller than the size threshold.

[0238] In addition, it must be ensured that no samples of the Merge shared node are outside the picture boundary. During the parsing phase, if an ancestor node satisfies criteria (1) and (2), but has some samples outside the picture boundary, then the ancestor node will not be a Merge shared node, and it continues to look for a Merge shared node for its child CU.

[0239] Fig.26 An example of the difference between type 1 definition and type 2 definition is shown. In this example, the parent node is trifurcated into 3 child nodes. The size of the parent node is 128. For type 1 definition, the 3 child nodes will each be a Merge shared node. But for type 2 definition, the parent node is a Merge shared node.

[0240] The proposed shared Merge candidate list algorithm supports translational Merge (including Merge mode and triangle Merge mode, and also supports history-based candidates) and sub-block-based Merge mode. For all kinds of Merge modes, the behavior of the shared Merge candidate list algorithm looks basically the same, and it just generates candidates at the Merge shared node when it assumes that the Merge shared node is a leaf CU. It has 2 main benefits. The first benefit is to achieve parallel processing of the Merge mode, and the second benefit is to share all calculations of all leaf CUs to the Merge shared node. Therefore, it significantly reduces the hardware cost of all Merge modes of the hardware codec. Through the proposed shared Merge candidate list algorithm, encoders and decoders can easily support parallel encoding of the Merge mode, and it alleviates the cycle budget problem of the Merge mode.

[0241] 3.7-chipset example

[0242] Existing implementations have been adopted where slices are removed in favor of slice groups, and the HEVC syntax element slice_address is replaced by tile_group_address in the tile_group_header (if there is more than one slice in the picture) as the address of the first slice in the slice group.

[0243] 4. Disadvantages of existing implementation methods

[0244] The current HEVC design can use the correlation between the current block and its neighboring blocks (immediately adjacent to the current block) to better encode and decode motion information. However, the neighboring blocks may correspond to different objects with different motion trajectories. In this case, the prediction efficiency from its neighboring blocks is not high.

[0245] Prediction based on motion information of non-neighboring blocks can bring additional coding gain, at the cost of storing all motion information (usually at 4×4 level) in cache, which significantly increases the complexity of hardware implementation.

[0246] 5. Example Embodiments of the Disclosed Technology

[0247] Embodiments of the currently disclosed technology overcome the shortcomings of existing implementations, thereby providing video codecs with higher codec efficiency. LUT-based motion vector prediction based on the disclosed technology that can enhance existing and future video codec standards is illustrated in the following examples described for various implementations. The examples of the disclosed technology provided below explain the general concepts and are not meant to be interpreted as limiting. In the examples, the various features described in these examples can be combined unless explicitly indicated to the contrary.

[0248] Definition of Lookup Table

[0249] 1. A lookup table may contain one or more motion candidates, where each candidate is associated with its motion information.

[0250] a. The motion information of the motion candidate here may include part or all of the prediction direction, reference index / picture, motion vector, LIC flag, affine flag, MVD precision, and MVD value.

[0251] b. The motion information may also include block position information and / or block shape to indicate where the motion information comes from.

[0252] c. A counter may also be assigned to each lookup table.

[0253] i. At the start of encoding / decoding a picture / slice / LCU (CTU) line / slice, the counter is initialized to zero.

[0254] ii. In one example, after encoding / decoding a CTU / CTB / CU / CB / PU / specific region size (e.g., 8×8 or 16×16), the counter is updated.

[0255] iii. In one example, each time a candidate is added to the lookup table, a counter is incremented by one.

[0256] iv. In one example, the counter should be no larger than the table size (the number of allowed motion candidates).

[0257] v. Alternatively, a counter may be used to indicate how many motion candidates have been attempted to be added to the lookup table (some of them are in the lookup table, but may be removed from the table later). In this case, the counter may be larger than the table size.

[0258] d. The table size (number of allowed motion candidates) and / or the number of tables can be fixed or adaptive. The table size can be the same for all tables or different for different tables.

[0259] i. Alternatively, different sizes may be used for different lookup tables (eg, 1 or 2).

[0260] ii. In one example, the table size and / or the number of tables may be predefined.

[0261] iii. In one example, the table size and / or the number of tables may be signaled in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a slice header, a codec tree unit (CTU), a codec tree block (CTB), a codec unit (CU) or a prediction unit (PU), or an area covering multiple CTUs / CTBs / CUs / PUs.

[0262] iv. The table size and / or the number of tables may also depend on the slice type, the temporal layer index of the picture, and the Picture Order Count (POC) distance between a slice and the nearest intra slice.

[0263] e. Assuming that there are N tables for encoding and decoding threads, N*P tables may be needed to encode and decode a slice, where P indicates the number of LCU rows or the number of slices.

[0264] i. Alternatively, only P tables may be needed to encode and decode a slice, where P indicates the number of LCU rows, where only one lookup table is used per LCU row, and N may be greater than 1 even when slices are disabled.

[0265] Lookup table selection

[0266] 2. To encode or decode a block, some or all of the motion candidates from a lookup table may be checked sequentially.

[0267] a. Alternatively, motion candidates from multiple lookup tables may be checked sequentially.

[0268] b. The lookup table index may be signaled in a CTU, CTB, CU or PU, or in a region covering multiple CTUs / CTBs / CUs / PUs.

[0269] 3. The choice of the lookup table may depend on the location of the block.

[0270] a. It can depend on the CTU address of the coverage block. Here, we take two lookup tables (dual lookup table, DLUT) as an example to illustrate the idea:

[0271] i. If the block is located in one of the first M CTUs in a CTU row, the first lookup table may be used to encode and decode the block, while for blocks located in the remaining CTUs in the CTU row, the second lookup table may be utilized.

[0272] ii. If the block is located in one of the first M CTUs in the CTU row, the motion candidates in the first lookup table may be checked first for encoding and decoding the block, and if there are not enough candidates in the first table, the second lookup table may be used. For blocks in the remaining CTUs in the CTU row, the second lookup table may be used.

[0273] iii. Alternatively, for blocks in the remaining CTUs in the CTU row, the motion candidates in the second lookup table may be checked first for encoding and decoding the blocks, and if there are not enough candidates in the second table, the first lookup table may also be used.

[0274] b. It may depend on the distance between the location of the block and a location associated with a motion candidate in one or more lookup tables.

[0275] i. In one example, if one motion candidate is associated with a smaller distance to the block to be encoded, then the motion candidate may be checked earlier than another motion candidate.

[0276] Use of Lookup Table

[0277] 4. The total number of motion candidates in the lookup table to be checked can be predefined.

[0278] a. It may also depend on codec information, block size, block shape, etc. For example, for AMVP mode, only m motion candidates may be checked, while for Merge mode, n motion candidates may be checked (eg, m=2, n=44).

[0279] b. In one example, the total number of motion candidates to be checked may be signaled in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a slice header, a codec tree unit (CTU), a codec tree block (CTB), a codec unit (CU) or a prediction unit (PU), or an area covering multiple CTUs / CTBs / CUs / PUs.

[0280] 5. The motion candidate(s) included in the lookup table may be directly inherited by the block.

[0281] a. They can be used for Merge mode encoding and decoding, i.e., motion candidates can be checked during the Merge candidate list derivation process.

[0282] b. They can be used for affine Merge mode encoding and decoding.

[0283] i. If the affine flag of a motion candidate in the lookup table is 1, it can be added as an affine Merge candidate.

[0284] c. They can be used for other kinds of Merge modes, such as sub-block Merge mode, affine Merge mode, triangle Merge mode, inter-intra Merge mode, and Merge with MVD (MMVD) mode.

[0285] d. Checking for motion candidates in the lookup table can be enabled when:

[0286] i. After inserting the TMVP candidate, the Merge candidate list is not full;

[0287] ii. After checking a specific spatial neighboring block for spatial Merge candidate derivation, the Merge candidate list is not full;

[0288] iii. After all airspace merge candidates are merged, the merge candidate list is not full;

[0289] iv. After the combined bidirectional prediction Merge candidate, the Merge candidate list is not full;

[0290] v. When the number of spatial or temporal Merge candidates (for example, including adjacent spatial domains and non-adjacent spatial domains, TMVP, STMVP, ATMVP, etc.) from other coding and decoding methods (for example, the Merge derivation process of HEVC design or JEM design) that have been placed in the Merge candidate list is less than the maximum allowed Merge candidate minus a given threshold.

[0291] (a) In one example, the threshold is set to 1 or 0.

[0292] (b) Alternatively, the threshold may be signaled or predefined in the SPS / PPS / sequence, picture, slice header / slice.

[0293] (c) Alternatively, the threshold may be adaptively changed from block to block. For example, it may depend on codec block information such as block size / block shape / slice type, and / or on the number of available spatial or temporal Merge candidates.

[0294] (d) In another example, when the number of Merge candidates of a specific category that have been put into the Merge candidate list is less than the maximum allowed Merge candidate minus a given threshold. A "Merge candidate of a specific category" may be a spatial candidate as in HEVC, or a non-adjacent Merge candidate.

[0295] vi. Before adding motion candidates to the Merge candidate list, pruning can be applied.

[0296] (a) In one example, the motion candidate may be pruned from all or part of the available spatial or temporal Merge candidates (e.g., including adjacent spatial and non-adjacent spatial, TMVP, STMVP, ATMVP, etc.) in the Merge candidate list from other coding methods.

[0297] (b) Motion candidates and sub-block based motion candidates may not be pruned, for example, ATMVP, STMVP.

[0298] (c) In one example, the current motion candidate and all or part of the available motion candidates in the Merge candidate list (inserted before the current motion candidate) may be pruned.

[0299] (d) The number of pruning operations related to motion candidates (i.e., how many times the motion candidate needs to be compared with other candidates in the Merge list) can depend on the number of available spatial or temporal Merge candidates. For example, when checking a new motion candidate, if there are M candidates available in the Merge list, the new motion candidate can be compared only with the first K (K <= M) candidates. If the pruning function returns false (e.g., different from any of the first K candidates), the new motion candidate is considered different from all of the M candidates, and it can be added to the Merge candidate list. In one example, K is set to min(K, 2).

[0300] (e) In one example, only the newly added motion candidates are compared with the first N candidates in the Merge candidate list. For example, N = 3, 4 or 5. N can be signaled from the encoder to the decoder.

[0301] (f) In one example, only the new motion candidates to be checked are compared with the last N candidates in the Merge candidate list. For example, N = 3, 4 or 5. N can be signaled from the encoder to the decoder.

[0302] (g) In one example, how to select a previously added candidate in the list from the table to be compared with a new motion candidate may depend on where the previously added candidate was derived from.

[0303] a. In one example, the motion candidates in the lookup table can be compared with the motion candidates derived from given temporal and / or spatial neighboring blocks.

[0304] b. In one example, different entries of motion candidates in the lookup table may be compared with different (ie, derived from different positions) previously added candidates.

[0305] e. Checking of motion candidates in the lookup table may be enabled before checking other Merge (or Affine Merge or other inter-coding methods) candidates such as those derived from adjacent / non-adjacent spatial or temporal blocks.

[0306] f. When there is at least one motion candidate in the lookup table, checking of the motion candidates in the lookup table may be enabled.

[0307] 6. The motion candidate(s) included in the lookup table may be used as prediction values ​​for encoding and decoding the motion information of the block.

[0308] a. They can be used for AMVP mode encoding and decoding, i.e., motion candidates can be checked during the AMVP candidate list derivation process.

[0309] b. They can be used for Symmetric Motion Vector Difference (SMVD) coding, where only a partial MVD (such as the MVD signaled for only one reference picture list and derived from another reference picture list) is used.

[0310] c. They can be used for Symmetric Motion Vector (SMV) coding, where only partial MVs (such as signaled only for one reference picture list and derived from another reference picture list) are used.

[0311] d. Checking for motion candidates in the lookup table can be enabled when:

[0312] i. After inserting the TMVP candidate, the AMVP candidate list is not full;

[0313] ii. The AMVP candidate list is not full after selecting from spatial neighbors and pruning just before inserting TMVP candidates;

[0314] iii. When there is no AMVP candidate from an upper neighboring block that is not scaled and / or when there is no AMVP candidate from a left neighboring block that is not scaled

[0315] iv. After inserting a specific AMVP candidate, the AMVP candidate list is not full;

[0316] v. Before adding a motion candidate to the AMVP candidate list, pruning may be applied.

[0317] vi. Similar rules as mentioned in bullet point 5.(c)(d) can be applied to the AMVP model.

[0318] e. Checking of motion candidates may be enabled before checking other AMVP (or SMVD / SMV / affine inter or other inter codec methods) candidates such as those derived from adjacent / non-adjacent spatial or temporal blocks.

[0319] f. When there is at least one motion candidate in the lookup table, checking of motion candidates may be enabled.

[0320] g. Check motion candidates that have the same reference picture as the current reference picture.

[0321] i. Furthermore, alternatively, motion candidates with reference pictures different from the current reference picture are also checked (MV is scaled).

[0322] ii. Alternatively, first check all motion candidates having the same reference picture as the current reference picture, and then check motion candidates having a different reference picture than the current reference picture.

[0323] iii. Alternatively, check motion candidates in the same way in Merge.

[0324] iv. When a motion candidate is a bidirectional prediction candidate, if the current target reference picture list is X, the reference pictures of reference picture list X (such as the reference picture index or picture order counter of the reference picture) can be checked first, and then the reference pictures of reference picture list Y (Y!=X, for example, Y=1-X) can be checked.

[0325] v. Alternatively, when a motion candidate is a bidirectional prediction candidate, if the current target reference picture list is X, the reference pictures of reference picture list Y (Y!=X, for example, Y=1-X) may be checked first, and then the reference pictures of reference picture list X may be checked.

[0326] vi. Alternatively, reference pictures of reference picture list X associated with all motion candidates to be checked may be checked before reference pictures of reference picture list Y (Y!=X, for example, Y=1-X) associated with all motion candidates to be checked.

[0327] 7. The motion candidates in the lookup table can be used to derive other candidates, and the derived candidates can be used to encode and decode the block.

[0328] a. In one example, an offset is added to the motion vector of a motion candidate in a lookup table to derive a new candidate.

[0329] b. In one example, a uni-directionally predicted motion candidate in the lookup table may be used to derive another uni-directionally predicted motion candidate with the same reference picture list index, such as via scaling the motion vector to another reference picture in the same reference picture list.

[0330] c. In one example, a uni-directionally predicted motion candidate in the lookup table can be used to derive another uni-directionally predicted motion candidate with a different reference picture list index, such as via scaling the motion vector to one reference picture of another reference picture list.

[0331] d. In one example, a uni-directional prediction motion candidate in the lookup table can be used to derive a bi-directional prediction motion candidate via scaling the motion vector to a reference picture of another reference picture list.

[0332] e. In one example, different motion candidates from the lookup table can be used to calculate a new motion candidate. For example, an average of two or more motion candidates can be derived as a new motion candidate.

[0333] 8. The order of checking motion candidates in the lookup table is defined as follows (assuming that K

[0334] (K>=1) motion candidates):

[0335] a. Look up the last K motion candidates in the table.

[0336] b. When K>=L, the first K%L candidates, where L is the size of the lookup table

[0337] c. When K>=1, search all candidates in the table (L candidates).

[0338] d. Also, alternatively, based on descending order of motion candidate index.

[0339] e. Furthermore, alternatively, based on ascending order of motion candidate index.

[0340] f. Alternatively, K motion candidates are selected based on candidate information such as distances of locations associated with the motion candidates and the current block.

[0341] i. In one example, select the K closest motion candidates.

[0342] ii. In one example, when calculating the distance, the candidate information may also take into account the block shape.

[0343] g. In one example, the order of checking K of the motion candidates from a table of L candidates can be defined as: Select the candidate with index equal to a in order 0 、a 0 +T 0 、a 0 +T 0 +T 1 、a 0 +T 0 +T 1 +T 2 , …, a 0 +T 0 +T 1 +T 2 +..+T K-1 Among the candidates, a 0 and T i (i is 0...K-1) is an integer value.

[0344] i. In one example, a 0 is set to 0 (i.e. the first entry in the table of motion candidates). Alternatively, a 0 is set to (K–L / K). The arithmetic operation " / " is defined as integer division with the result truncated toward zero. Alternatively, a 0 Can be set to any integer between 0 and L / K.

[0345] 1. Alternatively, a 0 The value of may depend on the coding information of the current block and the adjacent blocks.

[0346] ii. In one example, all intervals T i (i is 0...K-1) the same, such as L / K. The arithmetic operation " / " is defined as integer division with the result truncated towards zero.

[0347] iii. In one example, (K, L, a 0 ,T i ) is set to (4,16,0,4) or (4,12,0,3) or (4,8,0,1) or (4,16,3,4) or (4,12,2,3) or (4,8,1,2). For all i, T i same.

[0348] iv. Such a method can only be applied when K is smaller than L.

[0349] v. Furthermore, alternatively, when K is greater than or equal to a threshold, bullet 7.c. may be applied. The threshold may be defined as L, or it may depend on K or vary adaptively from block to block. In one example, the threshold may depend on the number of available motion candidates in the list before adding a new motion candidate from the lookup table.

[0350] h. In one example, the order of checking K of the motion candidates from a table of L candidates can be defined as: Select the candidate with index equal to a in order 0 、a 0 -T 0 、a 0 -T 0 -T 1 、a 0 -T 0 -T 1 -T 2 , …, a 0 -T 0 -T 1 -T 2 -..-T K-1 Among the candidates, a 0 and T i (i is 0...K-1) is an integer value.

[0351] i. In one example, a 0 is set to L-1 (i.e. the last entry of the motion candidate in the table). Alternatively, a 0 Set to any integer between L-1-L / K and L-1.

[0352] ii. In one example, all intervals T i (i is 0...K-1) the same, such as L / K.

[0353] iii. In one example, (K, L, a 0 ,T i ) is set to (4,16,L-1,4) or (4,12,L-1,3) or (4,8,L-1,1) or (4,16,L-4,4) or (4,12,L-3,3) or (4,8,L-2,2). For all i, T i same.

[0354] iv. Such a method can only be applied when K is smaller than L.

[0355] v. Furthermore, alternatively, when K is greater than or equal to a threshold, bullet 7.c. may be applied. The threshold may be defined as L, or it may depend on K or vary adaptively from block to block. In one example, the threshold may depend on the number of available motion candidates in the list before adding a new motion candidate from the lookup table.

[0356] i. How many and / or how to select motion candidates from the lookup table may depend on codec information such as block size / block shape.

[0357] i. In one example, for smaller block sizes, instead of selecting the last K motion candidates, other K motion candidates may be selected (not starting from the last one).

[0358] ii. In one example, the codec information may be AMVP or Merge mode.

[0359] iii. In one example, the codec information may be an affine mode, a non-affine AMVP mode, or a non-affine Merge mode.

[0360] iv. In an example, the codec information may be an affine AMVP (inter) mode, an affine Merge mode, or a non-affine AMVP mode or a non-affine Merge mode.

[0361] v. In one example, the codec information may be a current picture reference (CPR) mode or not a CPR mode. In some examples, the CPR mode is also referred to as an intra block copy (IBC) mode.

[0362] vi. Alternatively, how to select a motion candidate from the lookup table may also depend on the number of motion candidates in the lookup table, and / or the number of available motion candidates in the list before adding a new motion candidate from the lookup table.

[0363] j. In one example, the maximum number of motion candidates in the lookup table to be checked (ie, which can be added to the Merge / AMVP candidate list) can depend on the number of available motion candidates in the lookup table (given by N avaiMCinLUT ), and / or the maximum allowed motion candidates to be added (denoted by NUM maxMC ) (which may be predefined or signaled), and / or the number of available candidates in the candidate list before checking the candidates from the lookup table (denoted by N avaiC express).

[0364] i. In one example, the maximum number of motion candidates in the lookup table to be checked is set to (NavaiMCinLUT ,NUM maxMC ,N avaiC ) is the minimum value of .

[0365] ii. Alternatively, the maximum number of motion candidates in the lookup table to be checked is set to (N avaiMCinLUT ,NUM maxMC -N avaiC ) is the minimum value of .

[0366] iii. In one example, N avaiC Indicates the number of insertion candidates derived from spatial or temporal (adjacent and / or non-adjacent) neighboring blocks. avaiC The number of sub-block candidates (such as AMTVP, STMVP) is counted in .

[0367] iv.NUM maxMC It may depend on the codec mode. For example, for Merge mode and AMVP mode, NUM maxMC Can be set to different values. In one example, for Merge mode, NUM maxMC Can be set to 4, 6, 8, 10, etc. For AMVP mode, NUM maxMC Can be set to 1, 2, 4, etc.

[0368] v.Alternatively, NUM maxMC It may depend on other codec information, such as block size, block shape, slice type, etc.

[0369] k. The order in which different lookup tables are checked is defined in the next subsection Use of Lookup Tables.

[0370] l. Once the Merge / AMVP candidate list reaches the maximum allowed number of candidates, the checking process will terminate.

[0371] m. Once the Merge / AMVP candidate list reaches the maximum allowed number of candidates minus a threshold (Th), the checking process will terminate. In one example, Th may be predefined as a positive integer value, e.g., 1, 2, or 3. Alternatively, Th may be adaptively changed from block to block. Alternatively, Th may be signaled in the SPS / PPS / slice header, etc. Alternatively, Th may also depend on block shape / block size / codec mode, etc. Alternatively, Th may depend on how many available candidates are before adding motion candidates from the LUT.

[0372] n. Alternatively, once the number of added motion candidates reaches the maximum allowed number of motion candidates, it will terminate. The maximum allowed number of motion candidates can be signaled or predefined. Alternatively, the maximum allowed number of motion candidates can also depend on block shape / block size / coding mode, etc.

[0373] o. A syntax element may be signaled in the SPS, PPS, slice header, or slice header to indicate the table size and the number of motion candidates allowed to be checked (ie, K=L).

[0374] 9. When motion candidates from the lookup table are used to encode and decode a block (eg, Merge / AMVP mode), the insertion order of the motion candidates can be adaptively changed.

[0375] a. In one example, the insertion order of motion candidates may depend on the coding information of a block, for example, it may be different for different modes.

[0376] i. In one example, for Merge mode, they can be added after the TMVP candidates.

[0377] ii. In one example, the insertion order of motion candidates may be added before spatial candidates.

[0378] iii. In one example, for current picture reference (CPR) mode, they can be added after the spatial candidates.

[0379] b. In one example, the insertion order of motion candidates can be adaptively changed from block to block or from slice to slice or from slice group to slice group or from one video data processing unit to another video data processing unit.

[0380] c. In one example, the insertion order of motion candidates may be different for different block dimensions.

[0381] d. In one example, the insertion order of motion candidates may be different for different segmentation depths.

[0382] e. In one example, the insertion order of motion candidates may depend on the slice / slice group / slice / picture type.

[0383] f. In one example, the insertion order of motion candidates may depend on the low latency check flag.

[0384] g. In one example, the insertion order of motion candidates may depend on the low latency check flag.

[0385] h. How pruning is applied and / or how many motion candidates in the lookup tables are to be checked during candidate list construction and / or how many lookup tables there are and / or the maximum number of motion candidates in the lookup tables that can be utilized can be adaptively changed.

[0386] 10. The use of the lookup table for motion information encoding and decoding of a block may be enabled / disabled by signaling in an SPS, PPS, slice header, slice header, CTU, CTB, CU or PU, or in an area covering multiple CTUs / CTBs / CUs / PUs.

[0387] 11. Whether prediction from the lookup table is applied may also depend on codec information. When inference is not applied to a block, additional signaling of an indication of prediction is skipped. Alternatively, when inference is not applied to a block, no motion candidates from the lookup table need to be accessed, and the check of relevant motion candidates is omitted.

[0388] a. Whether to apply prediction from a lookup table may depend on the block size / block shape. In one example, for smaller blocks, such as 4x4, 8x4 or 4x8 blocks, prediction from a lookup table is not allowed to be performed.

[0389] b. Whether to apply prediction from a lookup table may depend on whether the block is encoded in AMVP mode or Merge mode. In one example, for AMVP mode, prediction from a lookup table is not allowed to be performed.

[0390] c. Whether to apply prediction from a lookup table may depend on whether the block is encoded with affine motion or other kind of motion (such as translational motion). In one example, for affine mode, prediction from a lookup table is not allowed to be performed.

[0391] 12. The motion candidates from the lookup table in a previously encoded or decoded frame / slice / slice can be used to predict the motion information of a block in a different frame / slice / slice.

[0392] a. In one example, only the lookup table associated with the reference picture of the current block can be used to encode and decode the current block.

[0393] b. In one example, only the lookup tables associated with pictures having the same slice type and / or the same quantization parameter of the current block may be used to encode and decode the current block.

[0394] Update of lookup table

[0395] 13. After encoding or decoding a block with motion information (ie, intra BC mode, inter encoding or decoding mode), one or more lookup tables may be updated.

[0396] a. In one example, whether to update the lookup table may reuse the rule for selecting the lookup table, for example, when a lookup table may be selected for encoding / decoding the current block, the selected lookup table may be further updated after encoding / decoding the block.

[0397] b. The lookup table to be updated may be selected based on the codec information and / or location of the block / LCU.

[0398] c. If the motion information is notified by direct signaling (such as AMVP mode, normal /

[0399] If the block is encoded or decoded in the MMVD mode of the affine inter mode, the AMVR mode of the normal / affine inter mode), the motion information of the block can be added to the lookup table.

[0400] i. Alternatively, if the block is coded with motion information directly inherited from a spatially neighboring block that has not undergone any refinement (eg, a spatial Merge candidate that has not undergone refinement), the motion information of the block should not be added to the lookup table.

[0401] ii. Alternatively, if the block is coded with motion information directly inherited from spatially neighboring blocks that have been refined (such as DMVR, FRUC), the motion information of the block should not be added to any lookup table.

[0402] iii. Alternatively, if the block is encoded with motion information directly inherited from a motion candidate stored in a lookup table, the motion information of the block should not be added to any lookup table.

[0403] iv. In one example, such motion information may be added directly to the lookup table, such as to the last entry of the table or to the entry used to store the next available motion candidate.

[0404] v. Alternatively, such motion information may be added directly to the lookup table without pruning, eg, without any pruning.

[0405] vi. Alternatively, such motion information can be used to reorder the lookup table.

[0406] vii. Alternatively, such motion information may be used to update the lookup table with limited pruning (eg, compared to the most recent one in the lookup table).

[0407] d. M (M>=1) representative positions within the block are selected, and the motion information associated with the representative positions is used to update the lookup table.

[0408] i. In one example, representative locations are defined as the four corner locations within a block (e.g., Fig. 27One of C0-C3).

[0409] ii. In one example, the representative location is defined as the center location within the block (e.g., Fig. 27 of Ca-Cd).

[0410] iii. When sub-block prediction is not allowed for a block, M is set to 1.

[0411] iv. When sub-block prediction is allowed for a block, M can be set to 1 or the total number of sub-blocks or any other value between [1, number of sub-blocks] (excluding endpoints).

[0412] v. Alternatively, when sub-block prediction is allowed for a block, M can be set to 1, and the selection of the representative sub-block is based on

[0413] 1. The frequency of motion information used,

[0414] 2. Is it a bidirectional prediction block?

[0415] 3. Based on reference image index / reference image

[0416] 4. Motion vector difference compared to other motion vectors (e.g., select the largest MV difference)

[0417] 5. Other codec information.

[0418] e. When M (M>=1) representative position sets are selected to update the lookup table, further conditions may be checked before adding them as additional motion candidates to the lookup table.

[0419] i. Pruning a new set of motion information may be applied to existing motion candidates in the lookup table.

[0420] ii. In one example, the new motion information set should not be identical to any or part of the existing motion candidates in the lookup table.

[0421] iii. Alternatively, for the same reference picture from the new motion information set and an existing motion candidate, the MV difference should be no less than one / multiple thresholds. For example, the horizontal and / or vertical components of the MV difference should be greater than 1 pixel distance.

[0422] iv. Alternatively, when K>L, the new motion information set is pruned with only the last K candidates or the first K%L existing motion candidates to allow reactivation of old motion candidates.

[0423] v. Alternatively, no pruning is applied.

[0424] f. If M motion information sets are used to update the lookup table, the corresponding counter should be increased by M.

[0425] g. Assuming that the counter of the lookup table to be updated is denoted by K before encoding and decoding the current block, after encoding and decoding the block, for a selected motion information set (using the above method), it is added as an additional motion candidate with an index equal to K%L (where L is the lookup table size). Fig.28A and Fig.28B An example is shown in .

[0426] i. Alternatively, it is added as an additional motion candidate with index equal to min(K+1, L-1). Also, alternatively, if K>=L, the first motion candidate (index equal to 0) is removed from the lookup table and the subsequent K candidate indices are reduced by 1.

[0427] ii. For the above two methods (adding the new motion candidate to the entry with index equal to K%L, or adding it to the entry with index equal to min(K+1,L-1)), they both try to keep the latest few sets of motion information from the previously encoded or decoded blocks, regardless of whether there are the same / similar motion candidates.

[0428] iii. Alternatively, when a new motion information set is added to the LUT as a motion candidate, a redundancy check is applied first. In this case, the LUT will retain the latest few motion information sets from previously encoded or decoded blocks, however, redundant motion information sets can be removed from the LUT. Such a method is called a LUT update method based on redundancy removal.

[0429] 1. If there are redundant motion candidates in the LUT, the counter associated with the LUT may not be incremented or decremented.

[0430] 2. Redundancy check can be defined as a pruning process during the Merge candidate list construction process, for example, checking whether the reference pictures / reference picture indexes are the same and whether the motion vector differences are within the range or the same.

[0431] 3. If a redundant motion candidate is found in the LUT, move the redundant motion candidate from its current position to the last position of the LUT.

[0432] a. Similarly, if a redundant motion candidate is found in the LUT, it is removed from the LUT. In addition, all motion candidates inserted into the LUT after the redundant motion candidate are shifted forward to refill the removed entries of the redundant motion candidate. After the shift, the new motion candidate is added to the LUT.

[0433] b. In this case, the counter remains unchanged.

[0434] c. Once redundant motion candidates are identified in the LUT, the redundancy checking process is terminated.

[0435] 4. Multiple redundant motion candidates may be identified. In this case, all of them are removed from the LUT. In addition, all of the remaining motion candidates may be moved forward in sequence.

[0436] a. In this case, the counter is decremented (the number of redundant motion candidates minus 1).

[0437] b. After maxR redundant motion candidates are identified (maxR is a positive integer variable), the redundancy check process is terminated.

[0438] 5. The redundancy checking process may start from the first motion candidate to the last motion candidate (ie, in the order of addition to the LUT, in the order of the decoding process of the blocks from which the motion information comes).

[0439] 6. Alternatively, when there are redundant motion candidates in the LUT, instead of removing one or more of the redundant motion candidates from the LUT, virtual motion candidates may be derived from the redundant motion candidates and used to replace the redundant motion candidates.

[0440] a. A virtual motion candidate may be derived from a redundant motion candidate by adding (multiple) offsets to the horizontal and / or vertical components of one or more motion vectors; or the average of two motion vectors if pointing to the same reference picture. Alternatively, a virtual motion candidate may be derived from any function that takes a motion vector in a lookup table as input. Exemplary functions are: adding two or more motion vectors; averaging two or more motion vectors. Motion vectors may be scaled before being input into the function.

[0441] b. Virtual motion candidates can be added to the same position as redundant motion candidates.

[0442] c. Virtual motion candidates may be added before all other motion candidates (eg, starting from the smallest entry index (eg, zero)).

[0443] d. In one example, it is only applied under certain conditions (eg, when the current LUT is not full).

[0444] 7. The LUT update method based on redundancy removal can be called under certain conditions, such as

[0445] a. The current block is encoded and decoded in Merge mode,

[0446] b. The current block is encoded and decoded in AMVP mode, but at least one component of the MV difference is non-zero;

[0447] c. Whether the current block is or is not encoded using a sub-block based motion prediction / motion compensation method (e.g., not encoded in affine mode)

[0448] d. The current block is encoded and decoded in Merge mode, and the motion information is associated with a specific type (e.g., from a spatial neighboring block, from a left neighboring block, or from a temporal block).

[0449] h. After encoding / decoding a block, one or more lookup tables may be updated by simply inserting the M motion information sets to the end of the table (ie after all existing candidates).

[0450] i. Furthermore, alternatively, some existing motion candidates in the table may be removed.

[0451] 1. In one example, if the table is full after inserting M motion information sets, the first few entries of the motion candidates may be removed from the table.

[0452] 2. In one example, if the table is full before inserting M motion information sets, the first few entries of the motion candidates may be removed from the table.

[0453] ii. Furthermore, alternatively, if the block is coded with a motion candidate from a table, the motion candidates in the table may be reordered so that the selected motion candidate is placed at the last entry of the table.

[0454] 1. In one example, before encoding / decoding a block, the lookup table may include 0 、HMVP 1 、HMVP 2 , …, HMVP K-1 、HMVP K 、HMVP K+1 , …, HMVP L-1 Represents the motion candidate, where HMVP i Represents the i-th entry in the lookup table. If the block is from HMVP K (K is in the range [0, L-1], including 0 and L-1) predicted, after encoding / decoding the block, the lookup table is reordered as: HMVP 0 、HMVP 1 、HMVP 2 , …, HMVP K-1 、HMVP K 、HMVP K+1 ,…,HMVP L-1 、HMVP K .

[0455] i. After encoding and decoding an intra-frame constrained block, the lookup table can be cleared.

[0456] j. If an entry of motion information is added to the lookup table, further entries of motion information may also be added to the table by deduction from the motion information. In this case, a counter associated with the lookup table may be incremented beyond 1.

[0457] i. In one example, the MV of an entry of motion information is scaled and placed in a table;

[0458] ii. In one example, the MV of the entry of motion information is added (dx, dy) and put into the table;

[0459] iii. In one example, the average of the MVs of two or more items of motion information is calculated and put into a table.

[0460] 14. If a block is located at a picture / slice / slice boundary, updates to the lookup table are always disallowed.

[0461] 15. The motion information of the above LCU row can be disabled to encode and decode the current LCU row.

[0462] a. In this case, the number of available motion candidates may be reset to 0 at the beginning of a new slice / slice / LCU row.

[0463] 16. When starting to encode a slice / slice with a new temporal layer index, the number of available motion candidates may be reset to 0.

[0464] 17. The lookup table may be continuously updated with one slice / slice / LCU row / slices having the same temporal layer index.

[0465] a. Alternatively, the lookup table may be updated only after encoding / decoding every S (S>=1) CTUs / CTBs / CUs / CBs or after encoding / decoding a specific region (e.g., size equal to 8×8 or 16×16).

[0466] b. Alternatively, the lookup table may be updated only after encoding / decoding every S (S>=1) blocks (e.g., CU / CB) in a specific mode (e.g., S inter-coded blocks). Alternatively, the lookup table may be updated only after encoding / decoding every S (S>=1) inter-coded blocks (e.g., CU / CB) that are not encoded / decoded using a sub-block-based motion prediction / motion compensation method (e.g., not encoded / decoded in affine and / or ATMVP mode).

[0467] c. Alternatively, the lookup table may be updated only when the upper left coordinate of the encoding / decoding block satisfies certain conditions. For example, the lookup table may be updated only when (x&M==0)&&(y&M==0), where (x, y) is the upper left coordinate of the encoding / decoding block. M is an integer such as 2, 4, 8, 16, 32, or 64.

[0468] d. Alternatively, once a lookup table reaches the maximum allowed counter, it may stop updating.

[0469] e. In one example, the counter may be predefined. Alternatively, the counter may be signaled in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a slice header, a codec tree unit (CTU), a codec tree block (CTB), a codec unit (CU) or a prediction unit (PU), or an area covering multiple CTUs / CTBs / CUs / PUs.

[0470] 18. The lookup table update process can be called from within different processes.

[0471] a. In one example, for a block encoded and decoded in Merge mode, the lookup table update process may be called after decoding a Merge candidate, after building a Merge list, or after decoding motion information with and / or without refinement.

[0472] b. In one example, for blocks encoded in AMVP mode, the lookup table update process may be invoked after decoding motion information with and / or without refinement.

[0473] c. When and / or how to update the lookup table may depend on the codec mode, block dimension, video processing data unit, low latency check, etc.

[0474] i. In one example, when a block is encoded and decoded in AMVP mode, the lookup table can be directly updated without pruning.

[0475] ii. Alternatively, when a block is encoded and decoded in Merge mode, the lookup table can be updated in case of pruning.

[0476] iii. Alternatively, when a block is coded in Merge mode and its motion information is derived from spatial and / or temporal blocks, the lookup table can be updated in case of pruning.

[0477] iv. Alternatively, when a block is encoded in Merge mode and its motion information is derived from motion candidates in the lookup table, the lookup table can be reordered without pruning.

[0478] v. Alternatively, when a block is encoded and decoded in Merge mode and its motion information is derived from virtual candidates in the lookup table (e.g., combined bidirectional, paired, zero motion vector candidates), the lookup table may not be updated.

[0479] vi. Alternatively, when a block is encoded and decoded in Sub-Block Merge mode and / or Triangle Merge mode, the lookup table may not be updated.

[0480] vii. Alternatively, when a block is coded in Merge with Motion Vector Difference (MMVD) mode and its motion information is derived from spatial and / or temporal blocks, the lookup table can be updated directly.

[0481] viii. In one example, when a block is encoded and decoded in illumination compensation (IC) mode and / or overlapped block motion compensation (OBMC) mode and / or decoding side motion vector derivation (DMVD) mode, the lookup table may not be updated. Alternatively, when a block is encoded and decoded in such a mode, the lookup table may be updated.

[0482] 19. Whether to reset the lookup table may also depend on an indication of enabling (or disabling) prediction crossing slices.

[0483] a. In one example, if such an indication indicates that cross-slice prediction is not allowed, the lookup table must be reset (eg, cleared) before encoding and decoding a new slice.

[0484] b. In one example, such an indication allows prediction of cross slices without resetting (e.g., clearing) the lookup table before encoding and decoding a new slice. That is, the lookup table used to encode and decode the second slice may depend on those used to encode and decode the first slice.

[0485] 20. Whether to enable Adaptive Loop Filter (ALF) time domain prediction (or prediction / inheritance of ALF filter from different slices) may also depend on an indication of enabling (or disabling) cross-slice prediction.

[0486] a. In one example, if such an indication indicates that cross-slice prediction is not allowed, then

[0487] ALF time domain prediction is not allowed.

[0488] b. In one example, such an indication allows prediction across slices, which may enable ALF time domain prediction.

[0489] 21. When the shared Merge list (or other kinds of shared motion candidate lists and / or shared intra mode information or other shared information) is enabled for the Merge sharing node, the update of the lookup table may be performed in the following manner:

[0490] a. One of the representative codec blocks (eg, leaf codec units) may be selected to update the lookup table.

[0491] i. In one example, it is defined as the last codec block in decoding order.

[0492] ii. Alternatively, it is defined as the last codec block that meets the conditions for lookup table update (such as the last codec block with non-affine and non-ATMVP inter-frame mode).

[0493] iii. In one example, it is defined as the first codec block under the parent node in decoding order.

[0494] iv. Alternatively, it is defined as the first codec block under the parent node that satisfies the conditions for lookup table update (such as the last codec block with non-affine and non-ATMVP, non-triangular inter mode).

[0495] b. More than one representative codec block (eg, leaf codec unit) may be selected to update the lookup table.

[0496] i. In one example, multiple sets of codec information associated with those representative codec blocks may be used to update a lookup table, wherein multiple entries of the lookup table may be updated accordingly.

[0497] ii. In one example, multiple sets of codec information associated with those representative codec blocks may be used to update the lookup table, wherein the multiple lookup tables may be updated accordingly.

[0498] iii. In one example, those representative codec blocks may be checked in a particular order to determine whether to be used to update one or more lookup tables.

[0499] iv. In one example, those representative codec blocks may be defined as the first codec block and the last codec block under the parent node.

[0500] c. Updates to the lookup table may be disabled at all times.

[0501] 22. For all of the above bullet points, the lookup table indicates codec information or information derived from codec information from a previously encoded or decoded block in decoding order.

[0502] a. The lookup table may include translation motion information, or affine motion information, or affine model parameters, or intra-mode information, or illumination compensation information, etc.

[0503] b. Alternatively, the lookup table may include at least two kinds of information, such as translational motion information, or affine motion information, or affine model parameters, or intra mode information, or illumination compensation information, etc.

[0504] 6. Additional Embodiments of LUT-Based Motion Vector Prediction

[0505] A history-based MVP (HMVP) method is proposed, where the HMVP candidate is defined as the motion information of the previously encoded or decoded block. A table with multiple HMVP candidates is maintained during the encoding / decoding process. When a new slice is encountered, the table is cleared. Whenever there is an inter-coded block, the associated motion information is added to the last entry of the table as a new HMVP candidate. Fig.29 The entire encoding and decoding process is described in .

[0506] In one example, the table size is set to L (eg, L=16 or 6, or 44), which indicates that up to L HMVP candidates may be added to the table.

[0507] (1) In one embodiment (corresponding to invention 11.gi), if there are more than L HMVP candidates from previously encoded or decoded blocks, a First-In-First-Out (FIFO) rule is applied so that the table always contains the latest L motion candidates previously encoded or decoded. Fig.30 An example is depicted in which the FIFO rule is applied to remove HMVP candidates and add new HMVP candidates to the table used in the proposed method.

[0508] (2) In another embodiment (corresponding to invention 11.g.iii), whenever a new motion candidate is added (such as when the current block is inter-coded and in non-affine mode), a redundancy check process is first applied to identify whether there are the same or similar motion candidates in the LUT.

[0509] Some examples are depicted below:

[0510] Fig.31A and Fig.31B An example of a redundancy removal-based LUT updating method in which one redundant motion candidate is removed is shown. Fig.31A shows the case when the LUT is full before adding a new motion candidate, and Fig.31B The situation is shown when the LUT is not full before adding a new motion candidate.

[0511] Fig.32A and Fig.32B An additional another example of a redundancy removal-based LUT updating method in which a plurality of redundant motion candidates are removed is shown. Fig.32A shows the case when the LUT is full before adding new motion candidates, and Fig.32B The situation is shown when the LUT is not full before adding a new motion candidate.

[0512] HMVP candidates can be used in the Merge candidate list construction process. All HMVP candidates from the last entry to the first entry in the table (or the last K0 HMVPs, for example, K0 equals 16 or 6) are inserted after the TMVP candidate. Pruning is applied to HMVP candidates. Once the total number of available Merge candidates reaches the maximum allowed Merge candidate notified by signaling, the Merge candidate list construction process is terminated. Alternatively, once the total number of added motion candidates reaches a given value, the extraction of motion candidates from the LUT is terminated.

[0513] Similarly, HMVP candidates can also be used in the AMVP candidate list construction process. The motion vectors of the last K1 HMVP candidates in the table are inserted after the TMVP candidate. Only HMVP candidates with the same reference picture as the AMVP target reference picture are used to construct the AMVP candidate list. Pruning is applied to HMVP candidates. In one example, K1 is set to 4.

[0514] The above examples and embodiments may be incorporated in the context of a method (eg, method 3300 ) described below, which may be implemented at a video decoder or a video encoder.

[0515] Fig.33 is a flow chart of a method for video processing. The method 3300 includes, at operation 3310, maintaining one or more tables, at least one of the one or more tables being associated with one or more entries.

[0516] The method 3300 includes, at operation 3320 , deriving a second motion candidate set based on the first motion candidate set in at least one table.

[0517] The method 3300 includes, at operation 3330, performing conversion between a current block of the video and a bitstream representation of the video based on the second motion candidate set.

[0518] Fig.34 is a flow chart of a method for video processing. The method 3400 includes, at operation 3410, maintaining one or more tables, wherein each of the one or more tables is associated with one or more entries.

[0519] The method 3400 includes, at operation 3420 , adaptively changing an insertion order of motion candidates from at least one of the one or more tables to a candidate list.

[0520] The method 3400 includes, at operation 3430, performing conversion between a current block of the video and a bitstream representation of the video based on the candidate list.

[0521] 7. Example simulation results of the disclosed technology

[0522] The effects of the disclosed techniques are summarized in the following complexity analysis and simulation results.

[0523] 7.1 Complexity Analysis

[0524] In terms of complexity analysis, two aspects were considered, including memory requirements and increased computational complexity. For the additional memory required for HMVP, up to 6 motion candidates need to be stored. That is, 6*(32 bits of a MV + 4 bits of a reference index)*2 = 54 bytes per strip.

[0525] As for the computational complexity, it mainly comes from the additional pruning operations. There are two processes of pruning operations required by HVMP.

[0526] • A procedure is called during table update when a new HMVP candidate is inserted, where up to 6 pruning operations are required.

[0527] Another process is called during the Merge list construction process. Due to the pruning process during the table update process, all HMVP candidates in the table are always different from each other. Therefore, during the Merge list construction process, if the second HMVP candidate has been marked as the same as the first HMVP candidate, there is no need to compare the second HMVP candidate with the spatial or temporal candidates. The number of additional pruning operations can be defined as C(N ST ,N mrgToBeAdded ,N HMVP ), where N ST Indicates the number of spatial or temporal Merge candidates already in the Merge candidate list, N mrgToBeAdded Indicates the maximum number of candidates from HMVP that can be added to the Merge list, and N HMVP Indicates the number of HMVP candidates to be checked. In this paper, N mrgToBeAdded Set to N mrg -1.

[0528] owhenN HMVP Not less than N mrgToBeAdded When the current N mrgToBeAdded - 1 candidate is different from the previous N STcandidates and the remaining HMVP candidates are the same as the first N ST The worst case occurs when one of the candidates is the same. In short, it requires ) times of pruning operations.

[0529] owhenN HMVP Less than N mrgToBeAdded When all N HMVP candidates are different from the first N ST The worst case occurs when there are N candidates, which requires N ST *N HMVP Pruning operation.

[0530] Table 5 shows the worst case scenario where N ST and N mrg The details of the pruning operation for each value representing the number of existing spatial or temporal merge candidates.

[0531] Table 5. Additional pruning operations introduced by HMVP in the worst case

[0532]

[0533] It should be noted that the table above lists only the worst case number of pruning operations. In hardware implementations, the throughput may be higher because the HMVP candidates are ready before decoding a block. Therefore, each HMVP can be compared in parallel with any available Merge candidates.

[0534] In addition, due to the reduced number of combined bi-predictive Merge candidate generation process pairs, i.e., from 12 to 6, it can be considered that the number of pruning operations is reduced by 3, because each combined bi-predictive Merge candidate generation process pair has 2 checks and each pruning process has 5 checks. Therefore, the number of additional pruning operations for the entire proposal in the worst case is listed as follows:

[0535] Table 6. Additional pruning operations for the entire proposal (for the worst case)

[0536]

[0537] 7.2 Simulation results

[0538] The following table shows the BD rate gain and processing time impact of the proposed method.Two sets of simulations were performed following the generic test conditions on top of the reference software VTM-2.0.1.

[0539] Enable HMVP for Merge mode only (test 4.4.7.a)

[0540] o Anchor: VTM (by disabling all BMS tools in BMS-1.0)

[0541] o Test: Anchoring with HMVP only applied in Merge mode

[0542] Enable HMVP for both AMVP and Merge modes (test 4.4.7.b)

[0543] oAnchor: BMS

[0544] oTest: With suggested anchors

[0545] For each test set, the Merge list size N was set to 6, 8, and 10 according to the CE4 description. mrg three subsets of .

[0546] 7.2.1 Simulation results of test 4.4.7.a

[0547] This subsection reports only mrg The encoding and decoding performance of HMVP applied to Merge mode when the bits are equal to 6, 8 and 10 respectively.

[0548] Table 7. Performance of test 4.4.7.a under random access Main 10

[0549]

[0550]

[0551] Table 8. Performance of Test 4.4.7.a under Low Latency B Main 10

[0552]

[0553]

[0554] 7.2.2 Simulation results with the same Merge list size

[0555] This subsection reports only mrg Codec performance of HMVP applied to both AMVP and Merge modes with bits equal to 6, 8 and 10 respectively.

[0556] Table 9. Performance of test 4.4.7.c under random access Main 10

[0557]

[0558]

[0559] Table 10. Performance of Test 4.4.7.c under Low Latency B Main 10

[0560]

[0561]

[0562] 8. Example Implementations of the Disclosed Technology

[0563] Fig.35 3500 is a block diagram of a video processing device 3500. The device 3500 can be used to implement one or more of the methods described herein. The device 3500 can be embodied in a smart phone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. The device 3500 may include one or more processors 3502, one or more memories 3504, and video processing hardware 3506. (Multiple) processors 3502 can be configured to implement one or more methods described in this document. Memory (multiple memories) 3504 can be used to store data and code for implementing the methods and techniques described herein. Video processing hardware 3506 can be used to implement some of the techniques described in this document in hardware circuits.

[0564] In some embodiments, the video encoding and decoding method can use the reference Fig.35 The device is implemented on the hardware platform.

[0565] Some embodiments of the disclosed technology include making a decision or determination to enable a video processing tool or pattern. In one example, when a video processing tool or pattern is enabled, the encoder will use or implement the tool or pattern when processing a block of a video, but will not necessarily modify the bitstream produced based on the use of the tool or pattern. That is, when enabling the video processing tool or pattern based on a decision or determination, the conversion from the block of a video to the bitstream representation of the video will use the video processing tool or pattern. In another example, when the video processing tool or pattern is enabled, the decoder will process the bitstream when it is known that the bitstream has been modified based on the video processing tool or pattern. That is, the video processing tool or pattern enabled based on a decision or determination will be used to perform the conversion from the bitstream representation of the video to the block of the video.

[0566] Some embodiments of the disclosed technology include making a decision or determination to disable a video processing tool or mode. In an example, when the video processing tool or mode is disabled, the encoder will not use the tool or mode in the conversion of blocks of video to a bitstream representation of the video. In another example, when the video processing tool or mode is disabled, the decoder will process the bitstream knowing that the bitstream has not been modified using the video processing tool or mode enabled based on the decision or determination.

[0567] Fig.36 3600 is a block diagram illustrating an example video processing system 3600 in which various techniques disclosed herein may be implemented. Various embodiments may include some or all of the components of system 3600. System 3600 may include an input 3602 for receiving video content. The video content may be received in a raw or uncompressed format, such as 8 or 10 bit multi-component pixel values, or may be in a compressed or codec format. Input 3602 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, a Passive Optical Network (PON), and the like, and wireless interfaces such as Wi-Fi or cellular interfaces.

[0568] System 3600 may include a coding component 3604 that can implement various encoding and encoding and decoding methods described in this document. Coding component 3604 can reduce the average bit rate of the video from input 3602 to the output of coding component 3604 to generate a coded representation of the video. Coding and decoding technology is therefore sometimes referred to as video compression or video transcoding technology. The output of coding component 3604 can be stored or sent via a communication connection such as represented by component 3606. The bitstream (or encoding) representation of the storage or communication transmission of the video received at input 3602 can be used by component 3608 to generate pixel values ​​or transmit to a displayable video of display interface 3610. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "encoding" operations or tools, it will be understood that the encoding tool or operation is used at the encoder, and the corresponding decoding tool or operation of the inverse encoding result will be performed by the decoder.

[0569] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or Displayport, etc. Examples of storage interfaces include SATA (Serial Advanced Technology Attachment), PCI, IDE interfaces, etc. The technology described in this document may be embodied in various electronic devices, such as mobile phones, laptops, smart phones, or other devices capable of performing digital data processing and / or video display.

[0570] It should be appreciated that several techniques have been disclosed that will be beneficial for incorporation into video encoder and decoder embodiments within video processing devices such as smartphones, laptops, desktops, and similar devices by allowing the use of intra-frame codec tools in encoding or decoding of videos or images.

[0571] In some embodiments, the following technical solutions may be implemented:

[0572] A1. A method for video processing, comprising: maintaining one or more tables, wherein at least one of the one or more tables is associated with one or more entries; deriving a second motion candidate set based on a first motion candidate set in at least one table; and performing conversion between a current block of a video and a bitstream representation of the video based on the second motion candidate set.

[0573] A2. The method according to solution A1, wherein at least one motion candidate in the second motion candidate set is derived based on adding an offset to a motion vector of at least one motion candidate in the first motion candidate set.

[0574] A3. A method according to solution A2, wherein the offset is added to the horizontal component or the vertical component of the motion vector.

[0575] A4. A method according to solution A2, wherein the offset is added to both the horizontal component and the vertical component of the motion vector.

[0576] A5. A method according to solution A1, wherein scaling a first unidirectional prediction candidate of a first motion candidate set to a reference picture in a reference picture list is used to derive a second unidirectional prediction candidate of a second motion candidate set, and wherein the first unidirectional prediction candidate and the second unidirectional prediction candidate have the same reference picture list index and share the reference picture list.

[0577] A6. A method according to solution A1, wherein scaling a first unidirectional prediction candidate of a first motion candidate set to a reference picture in a reference picture list is used to derive a second unidirectional prediction candidate of a second motion candidate set, wherein the first unidirectional prediction candidate and the second unidirectional prediction candidate have different reference picture list indexes, and wherein the reference picture list is associated with the second unidirectional prediction candidate.

[0578] A7. The method according to solution A1, wherein scaling unidirectional prediction candidates of the first motion candidate set to reference pictures in a reference picture list is used to derive bidirectional prediction candidates of the second motion candidate set, wherein the reference picture list is associated with the bidirectional prediction candidates.

[0579] A8. According to the method of solution A1, at least one motion candidate in the second motion candidate set is derived based on averaging two or more different motion candidates from the first motion candidate set.

[0580] A9. A method according to any one of solutions A1 to A8, wherein at least one motion candidate in the second motion candidate set is added to a motion candidate list, and wherein performing the conversion is based on the motion candidate list.

[0581] A10. A method according to solution A9, wherein the motion candidate list includes at least one of a motion candidate list of a Merge mode or a motion candidate list of an Advanced Motion Vector Prediction (AMVP) mode.

[0582] A11. The method according to any one of solutions A1 to A10 further includes: after performing the conversion, updating at least one table based on the motion information of the current block.

[0583] A12. A method according to any one of solutions A1 to A11, wherein the entry in at least one table includes a motion candidate in a first motion candidate set and its associated motion information, the associated motion information including a prediction direction, one or more reference pictures and / or indices, one or more motion vectors, a local illumination compensation (LIC) flag, an affine flag, a motion vector difference (MVD) value, or an MVD precision.

[0584] A13. A method according to solution A12, wherein the associated motion information also includes position information associated with the current block.

[0585] A14. According to the method described in any one of solutions A1 to A13, the conversion generates the current block from the bitstream representation.

[0586] A15. A method according to any one of solutions A1 to A13, wherein the conversion generates a bitstream representation from the current block.

[0587] A16. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement a method according to any one of solutions A1 to A15.

[0588] A17. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising a program code for executing a method according to any one of solutions A1 to A15.

[0589] B1. A method for video processing, comprising: maintaining one or more tables, wherein each of the one or more tables is associated with one or more entries; adaptively changing the insertion order of motion candidates from at least one of the one or more tables to a candidate list; and performing conversion between a current block of a video and a bitstream representation of the video based on the candidate list.

[0590] B2. According to the method described in solution B1, the update is based on the encoding and decoding mode of the current block.

[0591] B3. A method according to solution B2, wherein the insertion order includes inserting a new motion candidate after a temporal motion vector prediction (TMVP) candidate when it is determined that the coding mode is a Merge mode.

[0592] B4. A method according to solution B2, wherein the insertion order includes inserting the new motion candidate before the spatial candidate.

[0593] B5. A method according to solution B2, wherein the insertion order includes inserting a new motion candidate after the spatial candidate when it is determined that the coding mode is an intra block copy (IBC) mode.

[0594] B6. A method according to solution B1, wherein the insertion order changes adaptively from block to block, from slice to slice, from slice group to slice group, or from one video processing unit to another.

[0595] B7. A method according to solution B1, wherein the update or insertion order is based on the height or width of the current block.

[0596] B8. A method according to solution B1, wherein the update or insertion order is based on the partition depth of the current block.

[0597] B9. A method according to solution B1, wherein the update or insertion order is based on a slice type, slice group type, slice type or picture type associated with the current block.

[0598] B10. A method according to solution B1, wherein the update or insertion order is based on a low latency check flag in the bitstream representation.

[0599] B11. The method according to any one of solutions B1 to B10 also includes: during the construction of the candidate list, adaptively changing at least one of the following: the pruning process when adding motion candidates to the candidate list, or the number of motion candidates to be checked in a table in one or more tables, or the maximum number of motion candidates in the LUT to be checked, or the number of tables in one or more tables to be checked.

[0600] B12. A method according to any one of solutions B1 to B11, wherein the table comprises a history-based motion vector prediction (HMVP) table.

[0601] B13. A method according to solution B12, wherein the entries in the HMVP table include motion candidates and their associated motion information, the associated motion information including prediction direction, one or more reference pictures and / or indices, one or more motion vectors, a local illumination compensation (LIC) flag, an affine flag, a motion vector difference (MVD) value or MVD precision.

[0602] B14. The method according to any one of solutions B1 to B11 further includes: after performing the conversion, updating one or more tables based on the motion information of the current block.

[0603] B15. According to the method described in any one of solutions B1 to B14, the conversion generates the current block from the bitstream representation.

[0604] B16. A method according to any one of solutions B1 to B14, wherein the conversion generates a bitstream representation from the current block.

[0605] B17. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement a method according to any one of solutions B1 to B16.

[0606] B18. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising a program code for executing a method according to any one of solutions B1 to B16.

[0607] From the foregoing it will be appreciated that specific embodiments of the presently disclosed technology have been described herein for purposes of illustration, but that various modifications may be made without departing from the scope of the invention. Accordingly, the presently disclosed technology is not to be limited except as set forth in the appended claims.

[0608] The embodiments of the subject matter and functional operations described in this patent document can be implemented in various systems, digital electronic circuits, or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-temporary computer-readable medium, which are used to be executed by a data processing device or control the operation of the data processing device. The computer-readable medium can be a combination of a machine-readable storage device, a machine-readable storage substrate, a memory device, a material that affects a machine-readable propagation signal, or a combination of one or more of them. The term "data processing unit" or "data processing device" includes all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the device can also include code that creates an execution environment for the computer program in question, for example, code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0609] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language (including compiled or interpreted languages), and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). A computer program may be deployed to execute on one computer or on multiple computers located on one site or distributed across multiple sites and interconnected by a communication network.

[0610] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, and the apparatus can also be implemented as special purpose logic circuits, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits).

[0611] Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operably coupled to receive data from the one or more large-capacity storage devices or to transfer data to the one or more large-capacity storage devices, or to receive data from and transfer data to the one or more large-capacity storage devices. However, the computer does not need such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices. Processors and memory can be supplemented by or incorporated into a dedicated logic circuit.

[0612] Although this patent document contains many details, these details should not be interpreted as limitations on any invention or the scope that may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features described in the context of separate embodiments in this patent document may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be excluded from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0613] Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed, to achieve the desired results. In addition, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0614] Only a few implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and shown in this patent document.

Claims

1. A method for processing video data, comprising: maintaining one or more tables, wherein each of the one or more tables is associated with one or more entries; adaptively changing the insertion order of motion candidates from at least one of the one or more tables into the candidate list; and performing a conversion between a current block of the video and a bitstream of the video based on the candidate list, The change is based on the coding mode of the current block, and the insertion order includes inserting a new motion candidate after the spatial candidate when it is determined that the coding mode is an intra-block copy (IBC) mode, or wherein the changing or the inserting order is based on the height or width of the current block, or wherein the changing or the inserting order is based on the partition depth of the current block, or wherein the changing or the insertion order is based on a slice type, a slice group type, a slice type, or a picture type associated with the current block, or Wherein, the changing or the inserting order is based on a low delay check flag in the bitstream.

2. The method according to claim 1, wherein: The insertion order also includes inserting a new motion candidate after the temporal motion vector prediction TMVP candidate when it is determined that the coding mode is the Merge mode.

3. The method according to claim 1, wherein: The insertion order also includes inserting new motion candidates before spatial candidates.

4. The method according to claim 1, wherein: The insertion order changes adaptively from block to block, from slice to slice, from slice group to slice group, or from one video processing unit to another video processing unit.

5. The method according to any one of claims 1 to 4, further comprising: During construction of the candidate list, at least one of the following is adaptively changed: the pruning process when adding motion candidates to the candidate list, or the number of motion candidates to be checked in a table among the one or more tables, or the maximum number of motion candidates in a lookup table LUT to be checked, or the number of tables among the one or more tables to be checked.

6. The method according to any one of claims 1 to 4, wherein: At least one of the one or more tables comprises a history-based motion vector prediction (HMVP) table.

7. The method according to claim 6, wherein: The entries in the HMVP table include motion candidates and their associated motion information, wherein the associated motion information includes prediction direction, one or more reference pictures and / or indices, one or more motion vectors, local illumination compensation LIC flag, affine flag, motion vector difference MVD value or MVD precision.

8. The method according to any one of claims 1 to 4, further comprising: After performing the conversion, the one or more tables are updated based on the motion information of the current block.

9. The method according to any one of claims 1 to 4, wherein: The converting includes decoding the current block from the bitstream.

10. The method according to any one of claims 1 to 4, wherein: The converting includes encoding the bitstream from the current block.

11. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method according to any one of claims 1 to 10.

12. A non-transitory computer-readable medium storing instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 10.

Images