Video encoding and decoding method and device

Abstract

A video decoder determines whether to encode or decode a current block using an intra block copy mode. The video decoder identifies a list of one or more prediction candidates for the current block. When the current block is not encoded using the intra block copy mode, one or more neighboring blocks of the current block located in the same MER as the current block are excluded from the list of prediction candidates. When the current block is encoded using the intra block copy mode and the list of prediction candidates belongs to a predetermined subset of a plurality of different candidate lists, at least one of the identified prediction candidates is a spatial neighboring block of the current block located in the MER. The video decoder generates a prediction of the current block by reconstructing the current block using a prediction candidate selected from the list of prediction candidates.

Description

[0001] Cross-references

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 62 / 940,961, filed on November 27, 2019. The U.S. Provisional Patent Application is hereby incorporated by reference in its entirety. Technical Field

[0003] The present invention generally relates to video coding and decoding. In particular, the present invention relates to a method for facilitating parallel processing during video coding and decoding by using a merge mode. Background Art

[0004] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims listed below and are not admitted to be prior art by inclusion in this section.

[0005] High Efficiency Video Codec (HEVC) is a video codec standard developed by the Joint Collaboration Team on Video Codecs (JCT-VC). In HEVC, the coded image is partitioned into multiple non-overlapping square block regions represented by coding tree units (CTUs). The coded image can be represented by a number of slices, each slice consisting of an integer number of CTUs. The individual CTUs in a slice are processed in raster scan order. Bidirectionally predicted (B) slices can predict the sample values ​​of each block by intra prediction or inter prediction using up to two motion vectors and reference indices. Intra (I) slices are decoded using only intra prediction. Predicted (P) slices predict the sample values ​​of each block by intra prediction or inter prediction using up to one motion vector and reference indices.

[0006] One or more prediction units (PUs) are specified for each coding unit (CU). A prediction unit and the associated CU syntax serve as the basic unit for signaling prediction sub-information. A specified prediction process is applied to predict the values ​​of the associated pixel samples within the PU. Based on the selected PU type, a CU can be split into one, two, or four PUs. HEVC defines eight types of partitioning for splitting a CU into multiple PUs.

[0007] The CU can be further partitioned using a residual quadtree (RQT) structure to represent the associated prediction residual signal. The leaf nodes of the RQT correspond to the generated transform units (TUs). A transform unit consists of a transform block of luminance samples of size 8×8, 16×16, or 32×32 or size 4×4, and two corresponding transform blocks of chrominance samples for images in 4:2:0 color format. An integer transform is applied to the transform block and the quantized coefficient values ​​are encoded and decoded in the bitstream. The minimum and maximum transform block sizes are specified in the sequence parameter set.

[0008] In HEVC, the terms coding tree block (CTB), coding block (CB), prediction block (PB), and transform block (TB) are defined to refer to a 2-D array of samples from one color component of the associated CTU, CU, PU, ​​and TU, respectively. A CTU thus consists of one luma CTB, two chroma CTBs, and associated syntax elements from a color picture coded without three separate color planes. The signaled coding tree partitioning generally applies to both luma and chroma blocks, but some exceptions may be allowed when certain minimum size constraints are met. Summary of the Invention

[0009] The following summary is illustrative only and is not intended to be limiting in any way. That is, the following summary is provided to introduce the concepts, highlights, benefits, and advantages of the novel and non-obvious technologies described herein. Selected, but not all, aspects are further described in the detailed description. Therefore, the following summary is not intended to identify essential features of the claimed subject matter, nor is it intended to determine the scope of the claimed subject matter.

[0010] The present invention provides a method for identifying prediction candidates based on a merged estimation region (MER) and an intra block copy (IBC) mode. In some embodiments, a video decoder receives data from a bitstream for a current block of a current picture of a video to be decoded. The video decoder determines whether to encode or decode the current block using the intra block copy mode. The video decoder identifies one or more lists of prediction candidates for the current block. When the current block is not encoded using the intra block copy mode and the list of prediction candidates belongs to or is derived from a predetermined subset of a plurality of different candidate lists, one or more spatially neighboring blocks of the current block located in the same MER as the current block are excluded from the list of prediction candidates. When the current block is encoded using the intra block copy mode, at least one of the identified prediction candidates is a spatially neighboring block of the current block located in the MER. The video decoder generates a prediction for the current block by reconstructing the current block using the prediction candidate selected from the one or more lists of prediction candidates.

[0011] In some embodiments, when the current block is not encoded or decoded using the intra block copy mode, the current block and at least one other block in the MER can be reconstructed in parallel. In some embodiments, one or more syntax elements in the bitstream indicate whether the intra block copy mode is used to encode or decode the current block, and the decoder parses the one or more syntax elements in the bitstream and determines whether the intra block copy mode is used for the current block based on the parsed syntax elements. In some embodiments, the one or more syntax elements in the bitstream are used to identify or define the MER that includes the current block.

[0012] The prediction of the current block may be an affine prediction generated based on the selected prediction candidate. The prediction of the current block may be a triangle or geometric partitioning mode (GPM) prediction generated based on the selected prediction candidate. The prediction of the current block may be a combined inter-frame and intra-frame prediction (CIIP) generated based on the selected prediction candidate. The prediction of the current block may be generated by refining the selected prediction candidate with motion vector difference information. A predetermined subset of the plurality of different candidate lists may include candidates of a merge prediction mode, such as GPM, CIIP, and a conventional merge mode, and excludes candidates of an advanced motion vector prediction (AMVP). BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are included to provide a further understanding of the present invention and are incorporated into and constitute a part of this invention. The accompanying drawings illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the present invention. It will be understood that since some elements may be shown out of proportion to the sizes in the embodiments, the accompanying drawings are not necessarily drawn to scale to clearly illustrate the concepts of the present invention.

[0014] Figure 1 The splitting of a coding tree unit (CTU) into a quadtree is shown.

[0015] Figure 2 An example of a splitting process and its binary tree is shown.

[0016] Figure 3 An example of a quadtree plus binary tree (QTBT) structure is shown.

[0017] Figure 4 Various splitting types or patterns are shown for partitioning a CTU in a multi-type (MTT) tree structure.

[0018] Figure 5 An example of tree type signaling according to block partitioning of MTT block partitioning is shown.

[0019] Figure 6 The candidate set for inter prediction mode is shown.

[0020] Figure 7 A merge candidate list including a combined bi-predictive merge mode is shown.

[0021] Figure 8 A merge candidate list including the selected merge candidate is shown.

[0022] Figure 9 An example is shown in which a 0-vector candidate is added to a merge candidate list or an advanced motion vector prediction (AMVP) candidate list.

[0023] Figure 10 A four-parameter affine motion model is shown.

[0024] Figure 11 Motion vector predictor (MVP) derivation for affine inter mode is shown.

[0025] Figure 12 Various examples are shown for defining or identifying common boundaries for a common candidate list.

[0026] Figure 13 Subtrees of the split tree corresponding to CUs in a common boundary that can be encoded by a common candidate list are shown.

[0027] Figure 14 Common boundaries for identifying common candidate lists are shown.

[0028] Figure 15 Spatial merging candidates for a CU identified based on a merge estimation region (MER) are conceptually illustrated.

[0029] Figure 16 A reference area for the intra block copy mode (IBC) is shown.

[0030] Figure 17 The positions of spatial merging candidates are shown.

[0031] Figure 18 Candidate pairs considered for redundancy checking of spatial merging candidates are shown.

[0032] Figure 19 Motion vector scaling for temporal merging candidates is shown.

[0033] Figure 20 The position of the temporal candidate selected between candidates C0 and C1 is depicted.

[0034] Figure 21 The offset added to the horizontal component or the vertical component of the starting MV is shown.

[0035] Figure 22 The SbTVMP process is conceptually shown, specifically deriving a sub-CU motion field by applying motion shifts from spatially neighboring blocks and scaling the motion information from the corresponding co-located sub-CU.

[0036] Figure 23 as well as Figure 24 The regions of the tiles created by geometric segmentation are shown.

[0037] Figure 25 Inclusion and exclusion of spatially neighboring blocks from a merge candidate list based on MER and IBC modes are conceptually illustrated.

[0038] Figure 26An exemplary video encoder is shown that can use a prediction candidate list to generate a prediction when encoding a block.

[0039] Figure 27 Part of a video encoder based on MER and IBC mode recognition prediction candidates is shown.

[0040] Figure 28 A process for encoding blocks in a video image using prediction candidates based on MER and IBC pattern recognition is conceptually illustrated.

[0041] Figure 29 An exemplary video decoder is shown that can use a prediction candidate list to generate a prediction when decoding a block.

[0042] Figure 30 Part of a video decoder that recognizes prediction candidates based on MER and IBC patterns is shown.

[0043] Figure 31 A process for decoding blocks in a video image using prediction candidates based on MER and IBC pattern recognition is conceptually illustrated.

[0044] Figure 32 An electronic system is conceptually illustrated in which some embodiments of the present invention may be implemented. DETAILED DESCRIPTION

[0045] In the detailed description below, many specific details are provided by way of example to provide a thorough understanding of the relevant teachings. Any variations, derivatives, and / or extensions of the teachings described herein are within the scope of protection of the present invention. In some cases, well-known methods, processes, components, and / or circuits related to one or more exemplary embodiments disclosed herein may be described at a relatively high level without further details to avoid unnecessarily obscuring the teachings of the present invention.

[0046] I. Segmentation Structure

[0047] In HEVC, an image consists of multiple slices, and a slice consists of multiple CTUs. A CTU is a square of size 8×8, 16×16, 32×32, and 64×64. A CTU is divided into multiple coding units (CUs). The quadtree result is used to divide the CTU. The CTU size is M×M (M is one of 64, 32, or 16). A CTU can be a single CU or can be split into four units of size M / 2×M / 2, which are coding tree nodes. If the units are leaf nodes, they become CUs. Otherwise, the quadtree can be further split until the size of the node reaches the minimum CU size specified in the SPS. Figure 1The CTU is split into a quadtree. The solid lines are CU boundaries. There are one or more PUs (prediction units) in each CU.

[0048] In conjunction with the CU, the PU is a basic block for sharing prediction information. Within each PU, the same prediction process is applied. Depending on the PU partitioning type, the CU can be split into 1, 2 or 4 PUs. The PU can be split only once. Alternatively, a binary tree (BT) partitioning structure can be used. In particular, the block can be iteratively split into 2 smaller blocks. Symmetrical horizontal and vertical splitting are the most efficient and simple splitting types. In some embodiments, only these two splitting types are used. The binary tree splitting process can be iteratively split until the width or height of the split block reaches the minimum block width or height defined in the high-level syntax in the video bitstream.

[0049] Figure 2 An example of a splitting process (left) and its binary tree (right) is shown. In each split, a flag indicates whether horizontal or vertical splitting is used at a non-leaf node in the binary tree: 0 indicates horizontal splitting and 1 indicates vertical splitting. A binary tree splitting structure can be used to split a CTU into multiple CUs, i.e., the root node is the CTU and the leaf nodes are the CUs. For simplicity, as in another embodiment, no further partitioning from CU to PU is performed. This means that a CU is equivalent to a PU, and therefore, the leaf nodes of the binary tree are the basic units of prediction.

[0050] In some embodiments, a QTBT structure (quadtree plus binary tree structure) is used. It combines a quadtree and a binary tree. In the QTBT structure, first, a block is split by a quadtree splitting process, and the quadtree (QT) splitting process can be repeated until the size reaches the minimum leaf node size. Secondly, if the leaf quadtree block is not larger than the maximum allowed binary tree root node size, the leaf quadtree block can be further split into a binary tree (BT) split, and the binary split can be repeated until the width or height of the split block reaches the minimum allowed width or height or the binary split depth reaches the maximum allowed depth. Figure 3 An example of a QTBT structure is shown. The QTBT structure can be used to split a CTU into multiple CUs. That is, the root node is the CTU, which is split into multiple CUs by the QTBT structure, and the multiple CUs are further divided into prediction and transform codecs. For simplicity, as in another embodiment, no further splitting from CU to PU is performed. This means that a CU is equal to a PU, and the leaf node of the QTBT is the basic unit of prediction.

[0051] For I slices, the QTBT structure is applied separately for luma and chroma. For P and B slices, the QTBT structure is applied to both luma and chroma (except for chroma up to certain minimum sizes). In other words, in I slices, the QTBT structure block partitioning tree is different between the luma CTB and the two chroma CTBs. Specifically, the luma CTB has a QTBT structure block partitioning, and the two chroma coding tree units (CTBs) have another QTBT structure block partitioning.

[0052] Flexible CU structures like Quadtree Binary Tree (QTBT) show good encoding and decoding performance compared to the Quadtree (QT) structure in HEVC. Figure 3 As shown, a CTU is first partitioned using a quadtree structure. The leaves of the quadtree are further partitioned using a binary tree structure. In order to iteratively partition the block into two smaller blocks, in addition to the traditional symmetric horizontal and vertical split types, asymmetric horizontal and vertical split types can also be selected. After constructing the binary tree structure, the leaves of the binary tree are labeled as CUs, which are used for prediction and transform without any further partitioning.

[0053] To support more segmentation shapes and achieve more flexible segmentation, the ternary tree (TT) segmentation method aims to obtain the target located at the center of the block, while the quadtree (QT) and binary tree (BT) segmentation methods always split along the center of the block.

[0054] Multi-type tree (MTT) block partitioning extends the concept of QTBT two-level tree structure by allowing both binary tree and ternary tree partitioning methods in the second level of MTT. The two levels of trees in MTT are called region tree (RT) and prediction tree (PT), respectively. The first level RT is always quadtree (QT) partitioned, and the second level PT can be binary tree (BT) partitioned or ternary tree (PT) partitioned. For example, a CTU is first partitioned by RT, which is QT partitioned, and each RU leaf node can be further split by PT, which can be BT or TT partitioned. Blocks partitioned by PT can be further split by PT until the maximum PT depth is reached, for example, a block can first be partitioned by vertical BT to generate a left subblock and a right subblock, and the left subblock is further split by horizontal TT partitioning and the right subblock is further split by horizontal BT partitioning. The PT leaf node is the base CU used for prediction and transformation and will not be further split.

[0055] Figure 4Various split types or modes (a)-(e) for splitting a CTU in an MTT structure are shown. Split types (a), (b), and (c) correspond to quadtree splitting, vertical binary tree splitting, and horizontal binary tree splitting, respectively. Split types (d) and (e) are called ternary tree (or ternary tree) split types, which split the block into three smaller blocks. The three smaller blocks from the ternary tree split have a smaller size in one spatial dimension while maintaining the same size in the other spatial dimensions. The ternary tree splitting method can provide the ability to localize small objects along block boundaries faster by allowing quarter splitting vertically or horizontally.

[0056] Figure 5 An example of tree type signaling for block segmentation according to MTT block segmentation is shown. RT signaling can be similar to quadtree signaling for QTBT block segmentation. In order to signal the PT node, an additional binary number (bin) is signaled to indicate whether it is a binary tree segmentation or a ternary tree segmentation. For block segmentation by RT, the first binary number (bin) is signaled to indicate whether there is another RU split, and if the block is not further split by RU (i.e., the first binary number is 0), the second binary number is signaled to indicate whether there is a PT split. If the block is not further split by PT (i.e., the second binary number is 0), then the block is a leaf node. If the block is further split by PT (i.e., the second binary number is 1), the third binary number is sent to indicate horizontal or vertical segmentation followed by a fourth binary number used to distinguish between binary tree (BT) or ternary tree (TT) segmentation.

[0057] After constructing the MTT block partition, the MTT leaf node is a CU, which is used for prediction and transform without any partitioning. In MTT, the tree structure is coded separately for luma and chroma in I slices, and is applied to both luma and chroma in P and B slices (except for chroma up to a certain minimum size). That is, in an I slice, the luma CTB has its own block partitioning of QTBT structure, and the two chroma CTBs have another block partitioning of QTBT structure.

[0058] II. Inter-frame prediction mode candidates

[0059] In order to increase the codec efficiency of motion vector (MV) coding in HEVC, HEVC has skip and merge modes. Skip and merge modes obtain motion information from spatially neighboring blocks (spatial candidates) or temporally co-located blocks (temporal candidates). When the PU is in skip or merge mode, no motion information is encoded and decoded, instead, only the index of the selected candidate is encoded and decoded. For skip mode, the residual signal is forced to 0 and is not encoded and decoded. In HEVC, if a particular block is encoded as skip or merge, a candidate index is signaled to indicate which candidate in the candidate set is used for merging. Each merged PU reuses the MV, prediction direction, and reference image index of the selected candidate. For some embodiments, the term "merge mode" is used to refer to both skip mode and merge mode.

[0060] Figure 6 The diagram shows a candidate set of inter-frame prediction modes (i.e., skip, merge, AMVP, etc.). The diagram shows a current block 600 of a video image or frame being encoded or decoded. The current block 600 (which can be a PU or CU) refers to neighboring blocks to derive spatial and temporal MVPs as a candidate list for MVP or AMVP mode, merge mode, or skip mode.

[0061] For merge mode, such as Figure 6 As shown, at most four spatial MV candidates are derived from A0, A1, B0, and B1, and one temporal MV candidate is derived from T BR or T CTR (First use T BR , if T BR If not available, use T CTR ) to derive. Note that if any one of the four spatial MV candidates is not available, then position B2 is used to derive an MV candidate as an alternative. After the derivation process of four spatial MV candidates and one temporal MV candidate, redundancy removal (pruning) is applied to remove redundant MV candidates. If the number of available MV candidates is less than five after redundancy removal (pruning), three types of additional candidates are derived and added to the candidate set (candidate list). The encoder decides to select a final candidate in the candidate set for skip or merge mode based on rate-distortion optimization (RDO) and transmits the index to the decoder.

[0062] a.AMVP model

[0063] To achieve optimal codec efficiency within the hybrid codec architecture, HEVC employs intra-frame prediction and / or inter-frame prediction modes for each PU. For intra-frame prediction, spatially adjacent reconstructed pixels are used to generate directional predictions in 35 directions. For inter-frame prediction, motion information is used to reconstruct a temporal reference frame, which is then used to generate motion-compensated predictions. Motion information can include motion vectors, motion vector predictors, motion vector differences, and reference index frames used to select reference frames.

[0064] When the PU is encoded or decoded in inter-frame AMVP mode, motion compensated prediction is performed using the transmitted motion vector difference (MVD), which can be used together with the motion vector predictor (MVP) to derive the motion vector (MV). In order to determine the MVP in inter-frame AMVP mode, the advanced motion vector prediction (AMVP) scheme is used to select a motion vector from an AMVP candidate set including two spatial MVPs and one temporal MVP. Therefore, in AMVP mode, the MVP index and the corresponding MVD of the MVP need to be encoded and transmitted. In addition, the inter-frame prediction direction that specifies the prediction direction in bidirectional prediction and unidirectional prediction, which is list 0 (L0) and list 1 (L1), is encoded and transmitted together with the reference frame index of each list.

[0065] When a PU is encoded or decoded in skip or merge mode, no motion information is transmitted except the merge index of the selected candidate. This is because skip and merge modes use a motion reference method (MV=MVD+MVD, where MVD is 0) to obtain motion information from spatially neighboring blocks (spatial candidates) or temporal blocks (temporal candidates) located in a parity picture, which is the first reference picture in list 0 or list 1, which is signaled in the slice header. In the case of a skip PU, the residual signal is also ignored. To determine the merge index for skip and merge modes, a merge scheme is used to select a motion vector predictor from a merge candidate set consisting of four spatial MVPs and one temporal MVP.

[0066] For AMVP mode, the left MVP is the first available one from A0, A1, the top MVP is the first available one from B0, B1, B2, and the time MVP is the first available one from T BR or T CTR The first available one (first use T BR , if T BR Not available, use T CTRIf the left MVP is not available and the top MVP is not a scaled MVP, a second top MVP can be derived if there is a scaled MVP in B0, B1, and B2. Therefore, after the derivation process of two spatial MVPs and one temporal MVP, only the first two MVPs can be included in the candidate list. If the number of available MVPs is less than 2 after removing redundancy, a 0-vector candidate can be added to the candidate list.

[0067] For skip mode and merge mode, up to four spatial merge indices are derived from A0, A1, B0, and B1, and from T BR or T CTR (First use T BR , if T BR Not available, use T CTR ) derives a temporal merge index. If any of the four spatial merge indexes is not available, position B2 is used to derive a merge index as a replacement. After deriving four spatial merge indexes and one temporal merge index, redundant merge indexes are removed. If the number of non-redundant merge indexes is less than 5, additional candidates can be derived from the original candidates and added to the candidate list. There are three types of derived candidates:

[0068] 1. Combined bidirectional prediction merge candidate (derived candidate type 1)

[0069] 2. Scaled Bidirectional Prediction Merge Candidates (Derived Candidate Type 2)

[0070] 3.0 Motion Merging / AMVP Candidates (Derived Candidate Type 3)

[0071] For derived candidate type 1, a combined bi-predictive merge candidate is created by combining the original merge candidates. In particular, if the current slice is a B slice, further merge candidates can be generated by combining candidates from list 0 and list 1. Figure 7 A merge candidate list including a combined bi-predictive merge candidate is shown. As shown, two original candidates with mvL0 (motion vector in list 0) and refidxL0 (reference image index in list 0) or mvL1 (motion vector in list 1) and refidxL1 (reference image index in list 1) are used to create a bi-predictive merge candidate.

[0072] For the derived candidate type 2, a scaled merge candidate is created by scaling the original merge candidate. Figure 8A merge candidate list including scaled merge candidates is shown. As shown, the original merge candidate has mvLX (motion vector in list X, X can be 0 or 1) and refidxLX (reference image index in list X, X can be 0 or 1). For example, the original candidate A is a list 0 unidirectional prediction MV with mvL0_A and reference image index ref0. Candidate A is initially copied to list L1 as a unidirectional prediction MV with reference image index ref0'. The scaled MV mvL0'_A is calculated by scaling mvL0_A based on ref0 and ref0'. A scaled bidirectional merge candidate with mvL0_A and ref0 in list 0 and mvL0'_A and ref0' in list 1 is created and added to the merge candidate list. Similarly, a scaled bidirectional prediction merge candidate with mvL1'_A and ref'1 in list 0 and mvL1_A and ref1 in list 1 is created and added to the merge candidate list.

[0073] For derived candidate type 3, a 0-vector candidate is created by combining the 0-vector and the reference index. If the created 0-vector candidate is not a duplicate, it is added to the merge / AMVP candidate list. Figure 9 An example is shown in which a 0-vector candidate is added to a merge candidate list or an AMVP candidate list.

[0074] b. Sub-PU Temporal Motion Vector Prediction (ATMVP)

[0075] The ATMVP (Advanced Temporal Motion Vector Prediction) mode (also known as Sub-PU Temporal Motion Vector Prediction (SbTMVP)) is a sub-PU-based mode for merge candidates. The ATMVP mode uses spatial neighbors to obtain an initial vector, which is used to obtain the coordinates of the collocated block on the collocated picture. The sub-CU (typically 4×4 or 8×8) motion information of the collocated block on the collocated picture is retrieved and filled into the sub-CU (typically 4×4 or 8×8) motion buffer of the current merge candidate. There are various embodiments of the ATMVP. The ATMVP is described in the 3rd meeting of the Joint Video Exploration Team (JVET) of ITU-T SG 16 WP3 and ISO / IEC JTC 1 / SC 29 / WG 11, Geneva, Switzerland, May 26-June 1, 2016, titled "Algorithm Description of Joint Exploration Test Model 3." ATMVP is also described in ITU-T SG 16 WP3 and ISO / IEC JTC 1 / SC 29 / WG 11, Joint Video Exploration Team (JVET), 11th Meeting, Ljubljan, SI, July 10-18, 2018, document: JVET-K0346-v3, titled “CE4-related: One simplified design of advanced temporal motion vector prediction (ATMVP)”.

[0076] c. Spatial Temporal Motion Vector Prediction (STMVP)

[0077] The STMVP mode is a sub-PU based mode for merging candidates. The motion vectors of the sub-PUs are generated iteratively in raster scan order. The derivation of the MV for the current sub-PU identifies two spatial neighboring blocks and one temporal neighboring block before MV scaling. After retrieving and scaling the MV, all available motion vectors (up to 3) are averaged and assigned as the motion vector of the current sub-PU. STMVP is described in the 3rd meeting of the Joint Video Exploration Team (JVET) of ITU-T SG 16 WP3 and ISO / IEC JTC 1 / SC 29 / WG 11, Geneva, Switzerland, May 26-June 1, 2016, titled "Algorithm Description of Joint Exploration Test Model 3", in particular Section 2.3.1.2: Spatial Temporal Motion Vector Prediction (STMVP).

[0078] d. Historically based merger patterns and AMVP

[0079] A video codec that implements a history-based merge mode may also store merge candidates for some previous CUs in a history array. For a CU currently being encoded or decoded, the video codec may use one or more candidates in the history array to enhance the original merge mode candidates. The history-based approach is also applicable to the AMVP candidate list. The history-based merge mode is described in the 11th meeting of the Joint Video Exploration Team (JVET) of ITU-T SG 16WP3 and ISO / IEC JTC 1 / SC 29 / WG 11, Ljubljan, SI, July 10-18, 2018, document: JVET-K0104: "History-based Motion Vector Prediction".

[0080] e. Non-adjacent merge mode and AMVP

[0081] Non-adjacent merge candidates use some spatial candidates far away from the current CU. Non-adjacent based methods can also be applied to the AMVP candidate list. An example of a non-adjacent merge mode is shown in the 11th meeting of the Joint Video Exploration Team (JVET) of ITU-T SG 16 WP3 and ISO / IEC JTC 1 / SC 29 / WG 11, Ljubljan, SI, July 10-18, 2018, document: JVET-K0228, titled "CE 4-2.1: Adding non-adjacent spatial merge candidates". An example of a non-adjacent merge candidate is depicted in the 11th meeting of the Joint Video Exploration Team (JVET) of ITU-T SG 16 WP3 and ISO / IEC JTC 1 / SC 29 / WG 11, Ljubljan, SI, July 10-18, 2018, document: JVET-K0286, titled "CE4: Additional mergecandidates (Test 4.2.13)".

[0082] f. Affine merge mode

[0083] HEVC only uses a translational motion model for motion compensated prediction. There are many other types of motion in the real world, such as reduction and rotation, perspective motion, and other irregular motions. Some of these other types of motion can be represented by affine transformations or affine motions, which protect points, lines, and planes. Affine transformations do not need to protect the angles between lines or the distances between points, but they protect the ratio of distances between points on a line. When an affine motion block moves, the motion vector field of the block can be described by two control point motion vectors or four parameters, as follows:

[0084]

[0085] The transform block is a rectangular block. The motion vector field of each point in this motion block can be described by the following equation of the four-parameter mimetic model:

[0086]

[0087] Or a 6-parameter affine model to describe:

[0088]

[0089] Where (v 0x ,v 0y ) is the control point motion vector on the upper left corner, and (v 1x ,v 1y ) is another control point motion vector for the upper right corner of the block. In some embodiments, for inter-mode coded CUs, when the CU size is equal to or greater than 16×16, affine_flag is signaled to indicate whether affine inter mode is applied or not. If the current CU is in affine inter mode, the candidate MVP pair list is constructed using adjacent valid reconstructed blocks. Figure 10 A four-parameter affine motion model is shown.

[0090] Figure 11 The MVP derivation for the affine inter mode is shown. Figure 11 As shown, V0 is selected from the motion vectors of blocks A0, A1, or A2, and V1 is selected from the motion vectors of blocks B0 and B1. The index of the candidate MVP pair is signaled in the bitstream. The MV difference (MVD) of the two control points is encoded and decoded in the bitstream.

[0091] In some embodiments, if the current PU is a merged PU, the adjacent five blocks ( Figure 11 The C0, B0, B1, C1 and A0 in the PU are checked to see if one of them is in affine inter mode or affine merge mode. If so, affine_flag is signaled to indicate whether the current PU is in affine mode. When the current PU is coded in affine merge mode, the first block is coded from the valid neighboring reconstructed blocks in affine mode. The order of candidate block selection is as follows: Figure 11 As shown, from left, top, top right, bottom left to top left (C0→B0→B1→C1→A0), the affine parameters of the first affine codec block are used to derive V0 and V1 of the current PU.

[0092] In some embodiments, the root CU (or parent CU) or common boundary size / depth / shape / width / height is used to derive the candidate list. In the candidate list derivation, for any position-based derivation, the root CU or common boundary position and shape / size / depth / width / height are used to derive the reference block position based on the current block / CU / PU / position / size / depth / shape / width / height. In some embodiments, for affine inheritance candidate derivation, the reference block position is derived first. When the common list is applied, the reference block position is derived by using the root CU or common boundary position and shape / size / depth / width / height. In one example, the reference block position is stored. When the sub-CU is in the root CU or common boundary, the stored reference block position is used to find the reference block for affine candidate derivation. In another example, when encoding and decoding the sub-CU, the position and shape / width / height / size of the root CU or common boundary can be stored or derived for affine candidate reference block derivation. The 4-parameter affine model and / or the 6-parameter affine model can be used to derive the affine candidate or control point MV. For example, in Figure 14 In the example, the CU in the root CU can refer to blocks A0, A1, B0, B1, B2 and the same block T BR and T CTR To derive affine candidates. In some embodiments, for affine inherited candidate derivation, the current sub-CU position and shape / size / depth / width / height are used. If the reference block is within the root CU or common boundary, it is not used to derive affine candidates.

[0093] g. Shared candidate list

[0094] In order to simplify the complexity of encoding and decoding operations, some embodiments provide a method of using a shared candidate list to encode or decode multiple blocks. The candidate list refers to a merge mode or AMVP mode candidate list or other types of prediction candidate lists (such as DMVR or bilateral refinement candidate list, affine merge mode, sub-block merge mode, affine inter frame / AMVP mode, IBC merge, IBC AMVP). The shared candidate list is a candidate list generated based on a boundary larger than the leaf CU (such as a parent CU or a root of a subtree in a QTBT or QTBTTT tree or a node of a QT tree), and the generated candidate list can be shared for all leaf CUs within the boundary or within the subtree. In some embodiments, a shared merge region (SMR) is used to mark the common ancestor node area of ​​the shared candidate list.

[0095] In some embodiments, a common candidate list is a candidate list shared by multiple CUs within or surrounded by a common common boundary, also referred to as a common boundary. For some embodiments, a "common boundary" is defined as a rectangular area of ​​smallest blocks aligned within an image (the smallest block is typically 4×4). Each CU within the "common boundary" can use a common candidate list generated based on the "common boundary". In particular, the candidates of the common candidate list include spatially adjacent positions and temporally adjacent positions based on the "common boundary", or an area defined by the common boundary. The common boundary can be a square block or a non-square block. The common boundary can be a square block or a non-square block. The size / depth / width / height of the common boundary can be signaled in the bitstream at the sequence level, the picture level, or the slice level.

[0096] Figure 12 Various examples of common boundaries for defining or identifying a common candidate list are shown. Examples include: a square common boundary 1210 corresponding to an 8x8 CU that is split into four 4x4 CUs by QT; a square common boundary 1220 corresponding to an 8x8 CU that is split into two 4x8 CUs by BT; a rectangular common boundary 1230 corresponding to a 4x16 root CU that is split into two 4x8 CUs by BT; and a rectangular common boundary 1240 corresponding to a 4x16 CU that is split into two 4x4 CUs and one 4x8 CU by TT. CUs corresponding to a common boundary are also referred to as root CUs of the common boundary.

[0097] In some embodiments, the common candidate list is a candidate list shared by multiple CUs within a subtree.A "subtree" may refer to a QTBT, a subtree of a QTBTTT, or another type of split tree. Figure 13 A subtree of a split tree corresponding to CUs in a common boundary is shown, which can be encoded or decoded using a common candidate list. The diagram shows a CTU 1300 whose split structure is represented by a hierarchical tree 1350, which is a QTBT or QTBTTT split tree. Within CTU 1300, a common boundary 1310 defines a root CU, which is split into various sub-CUs at various split depths. The sub-CUs surrounded by the common boundary 1310 correspond to subtree 1360 in split tree 1350, and the leaf nodes in subtree 1360 correspond to the leaf CUs in the common boundary 1310. In other words, the common candidate list is shared by the leaf nodes of subtree 1360.

[0098] The common candidate list may be generated based on a common block boundary, such as the common boundary 1310 or the root CU boundary of the subtree 1360. The common candidate list is applied to some or all leaf CUs within the subtree. The common candidate list is generated for the root of the subtree, i.e., the spatial and temporal neighbors of the candidates of the common candidate list are defined based on the rectangular boundary (or common boundary) of the root CU or the subtree.

[0099] The candidates of the common candidate list are prediction candidates identified based on spatial or temporal neighboring blocks of the area defined by the common boundary. One or more CUs surrounded by the common boundary are then encoded using one or more prediction candidates selected from the common candidate list.

[0100] Figure 14 A common boundary 1400 for identifying a common candidate list is shown. The common boundary 1400 defines an area 1410. The area 1410 may correspond to a CU. The area 1410 may be part of a split tree or CTU (e.g., BT, QT, MTT, etc.) and may be split into multiple sub-CUs, such as CUs 1411, 1412, 1413, and 1414. A sub-CU may be a leaf CU that cannot be further split. A sub-CU may also be split into sub-CUs of greater split depth until a leaf CU is reached. In the example, the area 1410 defined by the common boundary 1400 (or a root CU corresponding to the area 1410) is split into several leaf CUs, including leaf CUs 1421, 1422, and 1423.

[0101] Leaf-CUs 1421, 1422, and 1423 are encoded and decoded using a common candidate list, which is identified or defined based on a common boundary 1400. The common candidate list may be a merge mode list, an AMVP list, an IBC merge list, an IBC AMVP list, or another type of prediction candidate list. The common candidate list may include prediction candidates derived (e.g., inherited) from neighboring blocks of region 1410, such as spatial MVPs from spatial neighboring blocks A0, A1, B0, B1, B2 and temporal MVPs from temporal neighbors TBR, TCTR. Typically, the common candidate list may include merge mode candidates, AMP mode candidates, IBC merge mode candidates, IBC AMVP mode candidates, affine merge mode candidates, sub-block merge mode candidates, affine AMVP mode candidates, or other types of prediction candidates. The common candidate list also includes one or more CPRs or IBC candidates that reference reconstructed reference samples in the current image.

[0102] h. Merged Estimated Region (MER)

[0103] The Merge Estimation Region (MER) is a region where candidate blocks are in the same MER (the same as the current block being coded) that are excluded from the merge candidate list (the current block being coded). When MER is applied, a spatial merge candidate can be added to the merge candidate list only if the current CU and the neighboring CU providing the spatial merge candidate are in a different MER. Therefore, by checking whether the corresponding neighboring CU is in the same MER region as the current CU, the video encoder can ensure that the spatial merge candidate for the current CU can be derived separately (e.g., using parallel processing).

[0104] Figure 15 The figure conceptually illustrates spatial merge candidates for CUs identified based on a merge estimation region (MER). As shown, MER 1500 is defined to include CU 1510 and CU 1515. Neighboring blocks of CUs 1510 and 1515 within MER 1500 (e.g., spatial neighboring block 1520) are not available for providing spatial merge candidates for encoding and decoding CU 1510. Coded neighboring blocks outside MER 1500 (e.g., spatial neighboring block 1530) are available for providing spatial merge candidates.

[0105] MER can be extended to QTBT or QTBTTT structure. MER can be non-square. Depending on the structural partitioning, MER can be of different shapes or sizes. The size / depth / area / width / height can be predefined or signaled at the sequence / image / slice level. For the width / height of the MER, the value of the log2 of the width / height can be signaled. For the area / size of the MER, the value of the log2 of the size / area can be signaled. When a MER is defined for a region, the CU / PU in this MER cannot be used as a reference CU / PU for merge mode candidate derivation. For example, the MV or affine parameters of the CU / PU in this MER are not available for merge candidate or affine merge candidate derivation by the CU / PU residual in the same MER. These MVs and / or affine parameters are treated as unavailable for the CU / PU in the same MER. When the MER area / size / depth / shape / width / height is defined (e.g., predefined or signaled), if the current CU is greater than or equal to the defined area / size / shape / width / height and one of the sub-partitions, all sub-partitions, or some of the sub-partitions are smaller than the area / size / shape / width / height (or if the depth of the current CU is less than or equal to the defined depth and the depth of one of the sub-partitions, all sub-partitions, or some of the sub-partitions is greater than the defined depth), the current CU is a MER. In another embodiment, if the current CU is less than or equal to the defined area / size / shape / width / height and the parent CU is greater than the defined area / size / shape / width / height (or if the depth of the current CU is greater than or equal to the defined depth and the parent CU is less than the defined depth), the current CU is a MER. For example, if the defined area is 1024 and the CU size is 64×32 (width is 64 and height is 32), and is used for vertical TT split (64×32 CU is split into 16×32 sub-CUs, 32×32 sub-CUs, and 16×32 sub-CUs), in one embodiment, 64×32 is a MER. The sub-CUs in this 64×32 use a common list. In another embodiment, the 64×32 is not MER, but the 16×32 sub-CU, the 32×32 sub-CU, and the 16×32 sub-CU are each MER. In another embodiment, for a defined MER area / size / shape / width / height, when performing a TT split, the MER area / size / shape / width / height may be different in the TT partition. For example, for the first and second partitions, the MER area / size / shape / width / height thresholds may be divided by 2 (or the depth may be increased by 1), while for the third partition, the MER area / size / shape / width / height thresholds may remain the same.

[0106] In some embodiments, MER is defined for QT partitioning or QT split CU. If the QT CU is equal to or larger than the defined area / size / QT depth / shape / width / height, MER is defined as the leaf QT CU area / size / QT depth / shape / width / height. All sub-CUs within the QT leaf CU (partitioned by BT or TT) use the QT leaf CU as MER. MER includes all sub-CUs in this leaf QTCU. If the QT CU (non-QT leaf CU) is equal to the defined area / size / QT depth / shape / width / height, this QT CU is used as MER. All sub-CUs within the QT CU (such as partitioned by QT, BT or TT) are included in this MER. In one embodiment, the area / size / QT depth / shape / width / height of the MER is used to derive the reference block position. In another embodiment, the area / size / QT depth / shape / width / height of the current CU is used to derive the reference position. If the reference block position is within the MER, the reference block position is moved outside the MER. In another example, the area / size / QT depth / shape / width / height of the current CU is used to derive the reference block position. If the reference block position is inside the MER, the reference block is not used for merge candidate or affine merge candidate derivation.

[0107] In the depth mentioned above, the depth can be equal to (((A*QT-depth)>>C)+((B*MT-depth)>>D)+E)>>F+G or (((A*QT-depth)>>C)+((B*BT-depth)>>D)+E)>>F+G, where A, B, C, D, E, F, and G are integers. For example, the depth can be equal to 2*QT depth+MT depth or 2*QT depth+BT depth or QT depth+MT depth or QT depth+BT depth. In some embodiments, the MER area cannot cross the image boundary. That is, all MER areas must be within the image, and no pixels of the MER area are outside the image boundary. MER can also be applied to AMVP modes other than merging or affine merging. QTMTT-based MER can be applied to all candidate derivation tools (such as AMVP, merging, affine merging, etc.).

[0108] In some embodiments, both MER and common lists can be enabled in the QTMTT structure. In some embodiments, common lists are used for normal merging and ATMVP, and MER based on QTMTT is used for affine merging. In some embodiments, common lists are used for some prediction modes, while MER is used for other merge modes or AMVP mode.

[0109] I. History-Based Merge Candidate Derivation

[0110] After spatial MVP and TVMP, history-based MVP (HMVP) merge candidates are added to the merge list. In HMVP, the motion information of previously coded blocks is stored in a table and used as the MVP for the current CU. A table with multiple HMVP candidates is maintained during the encoding / decoding process. The table is reset (cleared) when a new CTU row is encountered. Regardless of whether there are non-subblock inter-coded CUs, the relevant motion information is added to the last entry of the table as a new HMVP candidate.

[0111] In some embodiments, the HMVP table size S is set to 6, which indicates that at most 6 history-based MVP (HMVP) candidates can be added to the table. When inserting a new motion candidate into the table, a constrained first-in-first-out (FIFO) rule is utilized, where a redundancy check is first applied to find whether there is an equal HMVP in the table. If found, the equal HMVP and all subsequent HMVP candidates are removed from the table and moved forward. The HMVP candidate can be used for the merge candidate list reconstruction process. The latest HMVP candidates in the table are checked in order and inserted into the candidate list after the TMVP candidate. Redundancy check is applied to the HMVP candidate to spatial or temporal merge candidate.

[0112] To reduce redundant checking operations, some embodiments use the following simplification:

[0113] (1) The number of HMVP candidates used for merge list generation is set to (N<=4) × M: (8−N), where N indicates the number of existing candidates in the merge list and M indicates the number of available HMVP candidates in the table.

[0114] (2) Once the total number of available merge candidates reaches the maximum allowed merge candidate minus 1, the merge candidate list reconstruction process from HMVP is terminated.

[0115] j. Intra-block copy (IBC)

[0116] Intra-block copy (IBC) is also known as current picture reference (CPR). An IBC (or CPR) motion vector is an MV that refers to a reconstructed reference sample in the current picture. For some embodiments, IBC prediction mode is used as a third prediction mode for encoding and decoding a CU, in addition to intra or inter prediction modes.

[0117] Because IBC mode is implemented as a block-level codec, block matching (BM) is performed at the encoder to find the best block vector (or motion vector) for each CU. Here, the block vector is used to indicate the displacement from the current block to the reference block, which has been reconstructed in the current image. The luma block vector of an IBC-coded CU is integer precision.

[0118] Chroma block vectors are also rounded to integer precision. When combined with AMVR, IBC mode can switch between 1-pixel and 4-pixel motion vector precision. IBC mode also applies to CUs with width and height less than or equal to 64 luma samples.

[0119] On the encoder side, hash-based motion estimation is performed for IBC. The encoder performs RD checks for blocks with a width or height no greater than 16 luma samples. For non-merge mode, a block vector search is first performed using a hash-based search. If the hash search does not return a valid candidate, a local search based on block matching is performed. In the hash-based search, the hash key match (32-bit CRC) between the current block and the reference block is extended to all allowed block sizes. The hash key calculation for each position in the current image is based on 4×4 sub-blocks. For larger sized current blocks, the hash key is determined to match the reference block when all hash keys of all 4×4 sub-blocks match the hash keys in the corresponding reference positions. If the hash keys of multiple reference blocks are found to match the hash key of the current block, the block vector cost of each matching reference is calculated and the one with the minimum cost is selected.

[0120] In block matching search, the search range is set to cover the previous and current CTU. At the CU level, the IBC mode is signaled with a flag, which can be signaled as IBC AMVP mode or IBC skip / merge mode as follows:

[0121] IBC skip / merge mode: The merge candidate index is used to indicate which block from the list of adjacent candidate IBC coded blocks to use to predict the current block. The merge list includes spatial, HMVP, and paired candidates.

[0122] IBC AMVP mode: Block vector differences are encoded and decoded in the same way as motion vector differences. The block vector prediction method uses two candidates as predictors, one from the left neighbor and one from the top neighbor (if IBC is used). When no neighbor is available, the default block vector is used as the predictor. A flag is signaled to indicate the block vector predictor index.

[0123] In some embodiments, to reduce memory consumption and decoder complexity, IBC only allows reconstruction of a portion of a predetermined area, which includes the area of ​​the current CTU and some areas of the left CTU. Figure 16 The reference area of ​​the IBC mode is shown, where each block represents a 64x64 unit of luma samples. The diagram shows the processing order of the current CTU and the reference samples available in the current CTU 1610 and the left CTU 1620.

[0124] Depending on the location of the current coded CU within the current CTU, the following applies:

[0125] -If the current block (or current coding CU) falls into the upper left 64×64 block of the current CTU, then in addition to the reconstructed samples in the current CTU, it can also use the CPR mode to refer to the reference samples of the lower right 64×64 block of the left CTU. The current block can also use the CPR mode to refer to the reference samples of the lower left 64×64 block of the left CTU and the reference samples of the upper right 64×64 block of the left CTU.

[0126] - If the current block falls into the upper right 64×64 block of the current CTU, then in addition to the reconstructed samples in the current CTU, if the luma position (0, 64) relative to the current CTU has not been reconstructed, the current block can also use the CPR mode to refer to the reference samples of the lower left 64×64 block and the lower right 64×64 block of the left CTU. Otherwise, the current block can also refer to the reference samples of the lower right 64×64 block of the left CTU.

[0127] - If the current block falls into the lower left 64×64 block of the current CTU, then in addition to the reconstructed samples in the current CTU, if the luma position (64,0) relative to the current CTU has not been reconstructed, the current block can also use the CPR mode to refer to the reference samples of the upper right 64×64 block and the lower right 64×64 block of the left CTU. Otherwise, the current block can also use the CPR mode to refer to the reference samples of the lower right 64×64 block of the left CTU.

[0128] If the current block falls into the lower right 64×64 block of the current CTU, it can only use the CPR mode to reference reconstructed samples in the current CTU.

[0129] For some embodiments, these constraints allow IBC mode to be implemented using local on-chip memory implemented in hardware.

[0130] The interaction between IBC mode and other inter coding tools, such as paired merge candidates, history-based motion vector predictor (HMVP), combined intra / inter prediction mode (CIIP), merge mode with allowed vector differences (MMVD), and triangle partitioning, is as follows:

[0131] - IBC can be used with paired merge candidates as well as HMVP. A new paired IBC merge candidate can be generated by averaging two IBC merge candidates. For HMVP, IBC motion is inserted into the history buffer for future reference.

[0132] -IBC cannot be used in combination with the following interframe tools: affine motion, CIIP, MMVD, and triangle segmentation.

[0133] - When using DUAL_TREE partitioning, IBC is not allowed for chroma coded blocks.

[0134] Unlike the HEVC screen content codec extension, the current picture is no longer included as a reference picture in reference picture list 0 for IBC prediction. The derivation process of motion vectors for IBC mode excludes all neighboring blocks in inter mode, and vice versa. Applying the following IBC design:

[0135] - IBC shares the same process as regular MV merging, including pairwise merging candidates and history-based motion predictors, but TMVP and 0 vectors are not allowed since they are invalid for IBC mode.

[0136] - Separate HMVP buffers (5 candidates each) for traditional MV and IBC.

[0137] -Block vector constraints are enforced in the form of bitstream consistency constraints. The encoder needs to ensure that there are no invalid vectors in the bitstream, and if the merge candidate is invalid (out of range or 0), the merge will not be used. This bitstream consistency constraint is expressed in the form of a virtual buffer as follows.

[0138] - For deblocking, IBC is handled as inter mode.

[0139] - If the current block is coded using IBC mode, AMVP does not use quarter pixels, instead, AMVR is signaled to only indicate whether the MV is an integer pixel or 4 integer pixels.

[0140] - The number of IBC merge candidates may be signaled in the slice header, separately from the number of regular, sub-block, and triangle merge candidates.

[0141] In some embodiments, a virtual buffer is used to describe the available reference area for IBC prediction mode and valid block vectors. The CTU size is marked as ctbSize. The virtual buffer is marked as ibcBuf. The virtual buffer has a width of wlbBuf = 128*128 / ctbSize and a height of hlbcBuf = ctbSize. For example, for a CTU of size 128×128, the size of ibcBuf is also 128×128, for a CTU of size 64×64, the size of ibcBuf is 256×64, and for a CTU of size 32×32, the size of ibcBuf is 512×32. The size of the VPDU is min(ctbSize, 64) in each dimension, Wv = min(ctbSize, 64). The virtual IBC buffer, ibcBuf is maintained as follows:

[0142] - At the beginning of each CTU row, flush the entire ibcBuf with an invalid value of -1.

[0143] - At the start of decoding the VPDU (xVPDU, yVPDU) relative to the upper left corner of the picture, set ibcBuf[x][y] = -1, where xVPDU % wIbcBuf, ..., xVPDU % wIbcBuf + Wv–1, yVPDU % ctbSize, ..., yVPDU % ctbSize + Wv–1.

[0144] - After decoding the CU constraint (x, y) relative to the top left corner of the picture, set ibcBuf[x % wIbcBuf][y % ctbSize] = recSample[x][y].

[0145] For a block covering coordinates (x,y), if the subsequent block vector bv = (bv[0], bv[1]) is true, then it is valid, otherwise it is invalid.

[0146] Additionally, ibcBuf[(x+bv[0])%wIbcBuf][(y+bv[1])%ctbSize] will not be equal to -1.

[0147] k. Expanded Merger Forecast

[0148] In some embodiments, the merge candidate list is constructed by including the following five types of candidates:

[0149] -Spatial MVP from spatially adjacent CUs

[0150] -Time MVP from the same CU

[0151] -History-based MVP from FIFO table

[0152] -Pairwise average MVP

[0153] -0MV

[0154] In some embodiments, the size of the merge list is signaled in the slice header. The maximum allowed size of the merge list is 6. For each CU coded in merge mode, the index of the best merge candidate is encoded using truncated unary binarization (TU). The first bin of the merge index is coded using the context, and the remaining bins are bypassed.

[0155] Figure 17 The positions of spatial merging candidates are shown. Figure 18 The candidate pairs considered for redundancy check of spatial merging candidates are shown. In some embodiments, from the Figure 17The maximum of the four merge candidates is found among the candidates at positions A0, B0, B1, A1 and B2 shown. Position B2 is considered only if any of the CUs at positions A0, B0, B1, A1 is not available (e.g., because it belongs to another slice or tile) or is intra-coded. After the candidate at position A1 is added, the addition of the remaining candidates is subject to a redundancy check, which ensures that candidates with the same motion information are excluded from the list to improve coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only Figure 18 Pairs connected as shown by arrows in are considered and candidates are only added to the list if the corresponding candidates for redundancy checking do not have the same information.

[0156] In this step, only one candidate is added to the list.In particular, in the derivation of the temporal merging candidate, a scaled motion vector is derived based on the collocated CU belonging to the collocated reference picture. Figure 19 The motion vector scaling for temporal merge candidates is shown. The reference picture list for the derivation of the collocated CU is explicitly signaled in the slice header. Figure 19 The dashed line shows the scaled motion vector obtained for the temporal merge candidate, which is scaled from the motion vector of the co-located CU using the POC distance (tb and td), where tb is defined as the POC difference between the reference picture of the current picture and the current picture and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal merge candidate is set to 0.

[0157] Figure 20 The position of the temporal candidate selected between candidates C0 and C1 is shown. If the CU at position C0 is not available, is intra-coded, or is outside the current row of the CTU, position C1 is used. Otherwise, position C0 is used in the derivation of the temporal merge candidate.

[0158] After spatial MVP and TMVP, history-based MVP (HMVP) candidates are added to the merge list. In this method, the motion information of previously coded blocks is stored in a table and used as the MVP for the current CU. The table with multiple HMVP candidates is retained during the encoding / decoding process. When a new CTU row is encountered, the table is reset (cleared). Regardless of whether there are non-sub-block inter-coded CUs, the relevant motion information is added to the last entry of the table as a new HMVP candidate.

[0159] In some embodiments, the HMVP table size S is set to 6. This indicates that at most 6 history-based MVP (HMVP) candidates can be added to the table. When a new motion candidate is inserted into the table, a constrained first-in-first-out (FIFO) rule is used, where a redundancy check is first applied to find whether there is an identical HMVP in the table. If found, the identical HMVP is removed from the table and all candidates after the HMVP candidate are moved forward. The latest several HMVP candidates in the table are checked in turn and inserted into the candidate list after the TMVP candidate. Redundancy checks are applied to HMVP candidates to spatial or temporal merge candidates.

[0160] To reduce the number of redundant checking operations, the following simplifications are introduced:

[0161] - The number of HMVP candidates used for merge list generation is set to (N<=4)?M:(8-N), where N indicates the number of existing candidates in the merge list and M indicates the number of available HMVP candidates in the table.

[0162] - Once the total number of available merge candidates reaches the maximum allowed merge candidate minus 1, the merge candidate list construction process from HMVP is terminated.

[0163] The average candidate is generated by averaging predetermined pairs of candidates in the existing merge candidate lists, and the predetermined pairs are defined as {(0,1), (0,2), (1,2), (0,3), (1,3), (2,3)}, where the number represents the merge index of the merge candidate list. The average motion vector is calculated separately for each reference list. If two motion vectors are available in a list, they are averaged even when pointing to different reference images. If only one motion vector is available, this one is used directly. If no motion vector is available, this list is kept invalid. When the merge list is not full after adding pairwise average merge candidates, 0MVPs are inserted at the end until the maximum number of merge candidates is reached.

[0164] l. Merge mode with motion vector difference (MMVD)

[0165] In addition to the merge mode, in which explicitly derived motion information is directly used for prediction sample generation for the current CU, including the merge mode with motion vector difference (MVD), in some embodiments, the MMVD flag is signaled after the skip flag and the merge flag to specify whether MMVD mode is used for the CU.

[0166] In MMVD, after a merge candidate is selected, it is further refined using signaled MVD information. This further information includes a merge candidate flag, an index specifying the magnitude of motion, and an index indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected as the MV basis. The merge candidate flag is signaled to specify which candidate to use.

[0167] In some embodiments, the distance index specifies motion magnitude information and indicates a predetermined offset from a starting point. Figure 21 As shown, the offset is added to the horizontal component or the vertical component of the starting MV. The association of the distance index and the predetermined offset is specified in the following Table 1-1.

[0168] Table 1-1 Distance Index

[0169]

[0170] The direction index represents the direction of the MVD relative to the starting point. The direction index can represent four directions as shown in Table 1-2. Note that the meaning of the MVD symbol can change according to the information of the starting MV. When the starting MV is a single prediction MV or a bidirectional prediction MV with both lists pointing to the same side of the current image (i.e., the POCs of both references are greater than the POC of the current image, or both are less than the POC of the current image), the symbols in Table 1-2 specify the sign of the MV offset added to the starting MV. When the starting MV is a bidirectional prediction MV with two MVs pointing to different sides of the current image (i.e., the POC of one reference is greater than the POC of the current image, and the POC of the other reference is less than the POC of the current image), the symbols in Table 1-2 indicate that the sign of the MV offset of the list 0 MV component added to the starting MV and the sign of the list 1 MV has the opposite value.

[0171] Table 1-2 Symbols for specifying MV offsets by direction index

[0172] Direction Index 00 01 10 11 X-axis + – N / A N / A Y-axis N / A N / A + –

[0173] In some embodiments, in addition to normal uni-prediction and bi-prediction mode MVD signaling, a symmetric MVD mode for bi-prediction MVD signaling is applied. In the symmetric MVD mode, the motion information including the reference picture indices of List 0 and List 1 and the MVD of List 1 are not signaled but derived. The decoding process of the symmetric MVD mode is as follows:

[0174] At the stripe level, the variables BiDirPredFlag, RefIdxSymL0 and RefIdxSymL1 are derived as follows:

[0175] - If mvd_l1_zero_flag is 1, BiDirPredFlag is set equal to 0.

[0176] Otherwise, if the most recent reference picture in list 0 and the most recent reference picture in list 1 form a forward and backward pair of reference pictures or a backward and forward pair of reference pictures, and both the reference pictures in list 0 and list 1 are short-term reference pictures, BiDirPredFlag is set to 1. Otherwise, BiDirPredFlag is set to 0.

[0177] At the CU level, if the CU is bidirectionally predictively coded and BiDirPredFlag is equal to 1, the symmetric mode flag is explicitly signaled, indicating whether symmetric mode is used. When this mode flag is true, only mvp_l0_flag, mvp_l1_flag, and MVD0 are explicitly signaled. The reference indexes of list 0 and list 1, respectively, are set equal to the reference picture pair. MVD1 is set equal to (-MVD0). The resulting motion vector is shown in the following equation.

[0178]

[0179] At the encoder, symmetric MVD motion estimation begins with an initial MV evaluation. A set of initial MV candidates includes MVs obtained from a single search, MVs obtained from a unidirectional prediction search, and MVs from the AMVP list. The MV with the lowest rate-distortion cost is selected as the initial MV for the symmetric MVD motion search.

[0180] m. Affine candidate

[0181] Affine merge prediction or AF_MERGE mode can be applied to CUs with width and height greater than or equal to 8. In this mode, motion vectors are associated with the control points of the affine model, or the control point motion vector (CPMV) of the current CU is generated based on the motion information of spatially neighboring CUs. There may be up to 5 CPMVP candidates and the index is signaled to indicate the one to be used for the current CU. The following three types of CPMV candidates are used to form the affine merge candidate list:

[0182] - Inherited affine merge candidates inferred from the CPMV of neighboring CUs.

[0183] -Affine merged candidate CPMVP constructed using the translation MV of the adjacent CU

[0184] -0MV

[0185] Constructed affine candidates means that the candidate is constructed by combining the translational motion information of each control point. The motion information of the control point is derived from the specified spatial and temporal neighboring blocks.

[0186] Affine AMVP prediction or Affine AMVP mode can be applied for CUs with width and height both greater than or equal to 16. An affine flag in the CU level is signaled in the bitstream to indicate whether affine AMVP mode is used and then another flag is signaled to indicate whether it is 4-parameter affine or 6-parameter affine. In this mode, the difference of the CPMV of the current CU and their predicted sub-CPMVP is signaled in the bitstream. The affine AMVP candidate list size is 2 and it is generated by using the following four types of CPVM in sequence:

[0187] - Inherited affine AMVP candidates inferred from the CPMV of neighboring CUs.

[0188] - Affine AMVP candidates constructed using the shifted MVs of neighboring CUs.

[0189] -Translated MV from neighboring CU.

[0190] -0MV.

[0191] The order in which inherited affine AMVP candidates are checked is the same as the order in which inherited affine merge candidates are checked. The only difference is that for AVMP candidates, only affine CUs with the same reference picture as the current block are considered. When the inherited affine motion predictor is inserted into the candidate list, no pruning process is applied.

[0192] In some embodiments, the CPMV of an affine CU is stored in a separate buffer. The stored CPMV is only used to generate the inherited CPMVP for the most recently encoded CU in affine merge mode and affine AMVP mode. The sub-block MVs derived from the CPMVs are used for motion compensation, merging of translation MVs / MV derivation of AMVP lists, and deblocking.

[0193] To avoid the need for an extra CPMV in the image linear buffer, the affine data inheritance of the CU from the upper CTU is treated differently from the inheritance from a normal neighboring CU. If the candidate CU for affine motion data inheritance is in the upper CTU line, the bottom left and bottom right sub-block MVs in the linear buffer are used for affine MVP derivation instead of the CPMVs. This way, the CPMVs are only stored in the local buffer. If the candidate CU is a 6-parameter affine codec, the affine model degenerates to a 4-parameter model.

[0194] n.Sub-block based temporal motion vector prediction (SbTMVP)

[0195] In some embodiments, a sub-block based temporal motion vector prediction (SbTMVP) method is used. Similar to temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the collocated picture to improve the motion vector prediction and merge mode of the CU in the current picture. The same collocated picture used by TMVP is used for SbTVMP. SbTMVP differs from TMVP in that (1) TMVP predicts motion at the CU level while SbTMVP predicts motion at the sub-CU level, and (2) whereas TMVP extracts motion vectors from the collocated blocks of the collocated picture (the collocated block is the bottom right or center block relative to the current CU), SbTMVP applies a motion shift before extracting temporal motion information from the collocated picture, where the motion shift is obtained from the motion vector from one of the spatially neighboring blocks of the current CU.

[0196] Figure 22 The SbTMVP process is conceptually shown. The sub-CU motion field is derived by applying motion offsets from spatially neighboring blocks and scaling the motion information from the corresponding co-located sub-CU. In two steps, SbTMVP predicts the motion vectors of the sub-CU within the current CU. In the first step, check Figure 22 If A1 has a motion vector that uses the collocated image as its reference image, this motion vector is selected as the motion offset to be applied. If no such motion is identified, the motion offset is set to (0,0). In the second step, as Figure 22 As shown, the motion offset identified in the first step is applied (eg, added to the coordinates of the current block) to obtain sub-CU level motion information (motion vector and reference index) from the collocated image. Figure 22 The example assumes that the motion offset is set to the motion of block A1. Then, for each sub-SU, the motion information of its corresponding block in the concatenated image (the minimum motion grid covering the center sample) is used to derive the motion information of the sub-CU. After identifying the motion information of the co-located sub-CU, it is converted into the motion vector of the current sub-CU and the reference index in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference image of these current CU's temporal motion vectors.

[0197] In some embodiments, a combined sub-block merge list containing both SbTMVP candidates and affine merge candidates is used to signal the sub-block based merge mode. The SbTMVP mode is enabled / disabled by a sequence parameter set (SPS) flag and if the SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry in the sub-block based merge candidate list followed by the affine merge candidate. In some embodiments, the size of the sub-block based merge list is signaled in the SPS and the maximum allowed size of the sub-block based merge list is 5. The sub-CU size used in SbTMVP is fixed to 8×8 and like the affine merge mode, the SbTMVP mode is only applicable to CUs with both width and height greater than or equal to 8. The encoding logic for the additional SbTMVP merge candidates is the same as for the other merge candidates, i.e., for each CU in a P or B slice, an additional RD check is performed to decide whether to use the SbTMVP candidate.

[0198] o. Group Partitioning Mode (GPM)

[0199] Under geometric partitioning, a transform block may obtain pixels belonging to multiple (e.g., two) different partitions, and multiple different motion vectors are used for motion compensation of the multiple different partitions. As a result, pixels on the boundaries between different partitions may have large discontinuities, which may generate visual artifacts similar to blocking effects. This in turn reduces transform efficiency.

[0200] Figure 23 as well as Figure 24 The blocks created by geometric partitioning are shown. Pixels shaded in grey belong to region 1 and pixels shaded in white belong to region 2. A pixel of region 1 (or region 2) is defined as a boundary pixel if any of its four connected neighboring pixels (left, top, right, and bottom) belongs to region 2 (or region 1). If the pixel is a boundary pixel, motion compensation is performed using the weighted sum of the motion predictions of the two motion vectors of the two regions. The weight for the prediction of the motion vector of the region containing the boundary pixel is 3 / 4 and the weight for the prediction using the motion vector of the other region is 1 / 4. Overlapping boundaries improve the visual quality of the reconstructed video and also provide BD rate gain.

[0201] p. Combined Inter and Intra Prediction (CIIP)

[0202] Combined inter / intra prediction (CIIP) combines the inter prediction signal with the intra prediction signal. The inter prediction signal Pinter in CIIP mode is derived using the same inter prediction process applied to the regular merge mode, and the intra prediction signal Pintra is derived after the regular intra prediction process with planar mode. Then, the intra and inter prediction signals are combined using a weighted average. The weights are calculated based on the codec mode of the neighboring blocks above and to the left as follows:

[0203] - If the top neighboring block is available and intra-coded, then set isIntraTop to 1, otherwise set isIntraTop to 0.

[0204] If the left adjacent block can be intra-coded, set isIntraLeft to 1; otherwise, set isIntraLeft to 0.

[0205] - If (isIntraLeft + isIntraLeft) is equal to 2, then wt is set to 3;

[0206] - Otherwise, if (isintraLeft + isIntraLeft) is equal to 1, then wt is set to 2;

[0207] Otherwise, set wt to 1.

[0208] The CIIP forecast is formed as follows:

[0209] P CIIP =((4-wt)*P inter +wt*P intra +2)>>2

[0210] In some embodiments, when a CU is encoded in merge mode, if the CU contains at least 64 luma samples (i.e., CU width multiplied by CU height is equal to or greater than 64), and if both the CU width and the CU height are less than 128 luma samples, an additional flag is signaled to indicate whether CIIP mode is applied to the current CU.

[0211] III Merged Estimation Region (MER) and Merging Mode

[0212] Since the IBC mode refers to reference pixels on the current image, and since the motion compensation of the current CU needs to wait until the reconstruction of the previous CU, the parallel processing of IBC merge list generation is useless.

[0213] In some embodiments, the MER mode is applicable to sub-block merge lists (including affine merge and SbTMVP) and non-sub-block non-IBC merge lists (including GPM, MMVD merge, regular merge, CIIP merge), but is not used for IBC merge (and / or IBC AMVP). That is, for IBC merge (and / or IBC AMVP) mode, when generating the merge list, the video encoder can refer to spatially neighboring blocks within the same MER region.

[0214] In some embodiments, MER mode (including GPM, MMVD merge, regular merge, CIIP merge) is used in sub-block merges and non-sub-block non-IBC merges, but is disabled for IBC merges (and / or IBC AMVP). In particular, when deriving the spatial neighboring blocks of the merge list of the current CU in non-IBC mode, the spatial neighboring blocks in the MER region are excluded (set to unavailable) or pushed to the boundary of the MER region. For some embodiments, the video encoder decides whether to use the spatial neighboring blocks as merge candidates based on the following virtual code:

[0215]

[0216] In other words, if the IBC mode is used to encode the current CU, the spatial neighboring blocks of the current CU can be used as merge candidates regardless of whether the spatial neighboring blocks are in the same MER as the current CU. On the contrary, if the IBC mode is not used, the spatial neighboring blocks in the same MER as the current CU are excluded from the merge list as merge candidates.

[0217] Figure 25 The figure conceptually illustrates the inclusion and exclusion of spatially neighboring blocks from a merge candidate list based on MER and IBC modes. As shown, a current image 2500 has a MER 2505 defined therein. MER 2505 includes several CUs, including 2510 and CU 2515. The figure also illustrates several spatially neighboring blocks, including spatially neighboring blocks 2520-2526 within MER 2505, spatially neighboring blocks 2530-2536 outside MER 2505, and spatially neighboring blocks 2540-2542 being encoded and decoded after the CUs of MER 2505.

[0218] If IBC mode is off (or not applied) for CU 2510, spatial neighboring blocks 2530-2536 outside of MER 2505 may be used as merge candidates for encoding CU 2510. Spatial neighboring blocks 2540-2542 are within MER 2505 and are excluded (or unavailable) as merge candidates for encoding CU 2510. On the other hand, if IBC mode is applied for CU 2515, spatial neighboring blocks 2530-2536 and 2540-2542 may both be used (or available) as merge candidates regardless of MER 2505 for encoding CU 2515.

[0219] The aforementioned methods can be implemented at an encoder and / or a decoder. For example, the proposed methods can be implemented at an inter-frame prediction module of an encoder and / or an inter-frame prediction module of a decoder. In some embodiments, the encoder can signal (or generate) one or more syntax elements in a bitstream so that the decoder can parse the one or more syntax elements from the bitstream.

[0220] Exemplary Video Encoder

[0221] Figure 26 An exemplary video encoder is shown that can use a prediction candidate list (for merge mode or AMVP) to generate predictions when encoding a block. As shown, a video encoder 2600 receives an input video signal from a video source 2605 and encodes the signal into a bitstream 2695. The video encoder 2600 has various components or modules for encoding the signal from the video source 2605, including at least some components selected from a transform module 2610, a quantization module 2611, an inverse quantization module 2614, an inverse transform module 2615, an intra-frame image estimation module 2620, an intra-frame image prediction module 2625, a motion compensation module 2630, a motion estimation module 2635, a loop filter 2645, a reconstructed picture buffer 2650, an MV buffer 2665, an MV prediction module 2675, and an entropy encoder 2690. The motion compensation module 2630 and the motion estimation module 2635 are part of the inter-frame prediction module 2640.

[0222] In some embodiments, modules 2610-2690 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device or electronic device. In some embodiments, modules 2610-2690 are modules of hardware circuitry implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 2610-2690 are shown as separate modules, some modules may be combined into a single module.

[0223] Video source 2605 provides raw video data representing uncompressed pixel data for each video frame. Summer 2608 calculates the difference between the raw video pixel data from video source 2605 and the predicted pixel data from motion compensation module 2630 or intra prediction module 2625. Transform module 2610 converts the difference (or residual pixel data or residual signal 2609) into transform coefficients 2616 (e.g., by performing a discrete cosine transform or DCT). Quantization module 2611 quantizes the transform coefficients into quantized data (or quantized coefficients) 2612, which can be encoded into a bitstream 2695 by entropy encoder 2690.

[0224] The inverse quantization module 2614 dequantizes the quantized data (or quantized coefficients) 2612 to obtain transform coefficients, and the inverse transform module 2615 performs an inverse transform on the transform coefficients to generate a reconstructed residual 2619. The reconstructed residual is added to the predicted pixel data 2613 to generate reconstructed pixel data 2617. In some embodiments, the reconstructed pixel data 2617 is temporarily stored in a linear buffer (not shown) for intra-frame image prediction and spatial MV prediction. The reconstructed pixels are filtered by the loop filter 2645 and stored in a reconstructed image buffer 2650. In some embodiments, the reconstructed image buffer 2650 is an external memory of the video encoder 2600. In some embodiments, the reconstructed image buffer 2650 is an internal memory of the video encoder 2600.

[0225] The intra-frame image estimation module 2620 performs intra-frame prediction based on the reconstructed pixel data 2617 to generate intra-frame prediction data. The intra-frame prediction data is provided to the entropy encoder 2690 to be encoded into the bitstream 2695. The intra-frame prediction data is also used by the intra-frame prediction module 2625 to generate predicted pixel data 2613.

[0226] In addition to encoding the complete actual MV in the bitstream, the video encoder 2600 uses MV prediction to generate a predicted MV, and the difference between the MV used for motion compensation and the predicted MV is encoded as residual motion data and stored in the bitstream 2695.

[0227] MV prediction module 2675 generates a predicted MV based on a reference MV generated for encoding a previous video frame, i.e., a motion-compensated MV used to perform motion compensation. MV prediction module 2675 retrieves the reference MV from a previous video frame from MV buffer 2665. Video encoder 2600 stores the MV generated for the current video frame in MV buffer 2665 as a reference MV used to generate the predicted MV.

[0228] The MV prediction module 2675 uses the reference MV to create a predicted MV. The predicted MV can be calculated by spatial MV prediction or temporal MV prediction. The difference (residual motion data) between the predicted MV and the motion-compensated MV (MC MV) of the current frame is encoded into a bitstream 2695 by the entropy encoder 2690.

[0229] The entropy encoder 2690 encodes various parameters and data into a bitstream 2695 using entropy coding techniques, such as context-adjusted arithmetic coding (CABAC) or Huffman coding. The entropy encoder 2690 also encodes quantized transform coefficients 2612, various header elements, flags, and residual motion data as syntax elements into the bitstream 2695. The bitstream 2695 is then stored on a storage device or transmitted to a decoder via a communication medium such as a network.

[0230] The loop filter 2645 performs a filtering or smoothing operation on the reconstructed pixel data 2617 to reduce coding artifacts, particularly at block boundaries. In some embodiments, the filtering operation performed includes sample adaptive offset (SAO). In some embodiments, the filtering operation includes adaptive loop filtering (ALF).

[0231] Figure 27 The diagram shows a portion of a video encoder that identifies prediction candidates based on MER and IBC modes. Specifically, the diagram illustrates elements of the intra prediction module 2640 of the video encoder 2600. As shown, the prediction candidate list identification module 2710 determines which motion vectors from the MV buffer 2665 are included in the prediction candidate list for the current CU (either for merge mode or for AMVP). The video encoder provides a signal to indicate whether IBC mode is used for the current CU and to define MER. The IBC mode indication signal and the MER definition are provided to the entropy encoder 2690 for inclusion in the bitstream as syntax elements.

[0232] The prediction candidate list identification module 2710 uses the IBC mode and the MER definition to decide whether to include or exclude certain motion vectors. For example, when MER is enabled and IBC is disabled for the current CU, the prediction candidate list identification module 2710 may exclude motion vectors from the spatial neighboring blocks of the current CU in the same MER as the current CU, and if the IBC mode is enabled for the current CU, the prediction candidate list identification module 2710 may include motion vectors from the neighboring blocks of the current CU in the same MER as the current CU, i.e., the spatial neighboring blocks of the current CU may be included regardless of the MER definition.

[0233] The prediction candidate list identified for the current CU by the prediction candidate list identification module 2710 is in turn provided to the prediction generation module 2720, which generates a prediction for the current CU according to one of the possible prediction modes (CIIP, MMVD, GPM, affine, etc.) for the motion compensation module 2630.

[0234] Figure 28 A process 2800 for encoding a block in a video image using prediction candidates based on MER and IBC pattern recognition is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implement encoder 2600 and execute instructions stored on a computer-readable medium to perform process 2800. In some embodiments, an electronic device implementing encoder 2600 performs process 2800.

[0235] The encoder receives (at block 2810) raw data of a bitstream for a current block of a current picture to be encoded as a video. The encoder determines (at block 2820) whether to encode the current block using a prediction candidate (e.g., a list of merge mode candidates or a list of AMVP candidates). In some embodiments, the encoder signals a syntax element to the bitstream indicating whether to encode the current block using the prediction candidate. If the current block is to be encoded using the prediction candidate, the process proceeds to block 2830. Otherwise, the encoder encodes (at block 2825) the current block without using the prediction candidate.

[0236] At block 2830, the encoder identifies potential candidates included in the list of prediction candidates for the current block. The encoder determines (at block 2840) whether the current block is coded by intra block copy mode and (at block 2842) whether the list of prediction candidates belongs to or is derived from a predetermined subset of different candidate lists. In some embodiments, the predetermined subset of different candidate lists includes merge mode and merge-like candidates (e.g., CIIP, GPM, regular merge) and does not include AMVP candidates. If the current block is coded by intra block copy mode or if the list of prediction candidates is a list for AMVP, the process proceeds to 2850. Otherwise (the current block is not coded by IBC and the list of prediction candidates is merge or merge-like), the process proceeds to 2845.

[0237] When the current block is encoded using intra block copy mode, a motion vector that references coded reference samples in the current image is used to generate a prediction for the current block. When the current block is not encoded using intra block copy mode, the current block and at least one other block in the MER may be encoded in parallel. In some embodiments, the encoder signals one or more syntax elements in the bitstream to indicate whether intra block copy mode is used for the current block.

[0238] The encoder determines (at block 2845) whether the candidate is from a spatially neighboring block that is in the same MER as the current block. In some embodiments, one or more syntax elements in the bitstream may be used to identify or define the MER that includes the current block. If the spatially neighboring block is in the same MER as the current block, the process proceeds to block 2855. Otherwise, the process proceeds to 2850.

[0239] At block 2850, the encoder includes a candidate in the prediction candidate list for the current block. If there are additional potential candidates included in the prediction candidate list, the process may return to block 2830. If there are no additional potential candidates, the process proceeds to 2860.

[0240] At block 2855, the encoder excludes candidates from the prediction candidate list for the current block. In some embodiments, when the current block is not encoded using intra block copy mode, all spatially neighboring blocks of the current block located in the MER are excluded from the list of prediction candidates. If there are additional potential candidates included in the prediction candidate list, the process may return to block 2830. If there are no more potential candidates, the process proceeds to 2860.

[0241] The encoder encodes (at block 2860) the current block into the bitstream by generating a prediction for the current block using a prediction candidate selected from the prediction candidate list. The prediction for the current block may be an affine prediction generated based on the selected prediction candidate. The prediction for the current block may be a triangular or GPM prediction generated based on the selected prediction candidate. The prediction for the current block may be a combined inter- and intra-frame prediction (CIIP) generated based on the selected prediction candidate. The prediction for the current block may be generated by refining the selected prediction candidate with motion vector difference information.

[0242] Exemplary video decoder

[0243] Figure 29 An exemplary video decoder 2900 is shown that can use a prediction candidate list (for merge mode or AMVP) to generate predictions when decoding blocks. As shown, video decoder 2900 is an image decoding or video decoding circuit that receives a bitstream 2995 and decodes the contents of the bitstream into pixel data of a video frame for display. Video decoder 2900 has various components or modules for decoding bitstream 2995. These include selected components from an inverse quantization module 2911, an inverse transform module 2910, an intra-frame prediction module 2925, a motion compensation module 2930, a loop filter 2945, a decoded image buffer 2950, ​​an MV buffer 2965, an MV prediction module 2975, and a parser 2990. Motion compensation module 2930 is part of inter-frame prediction module 2940.

[0244] In some embodiments, modules 2910-2990 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 2910-2990 are modules of hardware circuitry implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 2910-2990 are shown as separate modules, some modules may be combined into a single module.

[0245] The parser 2990 (or entropy decoder) receives the bitstream 2995 and performs initial parsing according to the syntax defined by the video codec or image codec standard. The parsed syntax elements include various header elements, flags, and quantized data (or quantized coefficients) 2912. The parser 2990 parses the various syntax elements using entropy coding techniques such as context-sensitive binary arithmetic coding (CABAC) or Huffman coding.

[0246] The inverse quantization module 2911 dequantizes the quantized data (or quantized coefficients) 2912 to obtain transform coefficients, and the inverse transform module 2910 performs an inverse transform on the transform coefficients 2916 to generate a reconstructed residual signal 2919. The reconstructed residual signal 2919 is added to the predicted pixel data from the intra prediction module 2925 or the motion compensation module 2930 to generate decoded pixel data 2917. The decoded pixel data is filtered by the loop filter 2945 and stored in the decoded picture buffer 2950. In some embodiments, the decoded picture buffer 2950 is external to the video decoder 2900. In some embodiments, the decoded picture buffer 2950 is internal to the video decoder 2900.

[0247] The intra prediction module 2925 receives intra prediction data from the bitstream 2995 and generates predicted pixel data 2913 based on the intra prediction data from the decoded pixel data 2917 stored in the decoded picture buffer 2950. In some embodiments, the decoded pixel data 2917 is also stored in a linear buffer (not shown) for intra picture prediction and spatial MV prediction.

[0248] In some embodiments, the contents of the decoded image buffer 2950 are used for display. The display device 2955 retrieves the contents from the decoded image buffer 2950 directly for display, or retrieves the contents of the decoded image buffer into a display buffer. In some embodiments, the pixel device receives pixel values ​​from the decoded image buffer 2950 via a pixel interface.

[0249] The MV prediction module 2975 generates a predicted MV based on a reference MV generated for decoding a previous video frame, such as a motion-compensated MV for performing motion compensation. The MV prediction module 2975 retrieves the reference MV for the previous video frame from the MV buffer 2965. The video decoder 2900 stores the motion-compensated MV generated for decoding the current video frame in the MV buffer 2965 as a reference MV for the predicted MV.

[0250] The loop filter 2945 performs a filtering or smoothing operation on the decoded pixel data 2917 to generate coding artifacts, particularly at block boundaries. In some embodiments, the filtering operation performed includes sample adaptive offset (SAO). In some embodiments, the filtering operation includes an adaptive loop filter (ALF).

[0251] Figure 30 The diagram shows a portion of a video decoder 2900 that identifies prediction candidates based on MER and IBC modes. In particular, the diagram shows elements of the inter prediction module 2940 of the video decoder 2900. The prediction candidate list identification module 3010 determines which motion vectors from the MV buffer 2965 are included in the prediction candidate list as prediction candidates for the current CU.

[0252] As shown, the entropy decoder 2990 parses the syntax elements that signal whether IBC mode is used for the current CU and the syntax elements that define the MER from the bitstream. The prediction candidate list identification module 3010 uses the IBC mode indication and the MER definition to decide whether to include or exclude certain motion vectors. For example, when MER is enabled and IBC is disabled for the current CU, the prediction candidate list identification module 3010 may exclude motion vectors from neighboring blocks of the current CU in the same MER as the current CU, and if IBC mode is disabled for the current CU, the prediction candidate list identification module 3010 may include motion vectors from neighboring blocks of the current CU in the same MER as the current CU, i.e., may include spatial neighboring blocks of the current CU regardless of the MER definition.

[0253] The prediction candidate list identified by the prediction candidate list identification module 2710 for the current CU is in turn provided to the prediction generation module 3020, which generates a prediction for the current CU according to one of the possible prediction modes (CIIP / MMVD / GPM affine, etc.) for the motion compensation module 2930.

[0254] Figure 31The process 3100 for decoding a block in a video image using prediction candidates based on MER and IBC pattern recognition is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing the decoder 2900 perform the process 3100 by executing instructions stored on a computer-readable medium. In some embodiments, the electronic device implementing the decoder 2900 performs the process 3100.

[0255] The decoder receives (at block 3110) data from a bitstream for a current block of a current image to be decoded. The decoder determines (at block 3120) whether to encode or decode the current block using a prediction candidate (e.g., a merge mode candidate list or an AMVP candidate list). In some embodiments, the decoder parses a syntax element from the bitstream indicating whether to encode or decode the current block using a prediction candidate. If the current block is to be encoded or decoded using a prediction candidate, the process proceeds to block 3130. Otherwise, the decoder reconstructs (at block 3125) the current block without using the prediction candidate.

[0256] At block 3130, the decoder identifies potential candidates to add to the prediction candidate list for the current block. The decoder determines (at block 3140) whether the current block is coded by intra block copy mode and (at block 3142) whether the prediction candidate list belongs to or is derived from a predetermined subset of different candidate lists. In some embodiments, the predetermined subset of different candidate lists includes candidates for merge mode or quasi-merge (e.g., CIIP, GPM, regular merge) and excludes AMVP candidates. If the current block is coded by intra block copy mode or if the prediction candidate list is a list of AMVP, the process proceeds to 2850. Otherwise (the current block is not coded by IBC and the list of prediction candidates is merge or quasi-merge), the process proceeds to 3145.

[0257] When the current block is encoded and decoded using intra block copy mode, a motion vector that references decoded reference samples in the current image is used to generate a prediction for the current block. When the current block is not encoded and decoded using intra block copy mode, the current block and at least one other block in the MER may be decoded in parallel. In some embodiments, the decoder receives one or more syntax elements in the bitstream that indicate whether intra block copy mode is used for the current block.

[0258] The decoder determines (at block 3145) whether the candidate is from a neighboring block that is in the same MER as the current block. In some embodiments, one or more syntax elements in the bitstream may be used to identify or define the MER that includes the current block. If the spatially neighboring block is in the same MER as the current block, the process proceeds to block 3155. Otherwise, the process proceeds to 3150.

[0259] At block 3150, the decoder includes the candidate in the prediction candidate list for the current block. If there are additional potential candidates included in the prediction candidate list, the process may return to block 3130. If there are no additional potential candidates, the process proceeds to 3160.

[0260] At block 3155, the decoder excludes candidates from the prediction candidate list for the current block. In some embodiments, when the current block is not encoded using intra block copy mode, all neighboring blocks of the current block located in MER are excluded from the list of prediction mode candidates. If there are additional potential candidates included in the prediction candidate list, the process may return to block 3130. If there are no more potential candidates, the process proceeds to 3160.

[0261] The decoder reconstructs (at block 3160) the current block by generating a prediction for the current block using a merge candidate selected from the prediction candidate list. The prediction for the current block may be an affine prediction generated based on the selected prediction candidate. The prediction for the current block may be a triangle or GPM and guess generated based on the selected prediction candidate. The prediction for the current block may be a combined inter and intra prediction (CIIP) generated based on the selected prediction candidate. The prediction for the current block may be generated by refining the selected prediction candidate with motion vector difference information.

[0262] Exemplary Electronic Systems

[0263] Many of the features and applications described above are implemented as software processes specified as a set of instructions recorded on a computer-readable storage medium (also referred to as computer-readable media). When these instructions are executed by one or more computing or processing units (e.g., one or more processors, processor cores, or other processing units), they cause the processing units to perform the actions indicated by the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, random access memory (RAM) chips, hard drives, erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), etc. Computer-readable media does not include carrier waves and electronic signals transmitted wirelessly or via wired connections.

[0264] In the present invention, the term "software" is intended to include firmware residing in read-only memory or applications stored in magnetic storage that can be read into memory and processed by a processor. Additionally, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while retaining unique software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Ultimately, any combination of the separate programs implemented together as the software inventions described herein is within the scope of the present invention. In some embodiments, the software program, when installed to operate on one or more electronic systems, defines one or more specific machine implementations for running or executing the operations of the software program.

[0265] Figure 32 The electronic system 3200 is conceptually illustrated, in which some embodiments of the present invention may be implemented. The electronic system 3200 may be a computer (e.g., a desktop computer, a personal computer, a tablet), a phone, a PDA, or any other suitable electronic device. Such an electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 3200 includes a bus 3205, a processing unit 3210, a graphics processing unit (GPU) 3215, system memory 3220, a network 3225, read-only memory 3230, a persistent storage device 3235, an input device 3240, and an output device 3245.

[0266] Buses 3205 collectively represent all system, external, and chipset buses that connect the various internal devices of electronic system 3200. For example, bus 3205 connects processing unit 3210 to GPU 3215, read-only memory 3230, system memory 3220, and permanent storage 3235.

[0267] From various memory units, processing unit 3210 retrieves instructions to execute the data to be processed to perform the processes of the present invention. The processing unit can be a single processor or a multi-core processor in different embodiments. Some instructions are transmitted and executed by GPU 3215. GPU 3215 can offload various calculations or perform image processing provided by processing unit 3210.

[0268] Read-only memory (ROM) 3230 stores static data and instructions used by processing unit 3210 and other modules of the electronic system. Persistent storage 3235, on the other hand, is a read-write memory device. This device is a non-volatile storage unit that stores instructions even when electronic system 3200 is turned off. Some embodiments of the present invention use a mass storage device (such as a disk or optical disc and its corresponding drive) as permanent storage 3235.

[0269] Other embodiments use removable storage devices (such as floppy disks, flash storage devices, etc. and their corresponding drives) as permanent storage devices. Like permanent storage device 3235, system memory 3220 is a read-write storage device. However, unlike storage device 3235, system memory 3220 is a volatile read-write memory, such as random access memory. System memory 3220 stores some instructions and data used by the processor during operation. In some embodiments, processes according to the present invention are stored in system memory 3220, permanent storage device 3235 and / or read-only memory 3230. For example, various storage units include instructions for processing multimedia clips according to some embodiments. From these various storage units, processing unit 3210 retrieves instructions to be executed and data to be processed to perform the processes of some embodiments.

[0270] The bus 3295 also connects input 3240 and output devices 3245. The input devices 3240 enable a user to communicate information and select commands to the electronic system. The input devices 3240 include an alphanumeric keyboard and a pointing device (also known as a "cursor control device"), a camera (e.g., a webcam), a microphone, or a similar device for receiving voice commands, etc. The output devices 3245 display images generated by the electronic system or other input data. The output devices 3245 include a printer and a display device, such as a cathode ray tube (CRT) or a liquid crystal display (LCD), and a speaker or similar sound output device. Some embodiments include a touch screen that serves as both an input and an output device.

[0271] Finally, if Figure 32 As shown, bus 3205 also couples electronic system 3200 to a network 3225 via a network interface card (not shown). Thus, the computer can be part of a computer network, such as a local area network (LAN), a wide area network (WAN), or an intranet, or a network of networks (e.g., the Internet). Any or all components of electronic system 3200 may be used in conjunction with the present invention.

[0272] Some embodiments include electronic components such as microprocessors, storage, and memory that store computer program instructions on machine-readable or computer-readable media (or computer-readable storage media, and readable media thereof, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, compact disc read-only memory (CD-ROM), compact disc recordable memory (CD-R), compact disc writeable memory (CD-RW), read-only universal optical discs (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / writable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, miniSD card, microSD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable Blu-ray discs, ultra-high-density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media can store a computer program executed by at least one processing unit and include a set of instructions for performing various operations. Examples of computer programs or computer code include machine code (such as produced by a compiler) and files including higher-level code, which are executed by a computer, electronic device, or microprocessor using an interpreter.

[0273] While the above discussion primarily refers to microprocessors or multi-core processors executing software, many of the features and applications described above are implemented by one or more integrated circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuits themselves. Additionally, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM.

[0274] As used in this specification and any claims herein, the terms "computer," "server," "processor," and "memory" refer to electronic or other technological devices. These terms do not include people or groups of people. For the purposes of this specification, the terms "display" or "display device" refer to displays on electronic devices. As used in this specification and any claims herein, the terms "computer-readable medium," "computer-readable medium," and "machine-readable medium" are strictly limited to tangible, physical objects that store information in a computer-readable form. These terms exclude any wireless signals, wired download signals, and any other ephemeral signals.

[0275] Although the present invention has been described with reference to many specific details, it will be appreciated by those skilled in the art that the present invention may be implemented in other specific forms without departing from the spirit of the present invention. Figure 28 as well as Figure 31) conceptually illustrate the processes. The specific operations of these processes may not be performed in the specific order shown and described. Certain operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. In addition, the processes may be implemented using several sub-processes or as part of a larger macro-process. Therefore, it will be understood by those skilled in the art that the present invention is not limited by the foregoing illustrative details, but is defined by the appended claims.

[0276] Notes

[0277] The subject matter described herein sometimes shows different elements included in or connected to different other elements. It is understood that the architecture of this description is only an example, and in fact many other architectures that can be implemented to achieve the same function. Conceptually, any arrangement of elements that achieve the same function is effectively "associated" so as to achieve the desired function. Therefore, any two elements that are combined herein to achieve a specific function can be considered to be "associated" to each other so as to achieve the desired function, regardless of the architecture or intermediate elements. Similarly, any two elements that are so associated can also be considered to be "operably connected" or "operably coupled" to each other to achieve the desired function, and any two elements that can be so associated can also be considered to be "operably coupled" to each other to achieve the desired function. The specific examples of operably coupled include but are not limited to physically matchable and / or physically interactive elements and / or wirelessly understandable and / or wirelessly interactive elements and / or logically interactive and / or logically interactive elements.

[0278] In addition, with respect to the use of substantially any plural and / or singular terms herein, those skilled in the art can appropriately convert from the plural to the singular and / or from the singular to the plural, depending on the context and application. For clarity, various singular / plural permutations are expressly set forth herein.

[0279] In addition, those skilled in the art will understand that, generally, the terms used herein, and especially the terms used in the appended claims (e.g., the bodies of the appended claims), are generally intended to be "open-ended" terms, e.g., the term "including" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," the term "includes" should be interpreted as "including but not limited to," etc. Those skilled in the art will further understand that if a specific number of claim recitations is intended, such intent will be explicitly recited in the claim, and in the absence of such recitation, such intent is absent. For example, to aid understanding, the subsequent appended claims may include the use of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be construed as implying that a claim recitation introduced by the indefinite article "a" or "an" limits any particular claim containing such introduced claim recitation to embodiments containing only one such recitation. Even when the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an," "a" and / or "an" should be construed to mean "at least one" or "one or more," and the same applies to the definite article introducing the claim recitation. Furthermore, even if a specific number of introduced claim recitations is explicitly recited, those skilled in the art will recognize that such recitation should be construed to mean at least one of the recited number, such as the bare statement "two recitations" without further modification means at least two recitations, or two or more recitations. Furthermore, where a convention similar to "at least one of A, B, and C, etc." is used, such construction is generally intended to be understood by those skilled in the art, such that "a system having at least one of A, B, and C" would include, but is not limited to, a system having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc. In those instances where a convention similar to "at least one of A, B, or C" is used, generally such construction is intended to be understood by those skilled in the art, such that "a system has at least one of A, B, or C" would include, but is not limited to, a system having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc. Those skilled in the art will further understand that, in fact, any separator and / or phrase in the description, claims, or figures indicating two or more alternative terms will be understood to include the possibility of one of the terms, either of the terms, or both of the terms. For example, the phrase "A or B" will be understood to include the possibility of "A or B" or "A and B."

[0280] It will be appreciated from the foregoing that various embodiments of the present invention have been described herein for illustrative purposes, and that various modifications may be made without departing from the scope and spirit of the invention. Therefore, the various embodiments described herein are not intended to be limiting, with the true scope and spirit being indicated by the subsequent claims.

Claims

1. A video decoding method, the method comprising: receiving data from a bitstream to be decoded as a current block of a current picture of a video; Determining whether to encode or decode the current block by using an intra block copy mode; identifying a list of one or more prediction candidates for the current block, wherein when the current block is not encoded or decoded by using an intra block copy mode and the list of prediction candidates belongs to a predetermined subset of a plurality of different candidate lists, one or more spatial neighboring blocks of the current block located in the same merged estimation area as the current block are excluded from the list of prediction candidates; wherein when the current block is encoded and decoded by using an intra block copy mode, at least one identified merge mode candidate is a spatial neighboring block of the current block located in the merge estimation region; as well as A prediction of the current block is generated by reconstructing the current block using a merge mode candidate selected from the list of one or more prediction candidates.

2. The video decoding method according to claim 1, wherein: When the current block is not encoded and decoded by using the intra block copy mode, the current block and at least one other block in the merge estimation area are reconstructed in parallel.

3. The video decoding method according to claim 1, wherein: When the current block is encoded and decoded using the intra block copy mode, a motion vector of a previously reconstructed reference sample in the current picture is used to generate a prediction for the current block.

4. The video decoding method according to claim 1, wherein: One or more syntax elements in the bitstream indicate whether intra block copy mode is used to encode or decode the current block.

5. The video decoding method according to claim 1, wherein: The method further includes parsing one or more syntax elements in the bitstream and determining whether an intra block copy mode is used for the current block based on the parsed syntax elements.

6. The video decoding method according to claim 1, wherein: When the current block is not encoded by using the intra block copy mode, all spatial neighboring blocks of the current block located in the merge estimation region are excluded from the list of prediction candidates.

7. The video decoding method according to claim 1, wherein: The prediction of the current block is an affine prediction generated based on the selected prediction candidate.

8. The video decoding method according to claim 1, wherein: The prediction of the current block is a geometric partitioning mode prediction generated based on the selected prediction candidate.

9. The video decoding method according to claim 1, wherein: The prediction of the current block is a combined inter-frame and intra-frame prediction generated based on the selected prediction candidate.

10. The video decoding method according to claim 1, wherein: A prediction of the current block is generated by refining the selected prediction candidate using a motion vector difference information.

11. The video decoding method according to claim 1, wherein: The prediction of the current block is improved by adapting to the motion field in a parity picture at a sub-block level.

12. The video decoding method according to claim 1, wherein: The predetermined subset of the plurality of different candidate lists includes candidates for a class merge prediction mode.

13. The video decoding method according to claim 1, wherein: The predetermined subset of the plurality of different candidate lists excludes candidates for advanced motion vector prediction.

14. A video encoding method, the method comprising: receiving raw pixel data of a bitstream to be encoded as a current block of a current image of a video; Determine whether the current block is encoded or decoded by using the intra block copy mode; identifying a list of one or more prediction candidates for the current block; wherein when the current block is not encoded and decoded by using the intra block copy mode and the list of prediction candidates belongs to a predetermined subset of a plurality of different candidate lists, one or more neighboring blocks of the current block located in the same merged estimation area as the current block are excluded from the prediction candidate list, wherein when the current block is not encoded by using an intra block copy mode, at least one identified prediction candidate is a spatial neighboring block of the current block located in the merged estimation region; as well as A prediction of the current block is generated by encoding the current block using a prediction candidate selected from the list of one or more prediction candidates.

15. The video encoding method according to claim 14, wherein: Further included signaling one or more syntax elements in the bitstream to indicate whether intra block copy mode is used for the current block.

16. An electronic device, comprising: The video decoder circuit is configured to perform operations including: receiving data from a bitstream to be decoded as a current block of a current picture of a video; Determining whether to encode or decode the current block by using an intra block copy mode; identifying a list of one or more prediction candidates for the current block, wherein when the current block is not encoded and decoded by using an intra block copy mode and the list of prediction candidates belongs to a predetermined subset of a plurality of different candidate lists, one or more spatial neighboring blocks of the current block located in the same merged estimation area as the current block are excluded from the list of prediction candidates; wherein when the current block is encoded and decoded by using an intra block copy mode, at least one identified merge mode candidate is a spatial neighboring block of the current block located in the merge estimation region; as well as A prediction of the current block is generated by reconstructing the current block using a merge mode candidate selected from the list of one or more prediction candidates.

Images