Improvements to the affine prediction model

Abstract

Improvements to affine prediction modes are described. In one example, a method for video processing is disclosed. The method includes: using motion information associated with a first reference picture from a single reference picture list, deriving motion information associated with a second reference picture from another reference picture list different from the single reference picture list for a current block; and performing video processing on the current block based on the motion information associated with the first reference picture and the motion information associated with the second reference picture.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] In accordance with the applicable patent laws and / or the rules of the Paris Convention, this application promptly claims priority and benefits to International Patent Application PCT / CN2018 / 115354 filed on November 14, 2018. The entire disclosure of the above application is incorporated by reference as part of the disclosure of this patent. Technical Field

[0003] This patent document relates to image and video encoding and decoding. Background Art

[0004] Digital video accounts for the largest use of bandwidth on the Internet and other digital communications networks. As the number of networked user devices capable of receiving and displaying video increases, bandwidth demand for digital video usage is expected to continue to grow. Summary of the invention

[0005] The disclosed techniques may be used by video decoder or encoder embodiments during video decoding or encoding using affine motion prediction or compensation tools.

[0006] In one example aspect, a method for video processing is disclosed. The method includes: using motion information associated with a first reference picture from a single reference picture list, deriving, for a current block, motion information associated with a second reference picture from another reference picture list different from the single reference picture list; and performing video processing on the current block based on the motion information associated with the first reference picture and the motion information associated with the second reference picture.

[0007] In another example aspect, a method for video processing is disclosed. The method includes: determining an affine mode type associated with a current block; and performing video processing on the current block based on the determined affine mode type; wherein the affine mode type is restricted to a portion of a video picture in which the current block is located, and the determined affine mode type is prohibited from being used in other portions of the video picture.

[0008] In another example aspect, a video processing method is disclosed. The method includes: generating an affine candidate list for a current block, wherein the affine candidate list includes an additional affine candidate generated by averaging a plurality of affine candidates from the affine candidate list; and performing video processing on the current block based on the expanded affine candidate list.

[0009] In another example aspect, a video processing device is disclosed. The video processing device includes a processor configured to implement the method described herein.

[0010] In another exemplary aspect, a computer program product stored on a non-transitory computer readable medium is disclosed. The computer program product includes program code for executing the method described herein.

[0011] In yet another example aspect, a video encoder apparatus is disclosed. The video encoder apparatus includes a processor configured to implement the method described herein.

[0012] In yet another example aspect, a video decoder apparatus is disclosed. The video decoder apparatus includes a processor configured to implement the method described herein.

[0013] In yet another aspect, a computer readable medium having code stored thereon is disclosed. When executed by a processor, the code causes the processor to implement the method described herein.

[0014] These and other aspects are described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is an example of the derivation process used for Merge candidate list construction.

[0016] Figure 2 Example locations of spatial merge candidates are shown.

[0017] Figure 3 An example of candidate pairs considered for redundancy checking of spatial merge candidates is shown.

[0018] Figure 4A-4B Example locations of the second PU (Prediction Unit) for N×2N and 2N×N partitions are shown.

[0019] Figure 5 FIG. 4 is an example diagram of motion vector scaling of a temporal merge candidate.

[0020] Figure 6 Example candidate positions of time-domain Merge candidates C0 and C1 are shown.

[0021] Figure 7 An example of a combined bi-predictive Merge candidate is shown.

[0022] Figure 8 An example derivation process of motion vector prediction candidates is shown.

[0023] Fig. 9 is an example illustration of motion vector scaling of spatial motion vector candidates.

[0024] Fig.10An example of Alternative Temporal Motion Vector Prediction (ATMVP) motion prediction for a CU (Coding Unit) is shown.

[0025] Fig.11 An example of one CU having four sub-blocks (AD) and their neighboring blocks (ad) is shown.

[0026] Fig.12 This is a flowchart showing an example of encoding with different MV (Motion Vector) precisions.

[0027] Figure 13A-13B A 135 degree partition type (partitioning from the upper left corner to the lower right corner) and a 45 degree partition mode are shown. Illustration of the partitioning of a CU into two triangular prediction units (two partition modes).

[0028] Fig.14 An example of the positions of neighboring blocks is shown.

[0029] Fig.15 Examples of blocks on the upper side and blocks on the left side are shown.

[0030] Figure 16A-16B Examples of 2 control point motion vectors (CPMVs) and 3 CPMVs are shown.

[0031] Fig.17 Two examples of CPMV are shown.

[0032] Figure 18A-18B 4-parameter and 6-parameter affine model examples are shown.

[0033] Fig.19 The MVP (Motion Vector Predictor) of AF_INTER of the inherited affine candidate is shown.

[0034] Fig. 20 An example of constructing an affine motion predictor in AF_INTER is shown.

[0035] Figure 21A-21B An example of control point motion vectors when affine encoded under AF_MERGE is shown.

[0036] Fig. 22 An example of candidate positions of the affine merge mode is shown.

[0037] Fig.23 An example of an intra picture block copy operation is shown.

[0038] Fig.24 Shows candidate positions for the affine merge mode.

[0039] Fig.25 The modified Merge list building process is shown.

[0040] Fig.26 is a block diagram of an example of a video processing device.

[0041] Fig. 27 is a flow chart of an example of a video processing method.

[0042] Fig.28 is a flowchart of another example of a video processing method.

[0043] Fig.29 is a flowchart of another example of a video processing method. DETAILED DESCRIPTION

[0044] This document provides various techniques that can be used by decoders of video bitstreams to improve the quality of decompressed or decoded digital video. In addition, video encoders can also implement these techniques during the encoding process to reconstruct decoded frames for further encoding.

[0045] For ease of understanding, section headings are used in this document, and embodiments and techniques are not limited to the corresponding sections. Thus, embodiments from one section can be combined with embodiments from other sections.

[0046] 1. Overview

[0047] This document relates to video coding techniques. Specifically, it relates to affine prediction modes in video coding. It can be applied to existing video coding standards, such as HEVC, or standards that are about to be finalized (e.g., Versatile Video Coding). It can also be applied to future video coding standards or video codecs.

[0048] In this document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during the conversion from a pixel representation of a video to a corresponding bitstream representation, or vice versa.

[0049] 2. Introductory Notes

[0050] Video coding standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, ISO / IEC developed MPEG-1 and MPEG-4 Visual, and the two organizations jointly developed the H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Coding (AVC) and H.265 / HEVC standards. Since H.262, video coding standards have been based on a hybrid video coding structure, where temporal prediction plus transform coding is used. In order to explore future video coding technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and put them into a reference software called the Joint Exploration Model (JEM). In April 2018, the Joint Video Experts Group (JVET) between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) was established to work on the Versatile Video Coding (VVC) standard with the goal of a 50% bitrate reduction compared to HEVC.

[0051] 2.1 Inter-frame prediction of HEVC / H.265

[0052] Each inter-predicted PU has motion parameters for one or two reference picture lists. The motion parameters include motion vectors and reference picture indices. The use of one of the two reference picture lists may also be signaled using inter_pred_idc. The motion vector may be explicitly coded as a delta relative to the predictor.

[0053] When a CU is encoded in skip mode, one PU is associated with the CU and there are no significant residual coefficients, encoded motion vector increments, or reference picture indices. A Merge mode is specified, whereby the motion parameters of the current PU are obtained from neighboring PUs including spatial and temporal candidates. Merge mode can be applied to any inter-predicted PU, not just for skip mode. An alternative to Merge mode is explicit transmission of motion parameters, where each PU explicitly signals the motion vector (more accurately, the motion vector difference (MVD) compared to the motion vector predictor), the corresponding reference picture index for each reference picture list, and the use of the reference picture list. In the present disclosure, this mode is referred to as Advanced Motion Vector Prediction (AMVP).

[0054] When signaling indicates that one of the two reference picture lists is to be used, a PU is generated from a block of samples. This is called "unidirectional prediction". Unidirectional prediction is available for P slices and B slices.

[0055] When signaling indicates that both reference picture lists are to be used, a PU is generated from two blocks of samples. This is called "bi-prediction". Bi-prediction is only available for B slices.

[0056] The following text provides detailed information about the inter prediction modes specified in HEVC. The description will start with the Merge mode.

[0057] 2.1.1 Reference Image List

[0058] In HEVC, the term inter prediction is used to refer to predictions derived from data elements (e.g., sample values ​​or motion vectors) of reference pictures other than the currently decoded picture. As in H.264 / AVC, a picture can be predicted from multiple reference pictures. Reference pictures used for inter prediction are organized in one or more reference picture lists. A reference index identifies which reference picture in the list should be used to create the prediction signal.

[0059] A single reference picture list (list 0) is used for P slices, and two reference picture lists (list 0 and list 1) are used for B slices. It should be noted that the reference pictures included in lists 0 / 1 can be pictures from both the past and the future in terms of capture / display order.

[0060] 2.1.2 Merge mode

[0061] 2.1.2.1 Merge Mode Candidate Derivation

[0062] When predicting a PU using Merge mode, an index pointing to an entry in the Merge candidate list is parsed from the bitstream and used to retrieve motion information. The construction of this list is specified in the HEVC standard and can be summarized in the following sequence of steps:

[0063] Step 1: Initial candidate derivation

[0064] Step 1.1: Spatial Candidate Derivation

[0065] Step 1.2: Redundancy check of spatial candidates

[0066] οStep 1.3: Time Domain Candidate Derivation

[0067] Step 2: Additional candidate insertions

[0068] οStep 2.1: Create bidirectional prediction candidates

[0069] οStep 2.2: Insert zero motion candidates

[0070] These steps are also schematically depicted in Figure 1 For spatial Merge candidate derivation, a maximum of four Merge candidates are selected from candidates located at five different positions. For temporal Merge candidate derivation, a maximum of one Merge candidate is selected from two candidates. Since a constant number of candidates for each PU is assumed at the decoder, additional candidates are generated when the number of candidates obtained from step 1 does not reach the maximum number of Merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, the index of the best Merge candidate is encoded using Truncated Unarybinarization (TU). If the size of the CU is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

[0071] Hereinafter, operations associated with the above-mentioned steps will be described in detail.

[0072] 2.1.2.2 Spatial Candidate Derivation

[0073] In the derivation of spatial merge candidates, Figure 2 Up to four Merge candidates are selected from the candidates at the positions depicted in . The order of derivation is A1, B1, B0, A0, and B2. Position B2 is considered only when any PU at position A1, B1, B0, A0 is not available (for example, because it belongs to another slice or piece) or is intra-coded. After adding the candidate at position A1, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only pairs with Figure 3 The arrows in , and candidates are added to the list only if the corresponding candidates for redundancy checking do not have the same motion information. Another source of duplicate motion information is a "second PU" associated with a partition other than 2N×2N. As an example, Figure 4A and Figure 4B The second PU is depicted for the N×2N and 2N×N cases, respectively. When the current PU is partitioned into N×2N, the candidate at position A1 is not considered for list construction. In fact, adding this candidate would result in two prediction units having the same motion information, which is redundant for having only one PU in the coding unit. Similarly, when the current PU is partitioned into 2N×N, position B1 is not considered.

[0074] 2.1.2.3 Time Domain Candidate Derivation

[0075] In this step, only one candidate is added to the list. Specifically, in the derivation of the temporal Merge candidate, the scaled motion vector is derived based on the co-located PU, which belongs to the picture with the smallest POC (Picture Order Count) difference with the current picture in a given reference picture list. The reference picture list to be used for derivation of the co-located PU is explicitly signaled in the slice header. Figure 5 As shown by the dotted line in , a scaled motion vector for the temporal Merge candidate is obtained, which is scaled from the motion vector of the co-located PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal Merge candidate is set equal to zero. The actual implementation of the scaling process is described in the HEVC specification. For B slices, two motion vectors (one for reference picture list 0 and the other for reference picture list 1) are obtained and combined to generate a bidirectional prediction Merge candidate.

[0076] In the co-located PU (Y) belonging to the reference frame, the position of the temporal candidate is selected between candidates C0 and C1, such as Figure 6 If the PU at position C0 is not available, is intra-coded, or is outside the current coding tree unit (CTU, also known as LCU) row, position C1 is used. Otherwise, position C0 is used in the derivation of the time domain Merge candidate.

[0077] 2.1.2.4 Additional candidate insertion

[0078] In addition to the spatial and temporal Merge candidates, there are two additional types of Merge candidates: combined bidirectional prediction Merge candidates and zero Merge candidates. The combined bidirectional prediction Merge candidate is generated by utilizing the spatial and temporal Merge candidates. The combined bidirectional prediction Merge candidate is only used for B slices. The combined bidirectional prediction candidate is generated by combining the first reference picture list motion parameters of an initial candidate with the second reference picture list motion parameters of another initial candidate. If the two tuples provide different motion hypotheses, they will form a new bidirectional prediction candidate. As an example, Figure 7It is shown when two candidates in the original list (on the left) (which have mvL0 and refIdxL0 or mvL1 and refIdxL1) are used to create a combined bi-predictive Merge candidate that is added to the final list (on the right). There are many rules about combining that are considered to generate these additional Merge candidates.

[0079] Zero motion candidates are inserted to fill the remaining entries in the Merge candidate list and thus reach the MaxNumMergeCand capacity. These candidates have zero spatial displacement and a reference picture index that starts at zero and increases each time a new zero motion candidate is added to the list. The number of reference frames used by these candidates is 1 and 2 for unidirectional prediction and bidirectional prediction, respectively. Finally, no redundancy check is performed on these candidates.

[0080] 2.1.3 AMVP

[0081] AMVP exploits the spatiotemporal correlation of motion vectors with neighboring PUs for explicit transmission of motion parameters. For each reference picture list, a motion vector candidate list is constructed by first checking the availability of neighboring PU positions in the left and upper temporal domains, removing redundant candidates and adding zero vectors to make the candidate list a constant length. The encoder can then select the best predictor from the candidate list and send a corresponding index indicating the selected candidate. Similar to Merge index signaling, the index of the best motion vector candidate is encoded using truncated unary. In this case, the maximum value to be encoded is 2 (see Figure 8 ). In the following sections, details about the derivation process of motion vector prediction candidates will be provided.

[0082] 2.1.3.1 Derivation of AMVP Candidates

[0083] Figure 8 The process of deriving motion vector prediction candidates is outlined.

[0084] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For spatial motion vector candidate derivation, the final result is based on the location of Figure 2 The motion vectors of each PU are depicted at five different positions to derive two motion vector candidates.

[0085] For temporal motion vector candidate derivation, one motion vector candidate is selected from two candidates derived based on two different co-located positions. After generating the first spatio-temporal candidate list, duplicate motion vector candidates in the list are removed. If the number of potential candidates is greater than two, motion vector candidates whose reference picture index within the associated reference picture list is greater than 1 are removed from the list. If the number of spatio-temporal motion vector candidates is less than two, an additional zero motion vector candidate is added to the list.

[0086] 2.1.3.2 Spatial Motion Vector Candidates

[0087] In the derivation of spatial motion vector candidates, the Figure 2 Consider up to two candidates out of the five potential candidates derived in the PU at the depicted position, and those positions are the same as the positions of the motion Merge. The derivation order on the left side of the current PU is defined as A0, A1, and scaled A0, scaled A1. The derivation order on the upper side of the current PU is defined as B0, B1, B2, scaled B0, scaled B1, scaled B2. Therefore, for each side, there are four cases that can be used as motion vector candidates, two of which do not require the use of spatial domain scaling, and two use spatial domain scaling. The four different cases are summarized as follows:

[0088] No airspace scaling

[0089] -(1) Same reference picture list, and same reference picture index (same POC) -(2) Different reference picture list, but same reference picture (same POC)

[0090] Airspace scaling

[0091] -(3) Same reference picture list, but different reference pictures (different POC)

[0092] -(4) Different reference picture lists, and different reference pictures (different POC)

[0093] First check the non-spatial scaling case, then the spatial scaling. When the POC is different between the reference picture of the neighboring PU and the reference picture of the current PU, consider spatial scaling regardless of the reference picture list. If all PUs of the left candidate are unavailable or are intra-coded, then scaling for the upper motion vector is allowed to help parallel derivation of the left and upper MV candidates. Otherwise, spatial scaling is not allowed for the upper motion vector.

[0094] like Fig. 9As depicted, in the spatial scaling process, the motion vectors of neighboring PUs are scaled in a similar way to temporal scaling. The main difference is that the reference picture list and the index of the current PU are given as input; the actual scaling process is the same as that of temporal scaling.

[0095] 2.1.3.3 Temporal Motion Vector Candidates

[0096] Except for the reference picture index derivation, all the processes used to derive the temporal merge candidate are the same as those used to derive the spatial motion vector candidate (see Figure 6 ). The reference picture index is signaled to the decoder.

[0097] 2.2 Sub-CU-based motion vector prediction method in JEM

[0098] In JEM with QTBT (QuadTrees plus Binary Trees), each CU can have at most one set of motion parameters for each prediction direction. Two sub-CU level motion vector prediction methods are considered in the encoder by dividing the large CU into sub-CUs and deriving the motion information of all sub-CUs of the large CU. The optional temporal motion vector prediction (ATMVP) method allows each CU to extract multiple sets of motion information from multiple blocks in the co-located reference picture that are smaller than the current CU. In the Spatial-Temporal Motion Vector Prediction (STMVP) method, the motion vector of the sub-CU is recursively derived by using the temporal motion vector predictor and the spatial neighboring motion vector.

[0099] In order to maintain a more accurate motion field for sub-CU motion prediction, motion compression of reference frames is currently disabled.

[0100] 2.2.1 Optional temporal motion vector prediction

[0101] In the optional temporal motion vector prediction (ATMVP) method, the motion vector temporal motion vector prediction (TMVP) is modified by extracting multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU. As an example, a sub-CU is a square N×N block (N is set to 4 by default). Fig.10 An example of ATMVP motion prediction for a CU is shown.

[0102] ATMVP predicts the motion vector of a sub-CU within a CU in two steps. The first step is to identify the corresponding block in the reference picture with the so-called time domain vector. The reference picture is also called the motion source picture. The second step is to divide the current CU into sub-CUs and obtain the motion vector and reference index of each sub-CU from the block corresponding to each sub-CU, as an example.

[0103] In the first step, the reference picture and the corresponding block are determined by the motion information of the spatial neighboring blocks of the current CU. In order to avoid the repeated scanning process of the neighboring blocks, the first Merge candidate in the Merge candidate list of the current CU is used. The first available motion vector and its associated reference index are set to the index of the temporal vector and the motion source picture. In this way, in ATMVP, the corresponding block can be identified more accurately than in TMVP, where the corresponding block (sometimes called the co-located block) is always in the lower right or center position relative to the current CU.

[0104] In the second step, the corresponding block of the sub-CU is identified by the time domain vector in the motion source picture by adding the time domain vector to the coordinates of the current CU. For each sub-CU, the motion information of its corresponding block (the minimum motion grid covering the center sample point) is used to derive the motion information of the sub-CU. After identifying the motion information of the corresponding N×N block, it is converted into the motion vector and reference index of the current sub-CU in the same way as the TMVP of HEVC, where motion scaling and other processes apply. For example, the decoder checks whether the low latency condition is met (that is, the POC of all reference pictures of the current picture is less than the POC of the current picture), and may use the motion vector MV x (motion vector corresponding to reference picture list X) to predict the motion vector MV of each sub-CU y (where X equals 0 or 1, and Y equals 1-X).

[0105] 2.2.2 Spatiotemporal Motion Vector Prediction (STMVP)

[0106] In this method, the motion vector of a sub-CU is recursively derived in raster scan order. Fig.11 This concept is illustrated in Figure 1. Consider an 8x8 CU, which contains four 4x4 sub-CUs: A, B, C, and D. The neighboring 4x4 blocks in the current frame are labeled a, b, c, and d.

[0107] The motion derivation of sub-CU A starts by identifying its two spatial neighbors. The first neighbor is the N×N block (block c) on the upper side of the sub-CUA. If this block c is not available or is intra-coded, check the other N×N blocks on the upper side of the sub-CU A (from left to right, starting with block c). The second neighbor is the block on the left side of the sub-CU A (block b). If block b is not available or is intra-coded, check the other blocks on the left side of the sub-CU A (from top to bottom, starting with block b). The motion information obtained from the adjacent blocks in each list is scaled to the first reference frame of the given list. Next, the temporal motion vector predictor (TMVP) of sub-block A is derived by following the same process of TMVP derivation as specified by HEVC. The motion information of the co-located blocks at position D is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors (up to 3) are averaged separately for each reference list. The averaged motion vector is assigned as the motion vector of the current sub-CU.

[0108] 2.2.3 Sub-CU Motion Prediction Mode Signaling

[0109] Sub-CU modes are enabled as additional Merge candidates, and no additional syntax elements are required to signal these modes. Two additional Merge candidates are added to the Merge candidate list of each CU to represent ATMVP mode and STMVP mode. If the sequence parameter set indicates that ATMVP and STMVP are enabled, up to seven Merge candidates can be used. The encoding logic of the additional Merge candidates is the same as that of the Merge candidates in HM, which means that for each CU in a P slice or B slice, two RD checks are required for the two additional Merge candidates.

[0110] In JEM, all bins of the Merge index are context coded by CABAC, whereas in HEVC, only the first bin is context coded and the remaining bins are context bypass coded.

[0111] 2.3 Inter-frame prediction method in VVC

[0112] There are several new coding tools for inter-frame prediction improvements, such as Adaptive motion vector difference resolution (AMVR) for signaling MVD, affine prediction mode, Triangular prediction mode (TPM), ATMVP, Generalized Bi-Prediction (GBI), and Bi-directional Optical flow (BIO).

[0113] 2.3.1 Adaptive Motion Vector Difference Resolution

[0114] In HEVC, when use_integer_mv_flag in the slice header is equal to 0, the motion vector difference (MVD) (between the PU's motion vector and the predicted motion vector) is signaled in units of quarter luma samples. In VVC, Locally Adaptive Motion Vector Resolution (LAMVR) is introduced. In VVC, MVD can be encoded in units of quarter luma samples, integer luma samples, or four luma samples (i.e., 1 / 4 pixel, 1 pixel, 4 pixels). The MVD resolution is controlled at the coding unit (CU) level, and the MVD resolution flag is conditionally signaled for each CU with at least one non-zero MVD component.

[0115] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter luma sample MV precision is not used, another flag is signaled to indicate whether integer luma sample MV precision or quad luma sample MV precision is used.

[0116] When the first MVD resolution flag of a CU is zero or is not encoded for the CU (meaning that all MVDs in the CU are zero), a quarter luma sample MV resolution is used for the CU. When the CU uses integer luma sample MV precision or four luma sample MV precision, the MVP in the AMVP candidate list of the CU is rounded to the corresponding precision.

[0117] In the encoder, the CU-level RD check is used to determine which MVD resolution to use for the CU. That is, for each MVD resolution, three CU-level RD checks are performed. In order to speed up the encoder, the following encoding scheme is applied in JEM:

[0118] · During the RD check of a CU with normal quarter-luminance sample MVD resolution, store the motion information of the current CU (integer luminance sample accuracy). The stored motion information (after rounding) is used as the starting point for further small-range motion vector refinement during the RD check for the same CU with integer luminance samples and 4-luminance sample MVD resolution, so that the time-consuming motion estimation process is not repeated three times.

[0119] · Conditionally call the RD check for a CU with 4-luminance sample MVD resolution. For a CU, when the RD cost of integer luminance sample MVD resolution is much greater than the RD cost of quarter-luminance sample MVD resolution, skip the RD check for the 4-luminance sample MVD resolution of the CU.

[0120] The encoding process is as Fig.12 shown. First, test the 1 / 4 pixel MV, calculate the RD cost and denote it as RDCost0, then test the integer MV, and the RD cost is denoted as RDCost1. If RDCost1 < th * RDCost0 (where th is a positive value), then test the 4 pixel MV; otherwise, skip the 4 pixel MV. Basically, when checking the integer or 4 pixel MV, the motion information and RD cost, etc. for the 1 / 4 pixel MV are known, which can be reused to accelerate the encoding process of the integer or 4 pixel MV.

[0121] 2.3.2 Triangle Prediction Mode

[0122] The concept of the Triangle Prediction Mode (TPM) is to introduce a new triangle partitioning for motion compensation prediction. As Figure 13A-13B shown, it divides the CU into two triangle prediction units along the diagonal or the opposite diagonal direction. Each triangle prediction unit in the CU is inter-frame predicted using its own uni-directional prediction motion vector and reference frame index derived from a single uni-directional prediction candidate list. After predicting the triangle prediction units, an adaptive weighting process is performed on the diagonal edges. Then, the transform and quantization processes are applied to the entire CU. Note that this mode only applies to the Merge mode (Note: The skip mode is regarded as a special Merge mode).

[0123] Figure 13A-13B is a diagram showing the division of the CU into two triangle prediction units (two partitioning modes); Fig.13A : 135-degree partitioning type (dividing from the upper left corner to the lower right corner), and Fig. 13B : 45-degree partitioning mode.

[0124] 2.3.2.1 Uni-directional Prediction Candidate List of TPM

[0125] The unidirectional prediction candidate list, called the TPM motion candidate list, includes five unidirectional prediction motion vector candidates. Fig.14 As shown, it is derived from seven neighboring blocks including five spatial neighboring blocks (1 to 5) and two temporal co-located blocks (6 to 7). The motion vectors of the seven neighboring blocks are collected and put into the unidirectional prediction candidate list in the order of unidirectional prediction motion vector, L0 motion vector of bidirectional prediction motion vector, L1 motion vector of bidirectional prediction motion vector and average motion vector of L0 motion vector and L1 motion vector of bidirectional prediction motion vector. If the number of candidates is less than five, zero motion vector is added to the list. The motion candidate added to the TPM list is called TPM candidate, and the motion information derived from the spatial / temporal blocks is called regular motion candidate.

[0126] More specifically, the following steps are involved:

[0127] 1) From A1, B1, B0, A0, B2, Col and Col2 (corresponding to Fig.14 In blocks 1-7), regular motion candidates are obtained. none No trimming required .

[0128] 2) Set the variable numCurrMergeCand = 0.

[0129] 3) For each regular motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2 and numCurrMergeCand is less than 5, if the regular motion candidate is uni-predicted (from list 0 or list 1), it is added directly to the Merge list as a TPM candidate and numCurrMergeCand is increased by 1. This TPM candidate is called an "originally uni-predicted candidate".

[0130] application Fully trimmed (full pruning).

[0131] 4) For each motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2 and numCurrMergeCand is less than 5, if the regular motion candidate is bidirectionally predicted, the motion information from List 0 is added to the TPM Merge list as a new TPM candidate (i.e., modified to be unidirectionally predicted from List 0), and numCurrMergeCand is increased by 1. Such a TPM candidate is called a "truncated List0-predicted candidate".

[0132] application Fully trimmed .

[0133] 5) For each motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2 and numCurrMergeCand is less than 5, if the regular motion candidate is bidirectionally predicted, the motion information from List 1 is added to the TPM Merge list (i.e., modified to be unidirectionally predicted from List 1), and numCurrMergeCand is increased by 1. Such a TPM candidate is called a "truncated List1-predicted candidate".

[0134] application Fully trimmed .

[0135] 6) For each motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2 and numCurrMergeCand is less than 5, if the regular motion candidate is bidirectionally predicted,

[0136] – If the slice QP of the list 0 reference picture is less than the slice QP of the list 1 reference picture, the motion information of list 1 is first scaled to the list 0 reference picture, and the average of two MVs (one from the original list 0 and the other is the scaled MV from list 1) is added to the TPM Merge list, such a candidate is called the averaged uni-prediction from List 0 motion candidate, and numCurrMergeCand is increased by 1.

[0137] – Otherwise, first scale the motion information of list 0 to the list 1 reference picture, and add the average of two MVs (one from the original list 1 and the other is the scaled MV from list 0) to the TPM Merge list, such a TPM candidate is called the averaged uni-prediction from List 1 motion candidate, and numCurrMergeCand is increased by 1.

[0138] application Fully trimmed .

[0139] 7) If numCurrMergeCand is less than 5, add a zero motion vector candidate.

[0140] When a candidate is inserted into the list, if it must be compared with all previously added candidates to see if it is the same as one of them, this process is called full pruning.

[0141] 2.3.2.2 Adaptive Weighting Process

[0142] After predicting each triangle prediction unit, an adaptive weighting process is applied to the diagonal edges between two triangle prediction units to derive the final prediction for the entire CU. Two groups of weighting factors are defined as follows:

[0143] The first weighting factor group: {7 / 8, 6 / 8, 4 / 8, 2 / 8, 1 / 8} and {7 / 8, 4 / 8, 1 / 8} are used for luma and chroma samples respectively;

[0144] The second weighting factor group: {7 / 8, 6 / 8, 5 / 8, 4 / 8, 3 / 8, 2 / 8, 1 / 8} and {6 / 8, 4 / 8, 2 / 8} are used for luma and chroma samples respectively.

[0145] The weighting factor set is selected based on the comparison of the motion vectors of the two triangle prediction units. When the reference pictures of the two triangle prediction units are different from each other or the difference of their motion vectors is greater than 16 pixels, the second weighting factor set is used. Otherwise, the first weighting factor set is used. Example Fig.15 shown.

[0146] 2.3.2.3 Signaling of Triangle Prediction Mode (TPM)

[0147] A one-bit flag indicating whether TPM is used may be signaled first. Thereafter, indications of two partitioning modes (as depicted in FIG. 13 ) and the Merge index selected for each of the two partitions are further signaled.

[0148] 2.3.2.3.1 TPM Flag Signaling

[0149] Let's denote the width and height of a luma block by W and H respectively. If W*H<64, triangle prediction mode is disabled.

[0150] When a block is encoded in affine mode, triangle prediction mode is also disabled.

[0151] When a block is encoded in Merge mode, a one-bit flag may be signaled to indicate whether triangle prediction mode is enabled or disabled for the block.

[0152] The flag is encoded with 3 contexts based on the following equation.

[0153] Ctx index = ((block L on the left is available && L is encoded with TPM?) 1:0)

[0154] +((Block A on the upper side is available &&A is encoded with TPM?) 1:0);

[0155] 2.3.2.3.2 Signaling an indication of two partitioning modes (as depicted in FIG. 13 ) and the Merge index selected for each of the two partitions

[0156] Note that the partition mode and the Merge index of the two partitions are jointly encoded. As an example, the two partitions are restricted from using the same reference index. Therefore, there are 2 (partition mode) * N (maximum number of Merge candidates) * (N-1) possibilities, where N is set to 5. An indication is encoded, and the mapping between the partition mode, the two Merge indexes and the encoding indication is derived from the array defined below:

[0157] const uint8_t g_TriangleCombination[TRIANGLE_MAX_NUM_CANDS][3]={

[0158] {0,1,0},{1,0,1},{1,0,2},{0,0,1},{0,2,0},

[0159] {1,0,3},{1,0,4},{1,1,0},{0,3,0},{0,4,0},

[0160] {0,0,2},{0,1,2},{1,1,2},{0,0,4},{0,0,3},

[0161] {0,1,3},{0,1,4},{1,1,4},{1,1,3},{1,2,1},

[0162] {1,2,0},{0,2,1},{0,4,3},{1,3,0},{1,3,2},

[0163] {1,3,4},{1,4,0},{1,3,1},{1,2,3},{1,4,1},

[0164] {0,4,1},{0,2,3},{1,4,2},{0,3,2},{1,4,3},

[0165] {0,3,1},{0,2,4},{1,2,4},{0,4,2},{0,3,4}};

[0166] Division mode (45 degrees or 135 degrees) = g_TriangleCombination [signaling indication] [0];

[0167] Merge index of candidate A = g_TriangleCombination[signaling indication][1];

[0168] Merge index of candidate B = g_TriangleCombination[signaling indication][2];

[0169] Once two motion candidates A and B are derived, the motion information of the two partitions (PU1 and PU2) can be set according to A or B. Whether PU1 uses the motion information of Merge candidate A or B depends on the prediction direction of the two motion candidates. Table 1 shows the relationship between the two derived motion candidates A and B and the two partitions.

[0170] Table 1: Motion information of partitions derived from two derived Merge candidates (A, B)

[0171] The predicted direction of A The predicted direction of B PU1 motion information PU2 motion information L0 L0 A(L0) B(L0) L1 L1 B(L1) A(L1) L0 L1 A(L0) B(L1) L1 L0 B(L0) A(L1)

[0172] 2.3.2.3.3 Indicated entropy coding (indicated by merge_triangle_idx)

[0173] merge_triangle_idx In the range of [0,39] (inclusive), a K-order Exponential Golomb (EG) code is used for binarization of merge_triangle_idx, where K is set to 1.

[0174] K Tier EG

[0175] To encode larger numbers with fewer bits (at the expense of using more bits to encode smaller numbers), this can be done using Non-negative integer To encode a non-negative integer x using a k-order EGOLomb code:

[0176] 1. Use the above 0th order exponential Golomb code pair Encode, then

[0177] 2. Use binary to solve x mod 2 k Encoding

[0178] Table 2: Exponential Golomb-k encoding examples

[0179]

[0180] 2.3.3 Affine Motion Compensated Prediction

[0181] In HEVC, only the translational motion model is applied to motion compensation prediction (MCP). In the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, a simplified affine transformation motion compensation prediction is applied using a 4-parameter affine model and a 6-parameter affine model. Figure 16A-16B As shown, the affine motion field of the block is given by a 4-parameter affine model ( Fig.16A )'s two control point motion vectors (CPMV) and a 6-parameter affine model ( Fig. 16B ) is described by three CPMVs.

[0182] The motion vector field (MVF) of the block is described by the following equations respectively through the 4-parameter affine model in equation (1) (where the 4 parameters are defined as variables a, b, e and f) and the 6-parameter affine model in equation (2) (where the 6 parameters are defined as variables a, b, c, d, e and f):

[0183]

[0184]

[0185] Where (mv h 0,mv h 0) is the motion vector of the upper left control point, and (mv h 1,mv h 1) is the motion vector of the upper right control point, and (mv h 2,mv h 2) is the motion vector of the lower left control point. All three motion vectors are called control point motion vectors (CPMVs). (x, y) represents the coordinates of the representative point in the current block relative to the upper left sample point, and (mv h (x,y),mv v (x,y)) is the motion vector derived for the sample at (x,y). The CP motion vector can be signaled (like in affine AMVP mode) or derived on-the-fly (like in affine Merge mode). w and h are the width and height of the current block. In practice, the division is implemented by right shift and integer operations. In VTM, the representative point is defined as the center position of the sub-block, for example, when the coordinates of the upper left corner of the sub-block relative to the upper left corner sample in the current block are (xs,ys), the coordinates of the representative point are defined as (xs+2,ys+2). For each sub-block (i.e. 4×4 in VTM), the motion vector of the entire sub-block is derived using the representative point.

[0186] To further simplify motion compensation prediction, sub-block based affine transformation prediction is applied. To derive the motion vector of each M×N (in the current VVC, both M and N are set to 4) sub-block, as Fig.17 As shown, the motion vector of the center sample of each sub-block is calculated according to equations (1) and (2) and rounded to a fractional precision of 1 / 16. Then, a 1 / 16 pixel motion compensation interpolation filter is applied to generate a prediction of each sub-block with the derived motion vector. The affine mode introduces a 1 / 16 pixel interpolation filter.

[0187] After MCP, the high-precision motion vector of each sub-block is rounded and saved with the same precision as the normal motion vector.

[0188] 2.3.3.1 Signaling of Affine Prediction

[0189] Similar to the translational motion model, there are also two modes for signaling the side information due to affine prediction. They are AFFINE_INTER and AFFINE_MERGE modes.

[0190] 2.3.3.2 AF_INTER mode

[0191] AF_INTER mode can be applied for CUs with width and height both greater than 8. A CU-level affine flag is signaled in the bitstream to indicate whether AF_INTER mode is used.

[0192] In this mode, for each reference picture list (list 0 or list 1), an affine AMVP candidate list is constructed with three types of affine motion predictors in the following order, where each candidate includes the estimated CPMV of the current block. The best CPMV found at the encoder side (such as Fig. 20 The difference between mv0, mv1, mv2) and the estimated CPMV is signaled. In addition, the index of the affine AMVP candidate from which the estimated CPMV is derived is further signaled.

[0193] 1) Inherited affine motion prediction

[0194] The order of checking is similar to that of spatial MVP in HEVC AMVP list construction. First, the left-inherited affine motion predictor is derived from the first block in {A1, A0} that is affine coded and has the same reference picture as the current block. Second, the top-inherited affine motion predictor is derived from the first block in {B1, B0, B2} that is affine coded and has the same reference picture as the current block. Fig.19 Five blocks A1, A0, B1, B0, B2 are depicted in FIG.

[0195] Once a neighboring block is found to be coded in affine mode, the CPMV of the coding unit covering the neighboring block is used to derive the prediction of the CPMV of the current block. For example, if A1 is coded in non-affine mode and A0 is coded in 4-parameter affine mode, the prediction of the affine MV inherited from the left will be derived from A0. In this case, the CPMV of the CU covering A0 is, Fig.21B CPMV in the upper left corner and CPMV in the upper right corner denoted as, is used to derive the estimated CPMV of the current block, which is composed of the top left (with coordinates (x0, y0)), top right (with coordinates (x1, y1)) and bottom right (with coordinates (x2, y2)) positions of the current block. express.

[0196] 2) Constructed affine motion prediction

[0197] like Fig. 20 As shown, the constructed affine motion predictor includes control point motion vectors (CPMVs) derived from adjacent inter-frame coded blocks with the same reference picture. If the current affine motion model is 4-parameter affine, the number of CPMVs is 2, otherwise if the current affine motion model is 6-parameter affine, the number of CPMVs is 3. The CPMV in the upper left corner It is derived from the MV of the first block in group {A, B, C} that is inter-coded and has the same reference picture as the current block. is derived from the MV of the first block in group {D,E} that is inter-coded and has the same reference picture as the current block. is derived from the MV at the first block in group {F,G} that is inter-coded and has the same reference picture as the current block.

[0198] - If the current affine motion model is 4-parameter affine, only when and When both are established, and The affine motion predictor constructed using the CPMV estimated for the upper left (with coordinates (x0, y0)) and upper right (with coordinates (x1, y1)) positions of the current block is inserted into the candidate list.

[0199] - If the current affine motion model is 6-parameter affine, only when and When all are established, and The affine motion predictors constructed using the CPMVs used as estimates of the upper left (with coordinates (x0, y0)), upper right (with coordinates (x1, y1)) and lower right (with coordinates (x2, y2)) positions of the current block are inserted into the candidate list.

[0200] When inserting the constructed affine motion predictor into the candidate list, no pruning process is applied.

[0201] 3) Normal AMVP motion prediction

[0202] The following conditions apply until the number of affine motion predictors reaches a maximum value.

[0203] 1) By setting all CPMVs equal to (if available), derive affine motion predictors.

[0204] 2) By setting all CPMVs equal to (if available), derive affine motion predictors.

[0205] 3) By setting all CPMVs equal to (if available), derive affine motion predictors.

[0206] 4) Derive affine motion predictors by setting all CPMVs equal to HEVC TMVP (if available).

[0207] 5) Derive the affine motion predictor by setting all CPMVs to zero MV.

[0208] Notice, It has been derived in the constructed affine motion predictor.

[0209] Figure 18A-18B A 4-parameter affine model and a 6-parameter affine model are shown respectively.

[0210] Fig.19 An example of MVP of AF_INTER of inherited affine candidate is shown.

[0211] Fig. 20 An example of MVP of AF_INTER of constructed affine candidates is shown.

[0212] In AF_INTER mode, when using the 4 / 6 parameter affine mode, 2 / 3 control points are required, and therefore 2 / 3 MVDs need to be encoded for these control points, such as Figure 18A-18B In the example, it is proposed to derive MV as follows, that is, predict mvd1 and mvd2 from mvd0.

[0213]

[0214]

[0215]

[0216] in, mvd i and mv1 are the predicted motion vector, motion vector difference and motion vector of the upper left pixel (i=0), upper right pixel (i=1) or lower left pixel (i=2), respectively. Fig.18B Note that the addition of two motion vectors (e.g., mvA(xA, yA) and mvB(xB, yB)) is equal to the separate summation of the two components, i.e., newMV = mvA + mvB, and the two components of newMV are set to (xA + xB) and (yA + yB), respectively.

[0217] 2.3.3.3AF_MERGE Mode

[0218] When a CU is applied in AF_MERGE mode, it obtains the first block encoded in affine mode from the valid adjacent reconstructed blocks. And the selection order of candidate blocks is from left, top, top right, bottom left to top left, such as Fig.21A As shown (in turn represented by A, B, C, D, E). For example, if the adjacent lower left block is encoded in affine mode, such as Fig.21B In the figure, the control point (CP) motion vector mv0 of the upper left corner, upper right corner and lower left corner of the adjacent CU / PU containing block A is extracted. N 、mv1 N and mv2 N . And based on mv0 N 、mv1 N and mv2 N To calculate the motion vector mv0 of the upper left corner / upper right / lower left corner of the current CU / PU C 、mv1 C and mv2 C (It is only used for 6-parameter affine mode.) It should be noted that in VTM-2.0, if the current block is affine coded, the sub-block at the upper left corner (e.g., 4×4 block in VTM) stores mv0, and the sub-block at the upper right corner stores mv1. If the current block is coded with a 6-parameter affine model, the sub-block at the lower left corner will store mv2; otherwise (with a 4-parameter affine model), LB stores mv2'. The other sub-blocks store the MV for MC.

[0219] In deriving the CPMV mv0 of the current CU C 、mv1 C and mv2 CAfterwards, the MVF of the current CU is generated according to the simplified affine motion model equations (1) and (2). In order to identify whether the current CU is coded in AF_MERGE mode, an affine flag is signaled in the bitstream when there is at least one neighboring block coded in affine mode.

[0220] Figure 21A-21B Candidate and CPMV predictor derivation for AF_MERGE with five neighboring blocks are shown respectively.

[0221] In the example, the affine merge candidate list is constructed by the following steps:

[0222] 1) Insert inherited affine candidates

[0223] An inherited affine candidate is a candidate that is derived from the affine motion model of its valid neighboring affine coded block. Up to two inherited affine candidates are derived from the affine motion model of the neighboring block and inserted into the candidate list. For the left predictor, the scanning order is {A0,A1}; for the top predictor, the scanning order is {B0,B1,B2}.

[0224] 2) Insert the constructed affine candidate

[0225] If the number of candidates in the affine merge candidate list is less than MaxNumAffineCand (set to 5), the constructed affine candidate is inserted into the candidate list. The constructed affine candidate refers to a candidate constructed by combining the neighboring motion information of each control point.

[0226] The motion information of the control point is first obtained from Fig. 22 CPk(k=1,2,3,4) represents the kth control point. A0, A1, A2, B0, B1, B2 and B3 are the spatial locations of predicted CPk(k=1,2,3); T is the temporal location of predicted CP4.

[0227] The coordinates of CP1, CP2, CP3 and CP4 are (0,0), (W,0), (H,0) and (W,H), respectively, where W and H are the width and height of the current block.

[0228] The motion information for each control point is obtained according to the following priority order:

[0229] For CP1, the priority of checking is B2->B3->A2. If B2 is available, use B2. Otherwise, if B2 is not available, use B3. If both B2 and B3 are not available, use A2. If all three candidates are not available, the motion information of CP1 cannot be obtained.

[0230] For CP2, the checking priority is B1->B0.

[0231] For CP3, the checking priority is A1->A0.

[0232] For CP4, use T.

[0233] Second, affine merge candidates are constructed using combinations of control points.

[0234] Constructing a 6-parameter affine candidate requires motion information of three control points. The three control points can be selected from one of the following four combinations ({CP1, CP2, CP4}, {CP1, CP2, CP3}, {CP2, CP3, CP4}, {CP1, CP3, CP4}). The combination {CP1, CP2, CP3}, {CP2, CP3, CP4}, {CP1, CP3, CP4} will be converted into a 6-parameter motion model represented by the top left, top right, and bottom left control points.

[0235] Constructing a 4-parameter affine candidate requires motion information of two control points. These two control points can be selected from the following two combinations ({CP1, CP2}, {CP1, CP3}). These two combinations will be converted into a 4-parameter motion model represented by the top left and top right control points.

[0236] The combinations of constructed affine candidates are inserted into the candidate list in the following order:

[0237] {CP1,CP2,CP3}, {CP1,CP2,CP4}, {CP1,CP3,CP4}, {CP2,CP3,CP4}, {CP1,CP2}, {CP1,CP3}.

[0238] Only when the CPs have the same reference index, available combinations of the motion information of the CPs are added to the affine Merge list.

[0239] 3) Fill with zero motion vectors

[0240] If the number of candidates in the affine merge candidate list is less than 5, a zero motion vector with a zero reference index is inserted into the candidate list until the list is full.

[0241] 2.3.4 Current Image Reference

[0242] Intra block copy (IBC), also known as current picture referencing (CPR), is used in the HEVC screen content coding extension (SCC). This tool is very effective for coding screen content videos, because repeated patterns in text and graphic-rich content often appear in the same picture. Using previously reconstructed blocks with the same or similar patterns as predictors can effectively reduce prediction errors, thereby improving coding efficiency. Fig.23 An example of intra block compensation is shown.

[0243] Similar to the CRP design in HEVC SCC, in VVC, the use of IBC mode is signaled at both the sequence and picture levels. When IBC mode is enabled under the sequence parameter set (SPS), it can be enabled at the picture level. When IBC mode is enabled at the picture level, the currently reconstructed picture is considered as a reference picture. Therefore, on the basis of the existing VVC inter-frame mode, no syntax changes are required at the block level to signal the use of IBC mode.

[0244] Key Features:

[0245] - It is treated as normal inter mode. Therefore, Merge and Skip modes can also be used in IBC mode. The Merge candidate list construction is unified and contains Merge candidates from neighboring locations, either coded in IBC mode or coded in HEVC inter mode. Depending on the selected Merge index, the current block in Merge or Skip mode can be merged into a neighbor coded in IBC mode or otherwise merged into a neighbor coded in normal inter mode with a different picture as reference picture.

[0246] - The block vector prediction and coding scheme for IBC mode reuses the scheme used for motion vector prediction and coding in HEVC inter-frame mode (AMVP and MVD coding).

[0247] - For IBC mode, motion vectors, also called block vectors, are encoded with integer pixel precision, but after decoding are stored in memory with 1 / 16 pixel precision because the interpolation and deblocking stages require quarter pixel precision. When used for motion vector prediction in IBC mode, the stored vector predictor is right-shifted by 4.

[0248] -Search range: limited to the current CTU.

[0249] - CPR is not allowed when affine mode / triangle mode / GBI / weighted prediction is enabled.

[0250] 2.3.5 Merge List Design in VVC

[0251] VVC supports three different Merge list construction processes:

[0252] 1) Sub-block Merge candidate list: It includes ATMVP and affine Merge candidates. Affine mode and ATMVP mode share a Merge list construction process. Here, ATMVP and affine Merge candidates can be added in sequence. The sub-block Merge list size is signaled in the stripe header and the maximum value is 5.

[0253] 2) Unidirectional prediction TPM Merge list: For triangle prediction mode, two partitions share a Merge list construction process, even though the two partitions can select their own Merge candidate indexes. When building the Merge list, the spatial neighboring blocks and two temporal blocks of the block are checked. The motion information derived from the spatial neighboring blocks and the temporal blocks is referred to as regular motion candidates in this article. These regular motion candidates are further used to derive multiple TPM candidates. Note that the transform is performed at the whole block level, even though the two partitions can use different motion vectors to generate their own prediction blocks.

[0254] The one-way predictive TPM Merge list size is fixed to 5.

[0255] 3) Regular Merge List: For the remaining coding blocks, a Merge list construction process is shared. Here, spatial / temporal / HMVP, paired combined bidirectional prediction Merge candidates and zero motion candidates can be inserted in order. The size of the regular Merge list is signaled in the slice header, and the maximum value is 6.

[0256] 2.3.5.1 Sub-block Merge Candidate List

[0257] It is recommended to put all sub-block related motion candidates except the regular Merge list for non-sub-block Merge candidates into a separate Merge list.

[0258] Sub-block related motion candidates are put into a separate Merge list, which is named "Sub-block Merge Candidate List".

[0259] In one example, the sub-block Merge candidate list includes affine Merge candidates, and ATMVP candidates and / or sub-block-based STMVP candidates.

[0260] In the example, the ATMVP Merge candidate in the normal Merge list is moved to the first position of the affine Merge list, so that all Merge candidates in the new list (ie, the sub-block based Merge candidate list) are based on the sub-block coding tool.

[0261] 2.3.5.2 Rule Merge List

[0262] Different from the Merge list design, in VVC, the history-based motion vector prediction (HMVP) method is adopted.

[0263] In HMVP, previously encoded motion information is stored. The motion information of previously encoded blocks is defined as HMVP candidates. Multiple HMVP candidates are stored in a table called HMVP table, and the table is maintained in real time during the encoding / decoding process. When encoding / decoding a new slice starts, the HMVP table is cleared. Whenever there is an inter-frame coded block, the associated motion information is added to the last entry of the table as a new HMVP candidate. The entire encoding process is as follows: Fig.24 shown.

[0264] HMVP candidates can be used in the AMVP and Merge candidate list building process. Fig.25 The modified Merge candidate list construction process is depicted (highlighted in gray). When the Merge candidate list is not full after the TMVP candidate is inserted, the HMVP candidates stored in the HMVP table can be used to fill the Merge candidate list. Considering that a block usually has a higher correlation with the nearest neighboring block in terms of motion information, the HMVP candidates in the table are inserted in descending order of index. The last entry in the table is added to the list first, and the first entry is added last. Similarly, redundancy elimination also applies to HMVP candidates. Once the total number of available Merge candidates reaches the maximum number of allowed Merge candidates notified by signaling, the Merge candidate list construction process terminates.

[0265] 3. Examples of technologies solved by the disclosed embodiments

[0266] In the current VVC design, the affine prediction mode can achieve significant coding gains for sequences with affine motion. However, it may have the following problems:

[0267] 1) For the bi-prediction affine mode, the correlation of the affine motion information between the two reference picture lists is not considered.

[0268] 2) For the affine merge candidate derivation process, the affine model (4-parameter or 6-parameter) type is directly inherited from the neighboring block, which requires additional row buffer size to store the affine model type.

[0269] 4. Description of various technologies

[0270] The following detailed inventions should be considered as examples to explain the general concept. These inventions should not be interpreted narrowly. In addition, these inventions can be combined in any way.

[0271] 1. It is proposed that candidates added to one reference picture list can be used to predict the CPMV of another reference picture.

[0272] a. In one example, the CPMV of one reference picture can be used to predict the CPMV of another reference picture.

[0273] b. In one example, the encoded MVD of one reference picture may be used (scaled if necessary) to predict the MVD of another reference picture.

[0274] 2. A symmetric affine coding mode is proposed, where the motion information of one reference picture list (list X) is signaled, while the motion information of another reference picture list (list Y, where Y is not equal to X) is always skipped.

[0275] a. In one example, the motion information (such as CPMV) of the reference picture list (list Y) that is not signaled can be derived from the motion information of the reference picture list (list X).

[0276] b. In one example, the prediction direction of this mode is also set to bi-directional prediction.

[0277] c. In one example, it is added as a new encoding mode. Alternatively, it can be used to replace the unidirectional affine encoding mode.

[0278] 3. It is proposed that the affine model type (e.g., 4-parameter or 6-parameter) can be used to determine the insertion order of affine candidates when constructing an affine candidate list (e.g., affine AMVP / Merge candidate list, sub-block Merge candidate list).

[0279] a. For the affine AMVP candidate list, neighboring blocks with the same affine model type may be given higher priority. For example, the motion information of a neighboring block with the same affine model type may be added to the AMVP list before the motion information of a second neighboring block with a different affine model type.

[0280] b. In one example, the affine type may be further signaled for the affine Merge mode.

[0281] c. For the affine merge candidate list and / or sub-block merge candidate list, neighboring blocks with the same affine model type may be given a higher priority.

[0282] i. In one example, the motion information of a neighboring block having the same affine model type as the first affine candidate may be added to the Merge list before the motion information of a second neighboring block having a different affine model type.

[0283] ii. In one example, the combination of constructed affine candidates may be reordered with the 4-parameter affine candidate (2 CPMVs) added before the 6-parameter affine candidate.

[0284] d. For the affine merge candidate list and / or the sub-block merge candidate list, more constructed affine candidates having the same affine model type as that of the selected merge candidate may be constructed.

[0285] i. In one example, the selected Merge candidate is the first available affine Merge candidate.

[0286] ii. In one example, the selected Merge candidate is an affine Merge candidate associated with a specific position of a spatially neighboring block.

[0287] e. For the affine merge candidate list and / or the sub-block merge candidate list, the order of constructed affine candidates may depend on the affine model type of the selected affine merge candidate.

[0288] i. In one example, the selected Merge candidate is the first available affine Merge candidate.

[0289] ii. In one example, the selected Merge candidate is an affine Merge candidate associated with a specific position of a spatially neighboring block.

[0290] 4. It is proposed not to store the affine model type associated with a block and not to use it for encoding subsequent blocks.

[0291] a. Alternatively, such information may be stored but only used to encode subsequent blocks within the current CTU or within the same M×N region or current CTU row. In one example, a picture / slice / strip may be divided into non-overlapping regions of size equal to M×N (e.g., 64×64).

[0292] b. In one example, after decoding an AMVP affine coded block with a 4-parameter affine model, instead of storing 2 CPMVs (from the top left and top right positions), 3 CPMVs (from the top left, top right, and bottom left positions) may be stored.

[0293] i. In one example, the CPMV of the upper left and upper right can be used to derive the CPMV of the lower left.

[0294] c. In one example, for each affine merge candidate, a 6-parameter affine model is used. Alternatively, for each affine merge candidate, a 4-parameter affine model is used.

[0295] 5. It is proposed that affine candidates can be reordered instead of using a fixed insertion order.

[0296] a. In one example, the reordering depends on the derived MVs of representative neighboring positions relative to the current block. Each affine candidate is used to derive motion vectors for several representative neighboring positions, and then the difference between the derived MV and the decoded MVs associated with those representative neighboring positions is calculated. Finally, the affine candidates are reordered in ascending order of difference.

[0297] b. In one example, the difference metric is MSE (mean squared error).

[0298] c. Alternatively, in addition, before computing the difference, if the affine candidate has a different reference picture than the representative neighboring block, the derived MV can be further scaled.

[0299] d. Alternatively, in addition, both the derived MV and the representative neighbor MV may be scaled to some selected reference picture before computing the difference.

[0300] e. In one example, only some affine candidates are reordered. For example, only neighboring affine candidates are reordered. They can always be inserted before the constructed affine candidate.

[0301] f. In one example, only constructed affine candidates are reordered. They may always be inserted after neighboring affine candidates.

[0302] g. In one example, only the top N affine candidates are re-ranked.

[0303] h. In one example, only the first N reordered affine merge candidates are inserted into the sub-block merge list.

[0304] i. In one example, if such a reordering is performed, the maximum length of the sub-block Merge list is reduced by K. For example, K=2.

[0305] 6. It is proposed to apply the re-ranking method described in 4 above to the affine AMVP list construction.

[0306] a. In one example, no affine AMVP index is signaled, and only the first of the reordered affine AMVP candidates is used as a predictor.

[0307] 7. It is proposed that multiple (eg, 2) affine candidates can be averaged to generate new affine candidates.

[0308] a. In one example, only affine candidates with the same reference picture are used to generate the average affine candidate.

[0309] b. In one example, affine candidates with different reference pictures can be used to generate an average affine candidate, and all affine candidates are scaled to the same reference picture.

[0310] i. In one example, the reference picture of any of these affine candidates can be used as the reference picture of the average affine candidate.

[0311] ii. In one example, the reference pictures of the average affine candidates may be defined for each CU / slice / slice / picture / video / slice and may be signaled in the slice header / slice header / PPS / VPS / SPS.

[0312] iii. In one example, reference pictures are implicitly predefined at both the encoder and the decoder.

[0313] iv. In one example, no scaling is performed.

[0314] Fig.26 is a block diagram of a video processing device 2600. Device 2600 can be used to implement one or more methods described herein. Device 2600 can be embodied in a smart phone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. Device 2600 may include one or more processors 2602, one or more memories 2604, and video processing hardware 2606. (Multiple) processors 2602 can be configured to implement one or more methods described in this document. Memory (multiple memories) 2604 can be used to store data and code for implementing the methods and techniques described herein. Video processing hardware 2606 can be used to implement some of the techniques described in this document in hardware circuits.

[0315] Fig. 272700 is a flow chart of an example method 2700 of video processing. The method may be performed by a video encoder, in a decoding loop thereof, or by a video decoder. The method 2700 includes: using motion information associated with a first reference picture from a single reference picture list, for a current block, deriving (2702) motion information associated with a second reference picture from another reference picture list different from the single reference picture list; and performing (2704) video processing on the current block based on the motion information associated with the first reference picture and the motion information associated with the second reference picture.

[0316] Fig.28 28 is a flowchart of an example method 2800 of video processing. The method may be performed by a video encoder, in a decoding loop thereof, or by a video decoder. The method 2800 includes determining (2802) an affine mode type associated with a current block; and performing (2804) video processing on the current block based on the determined affine mode type; wherein the affine mode type is restricted for use in a portion of a video picture in which the current block is located, and the determined affine mode type is prohibited for use in other portions of the video picture.

[0317] Fig.29 2900 is a flowchart of an example method 2900 of video processing. The method can be performed by a video encoder, in its decoding loop, or by a video decoder. The method 2900 includes generating (2902) an affine candidate list for a current block, wherein the affine candidate list includes an additional affine candidate generated by averaging multiple affine candidates from the affine candidate list; and performing (2904) video processing on the current block based on the expanded affine candidate list.

[0318] It should be appreciated that several techniques have been disclosed that are beneficial to video encoder and decoder embodiments incorporated into video processing devices such as smartphones, laptops, desktop computers, and similar devices by allowing the use of affine models in video compression and decompression as described in many of the techniques and embodiments in this document.

[0319] Some embodiments may be described using the following examples.

[0320] 1. A method for video processing, comprising:

[0321] Using the motion information associated with the first reference picture from the single reference picture list, deriving, for the current block, motion information associated with a second reference picture from another reference picture list different from the single reference picture list; and

[0322] Video processing is performed on the current block based on the motion information associated with the first reference picture and the motion information associated with the second reference picture.

[0323] 2. The method of example 1, wherein motion information associated with the first reference picture is signaled, and deriving motion information associated with the second reference picture comprises:

[0324] Motion information associated with the second reference picture is derived from the signaled motion information associated with the first reference picture.

[0325] 3. A method according to any one of Examples 1-2, wherein the motion information includes a control point motion vector (CPMV), and the CPMV associated with the second reference picture is derived from the CPMV associated with the first reference picture.

[0326] 4. The method of any one of Examples 1-2, wherein the motion information comprises a motion vector difference (MVD), and the MVD associated with the second reference picture is derived from the MVD associated with the first reference picture.

[0327] 5. A method according to any one of Examples 1-4, wherein performing video processing includes performing prediction for the current block based on a bi-predictive affine mode.

[0328] 6. A method according to any one of Examples 1-4, wherein performing video processing includes performing prediction for the current block based on a symmetric affine pattern.

[0329] 7. A method according to Example 6, wherein the symmetric affine mode is set to a bidirectional affine mode.

[0330] 8. A method according to Example 6, wherein the symmetric affine mode is a mode independent of the bidirectional affine mode and the unidirectional affine mode; or the selectable affine mode of the current block includes the symmetric affine mode but does not include the unidirectional affine mode.

[0331] 9. A method for video processing, comprising:

[0332] determining an affine mode type associated with the current block; and

[0333] performing video processing on the current block based on the determined affine mode type;

[0334] The affine mode type is restricted to be used in a portion of the video picture where the current block is located, and the determined affine mode type is prohibited to be used in other portions of the video picture.

[0335] 10. The method of Example 9, wherein the portion of the video picture is the current block and the affine pattern type is not stored.

[0336] 11. The method of example 9 or 10, wherein the portion of the video picture corresponds to at least one of:

[0337] The coding tree unit (CTU) where the current block is located:

[0338] the M×N region where the current block is located; and

[0339] The CTU row where the current block is located.

[0340] 12. The method of example 11, wherein the MxN region is one of a plurality of non-overlapping MxN regions having the same size partitioned from at least one of a picture, a slice, and a stripe.

[0341] 13. The method of example 11 or 12, wherein M=N=64.

[0342] 14. The method of any of Examples 9-13, wherein the motion information associated with the current block comprises a control point motion vector (CPMV) of the current block, the method comprising:

[0343] The three control point motion vectors (CPMVs) of the current block are stored regardless of whether a 6-parameter affine model or a 4-parameter affine model is applied to the current block.

[0344] 15. The method according to Example 14, wherein the three CPMVs include CPMVs located at the upper left corner, the upper right corner, and the lower left corner of the current block, respectively.

[0345] 16. The method according to example 15, further comprising:

[0346] The CPMV at the lower left corner of the current block is derived from the CPMVs at the upper left corner and the upper right corner of the current block, respectively.

[0347] 17. The method of any one of Examples 14-16, wherein an advanced motion vector prediction (AMVP) mode is applied to the current block.

[0348] 18. The method of any one of Examples 14-16, wherein applying an affine merge mode to the current block comprises:

[0349] The motion information of the current block is derived from multiple affine Merge candidates, each of which uses a 6-parameter affine model or a 4-parameter affine model.

[0350] 19. A method for video processing, comprising:

[0351] generating an affine candidate list for the current block, wherein the affine candidate list includes an additional affine candidate generated by averaging a plurality of affine candidates from the affine candidate list; and

[0352] Video processing is performed on the current block based on the expanded affine candidate list.

[0353] 20. The method of Example 19, wherein the number of the plurality of affine candidates is 2.

[0354] 21. A method according to Example 19 or 20, wherein multiple affine candidates refer to the same reference image.

[0355] 22. A method according to example 19 or 20, wherein at least two of the multiple affine candidates refer to different reference pictures.

[0356] 23. The method of example 22, wherein all of the multiple affine candidates are scaled to the same reference picture of the additional candidates.

[0357] 24. A method according to example 22 or 23, wherein a reference picture referenced by any one of the plurality of affine candidates is used as a reference picture for the additional candidate.

[0358] 25. The method of any one of Examples 21-24, wherein additional candidate reference pictures are defined for at least one of a coding unit level, a slice level, a slice level, a picture level, and a video level.

[0359] 26. A method according to any one of Examples 21-25, wherein the additional candidate reference pictures are signaled in at least one of a slice header, a slice header, a picture parameter set (PPS), a video parameter set (VPS), and a sequence parameter set (SPS).

[0360] 27. A method according to any of Examples 21-25, wherein additional candidate reference pictures are implicitly predefined at both the encoding and decoding sides.

[0361] 28. The method of any one of Examples 1-27, wherein the video processing comprises at least one of encoding the video block into a bitstream representation of the video block and decoding the video block from the bitstream representation of the video block.

[0362] 29. A video processing device comprising a processor configured to implement the method of any one of Examples 1 to 28.

[0363] 30. A computer program product stored on a non-transitory computer readable medium, the computer program product comprising program code for performing the method of any of Examples 1 to 28.

[0364] The disclosed and other solutions, examples, embodiments, modules and functional operations described in this document may be implemented in digital electronic circuits, or in computer software, firmware or hardware, including the structures disclosed in this document and their structural equivalents, or in a combination of one or more of them. The disclosed and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium, for execution by a data processing device or for controlling the operation of the data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material combination that implements a machine-readable propagation signal, or a combination of one or more of them. The term "data processing device" covers all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an operating environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal, for example, a machine-generated electrical signal, an optical signal, or an electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0365] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one or more computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0366] The processes and logic flows described herein may be performed by one or more programmable processors running one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0367] For example, processors suitable for executing computer programs include general and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operably coupled to receive data from one or more mass storage devices or to transfer data to them or both. However, a computer does not necessarily require such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, for example, including semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

[0368] Although this patent document contains many details, these should not be interpreted as limitations on any subject matter or the scope of the claimed protection, but rather as descriptions of features unique to specific embodiments of specific technologies. Certain features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although the above-mentioned features may be described as working in certain combinations and even initially claimed to be protected in this way, in some cases, one or more features in the combination may be deleted from the claimed combination, and the claimed combination may point to a sub-combination or a variant of the sub-combination.

[0369] Similarly, although operations are described in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in sequence, or that all of the illustrated operations be performed, in order to achieve the desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0370] Only a few implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A method for video processing, comprising: Using motion information associated with a first reference picture from a single reference picture list, deriving, for a current block, motion information associated with a second reference picture from another reference picture list different from the single reference picture list; as well as performing video processing on the current block based on motion information associated with the first reference picture and motion information associated with the second reference picture; Wherein, performing video processing includes performing prediction on the current block based on a symmetric affine pattern, The symmetric affine mode is a mode independent of the bidirectional affine mode and the unidirectional affine mode, the selectable affine mode of the current block includes the symmetric affine mode but does not include the unidirectional affine mode, and In the symmetric affine mode, the motion information related to the first reference picture is signaled, and the motion information related to the second reference picture is skipped.

2. The method according to claim 1, wherein: The motion information associated with the first reference picture is signaled, and deriving the motion information associated with the second reference picture includes: Motion information associated with the second reference picture is derived from the signaled motion information associated with the first reference picture.

3. The method according to any one of claims 1 to 2, wherein: The motion information comprises a control point motion vector CPMV, and the CPMV associated with the second reference picture is derived from the CPMV associated with the first reference picture.

4. The method according to any one of claims 1 to 2, wherein: The motion information comprises a motion vector difference MVD, and the MVD associated with the second reference picture is derived from the MVD associated with the first reference picture.

5. The method according to claim 1, comprising: determining an affine pattern type associated with the current block; as well as performing video processing on the current block based on the determined affine mode type; The affine mode type is restricted to be used in a portion of the video picture where the current block is located, and the determined affine mode type is prohibited to be used in other portions of the video picture.

6. The method according to claim 5, wherein: The portion of the video picture is the current block and the affine mode type is not stored.

7. The method according to claim 5 or 6, wherein: The portion of the video picture corresponds to at least one of: The coding tree unit CTU where the current block is located: the M×N region where the current block is located; and The CTU row where the current block is located.

8. The method according to claim 7, wherein: The M×N region is one of a plurality of non-overlapping M×N regions having the same size partitioned from at least one of a picture, a slice, and a slice.

9. The method according to claim 7, wherein: M=N=64.

10. The method according to claim 5 or 6, wherein: The motion information associated with the current block includes a control point motion vector CPMV of the current block, and the method includes: The three control point motion vectors CPMV of the current block are stored, whether a 6-parameter affine model or a 4-parameter affine model is applied to the current block.

11. The method according to claim 10, wherein: The three CPMVs include CPMVs located at the upper left corner, the upper right corner, and the lower left corner of the current block, respectively.

12. The method according to claim 11, further comprising: The CPMV at the lower left corner of the current block is derived from the CPMVs at the upper left corner and the upper right corner of the current block respectively.

13. The method according to claim 10, wherein: An advanced motion vector prediction (AMVP) mode is applied to the current block.

14. The method according to claim 10, wherein: Applying an affine Merge mode to the current block, the method comprising: The motion information of the current block is derived from a plurality of affine Merge candidates, each of which utilizes a 6-parameter affine model or a 4-parameter affine model.

15. The method according to claim 1, comprising: generating an affine candidate list for a current block, wherein the affine candidate list includes an additional affine candidate generated by averaging a plurality of affine candidates from the affine candidate list; and Video processing is performed on the current block based on the expanded affine candidate list.

16. The method according to claim 15, wherein: The number of the plurality of affine candidates is 2.

17. The method according to claim 15 or 16, wherein: The multiple affine candidates refer to the same reference picture.

18. The method according to claim 15 or 16, wherein: At least two of the plurality of affine candidates refer to different reference pictures.

19. The method according to claim 18, wherein: The plurality of affine candidates are all scaled to the same reference picture of the additional affine candidate.

20. The method according to claim 18, wherein: A reference picture referred to by any one of the plurality of affine candidates is used as a reference picture of the additional affine candidate.

21. The method according to claim 17, wherein: The reference picture of the additional affine candidate is defined for at least one of a coding unit level, a slice level, a slice level, a picture level, and a video level.

22. The method according to claim 17, wherein: The reference picture of the additional affine candidate is signaled in at least one of a slice header, a slice header, a picture parameter set (PPS), a video parameter set (VPS), and a sequence parameter set (SPS).

23. The method according to claim 17, wherein: The reference pictures of the additional affine candidates are implicitly predefined at both encoding and decoding sides.

24. The method according to claim 1 or 2, wherein: The video processing includes at least one of encoding a video block into a bitstream representation of the video block and decoding the video block from a bitstream representation of the video block.

25. A video processing apparatus comprising a processor configured to implement the method of any one of claims 1 to 24.

26. A non-transitory computer readable medium having stored thereon a computer program for executing the method of any one of claims 1 to 24.

Images