MMVD and combination of SMVD with motion and prediction models

Abstract

The present invention relates to MMVD and combination of SMVD with motion and prediction models, mainly to extending motion modes (e.g., merge motion vector differences and symmetric motion vector differences) to motion models other than simple translation models, e.g., in combination with merge and alternative temporal motion vector prediction modes. Embodiments extend the use of MMVD and SMVD motion vector coding tools to all motion model derivation and temporal prediction methods supported in the proposed video standard in order to improve overall compression performance. Particular embodiments describe combining MMVD or SMVD with an affine motion model, an ATMVP motion model, a planar motion model, a regression motion field, a triangular partition based motion model, a GBI time prediction method, an LIC time prediction method, and a multi-hypothesis prediction method.

Description

[0001] This application is a divisional application of Chinese invention patent application No. 201980083741.1, filed on December 16, 2019, entitled “Combination of MMVD and SMVD with Motion and Prediction Models”, and the contents of the parent application are incorporated herein by reference. Technical Field

[0002] At least one embodiment of the present invention generally relates to a method or apparatus for video encoding or decoding, compression or decompression. Background Art

[0003] In order to achieve high compression efficiency, image and video coding schemes generally use prediction (including motion vector prediction) and transformation to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlation, and then the difference between the original image and the predicted image (which is usually expressed as a prediction error or prediction residual) is transformed, quantized, and entropy encoded. In order to reconstruct the video, the compressed data is decoded by the inverse process corresponding to the entropy encoding, quantization, transformation, and prediction. Summary of the invention

[0004] At least one embodiment of the present invention generally relates to a method or apparatus for video encoding or decoding, and more particularly, to a method or apparatus for simplifying a decoding mode based on a parameter model related to adjacent samples.

[0005] According to a first aspect, a method is provided. The method comprises the steps of: indicating a first motion mode by syntax in a video bitstream; indicating the use of a second motion mode by the presence of syntax in the video bitstream and, if present, including information related to the second motion mode; and encoding a video block using motion information corresponding to the first and second motion modes.

[0006] According to a second aspect, a method is provided. The method comprises the steps of: parsing a video bitstream for syntax indicating a first motion mode; parsing the video bitstream for syntax indicating the presence of a second motion mode and, if present, determining information related to the second motion mode; obtaining motion information corresponding to the first motion mode; and decoding the block using the motion information.

[0007] According to another aspect, a device is provided. The device includes a processor. The processor can be configured to encode a block of a video or decode a bitstream by performing any of the above methods.

[0008] According to another main aspect of at least one embodiment, there is provided an apparatus comprising: an apparatus according to any of the decoding embodiments; and (i) an antenna configured to receive a signal comprising a video block, (ii) a band limiter configured to limit the received signal to a frequency band comprising the video block, or (iii) a display configured to display an output representing the video block.

[0009] According to another main aspect of at least one embodiment, a non-transitory computer-readable medium is provided, containing data content generated according to any of the described encoding embodiments or variations.

[0010] According to another main aspect of at least one embodiment, there is provided a signal comprising video data generated according to any of the described encoding embodiments or variations.

[0011] According to another main aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.

[0012] According to another main aspect of at least one embodiment, there is provided a computer program product comprising instructions which, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variations.

[0013] These and other aspects, features and advantages of the broad aspects will become apparent from the following detailed description of illustrative embodiments, which is to be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 An example of a coding tree unit and a coding tree concept for representing a compressed HEVC picture is shown.

[0015] Figure 2 An example of dividing a coding tree unit into a coding unit, a prediction unit, and a transform unit is shown.

[0016] Figure 3 A standard, common video compression scheme is shown.

[0017] Figure 4 A standard, common video decompression scheme is shown.

[0018] Figure 5 A simplified block diagram of the decoding process in the current scheme is shown.

[0019] Figure 6 A diagram of the MMVD in the case of bi-prediction mode is shown.

[0020] Figure 7An example of SMVD (Symmetric Motion Vector Difference) is shown.

[0021] Figure 8 An example of ATMVP motion prediction for a coding unit is shown.

[0022] Fig. 9 An example of a simple affine model used in the joint exploration model and VTM is shown.

[0023] Fig.10 An example of an affine motion vector field based on a 4×4 sub-CU is shown.

[0024] Fig.11 An example of a motion vector prediction process for a coding unit of affine inter decoding is shown.

[0025] Fig.12 Motion vector prediction candidates in affine merge mode are shown.

[0026] Fig.13 The spatial derivation of the affine motion field control points in the case of affine merging is shown.

[0027] Fig.14 An example planar motion vector prediction process is shown.

[0028] Fig.15 Regression-based motion vector field construction is shown.

[0029] Fig.16 An example of dividing a coding unit into two triangle prediction units (PUs) is shown.

[0030] Fig.17 Non-rectangular partitions and associated OBMC diagonal weightings are shown.

[0031] Fig.18 An example of how LIC parameters are derived from (a) neighboring reconstructed samples and (b) corresponding collocated reference samples is shown.

[0032] Fig.19 An example of neighboring samples used to derive IC parameters is shown.

[0033] Fig. 20 The spatial positions considered when computing non-sub-block STMVP merge candidates are shown.

[0034] Fig.21 A simplified block diagram of the motion vector decoding process when MMVD and bi-directional prediction are combined is shown.

[0035] Fig. 22 An example of a CPR limit is shown.

[0036] Fig.23 Shown is the application of restrictions to CPR MV in MMVD mode.

[0037] Fig.24 The MMVD mode adaptation in the ATMVP case is shown.

[0038] Fig.25 A simplified block diagram of a first version of the affine motion generation process using MMVD is shown.

[0039] Fig.26 A simplified block diagram of a second version of the affine motion generation process using MMVD is shown.

[0040] Fig. 27 The limits on the allowed magnitude of the differential MVD based on the MVD index of the first CPMV for the considered affine motion field are shown.

[0041] Fig.28 An example of a planar MVP mode combined with MMVD is shown.

[0042] Fig.29 A simplified block diagram of a first version of the PMVD motion generation process using MMVD is shown.

[0043] Fig.30 A simplified block diagram of a second version of the PMVD motion generation process using MMVD is shown.

[0044] Fig.31 A simplified block diagram of a third version of the PMVD motion generation process using MMVD is shown.

[0045] Fig.32 Regression-based motion vector field construction is shown.

[0046] Fig.33 A simplified block diagram of one version of the affine motion generation process using SMVD is shown.

[0047] Fig.34 A processor-based system for encoding and decoding is shown for use with the described aspects.

[0048] Fig.35 An embodiment of a method of encoding according to the mainly described aspects is shown.

[0049] Fig.36 An embodiment of a method of decoding according to the mainly described aspects is shown.

[0050] Fig.37 An embodiment of a device according to the mainly described aspects is shown. DETAILED DESCRIPTION

[0051] The embodiments described herein belong to the field of video compression and are primarily concerned with video compression and video encoding and decoding. The main aspects described are directed to providing a mechanism to manipulate restrictions in high-level video coding syntax or video coding semantics to constrain the possible set of tool combinations.

[0052] In order to achieve high compression efficiency, image and video coding schemes generally use prediction (including motion vector prediction) and transform to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlation, and then the difference between the original image and the predicted image, which is usually represented as a prediction error or prediction residual, is transformed, quantized, and entropy decoded. In order to reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy decoding, quantization, transformation, and prediction.

[0053] In the HEVC (High Efficiency Video Coding, ISO / IEC 23008-2, ITU-T H.265) video compression standard, motion compensated temporal prediction is used to exploit the redundancy existing between consecutive pictures in a video.

[0054] To do this, a motion vector is associated with each prediction unit (PU). Each coding tree unit (CTU) is represented by a coding tree in the compressed domain. This is a quadtree partitioning of the CTU, where each leaf is called a coding unit (CU). Figure 1 as shown in .

[0055] Each CU is then given some intra or inter prediction parameters (prediction information). To this end, it is spatially partitioned into one or more prediction units (PUs), each of which is assigned some prediction information. Figure 2 In , intra or inter coding modes are assigned at the CU level.

[0056] Specifically, one motion vector is assigned to each PU in HEVC. This motion vector is used for motion compensated temporal prediction of the considered PU. Therefore, in HEVC, the motion model of the linked prediction block and its reference blocks simply consists of translation.

[0057] HEVC uses two modes to encode the motion data, which are called AMVP (Adaptive Motion Vector Prediction) and Merge.

[0058] AMVP basically consists in signaling the reference picture(s) used to predict the current PU, the motion vector predictor index (taken from a list with 2 predictors), and the motion vector difference. This article deals with the merge mode, so AMVP is not involved in the following.

[0059] The merge mode consists in signaling and decoding the index of some motion data collected in a list of motion data predictors. This list consists of 5 candidates and is constructed in the same way on the decoder and encoding side. Therefore, the merge mode aims to derive some motion information taken from the merge list. The merge list usually contains motion information associated with some spatial and temporal surrounding blocks that are available in their decoding state when the current PU is being processed.

[0060] In VTM-3 (VVC Draft 3), a new type of motion vector decoding is adopted, which is called MMVD (Merge with Motion Vector Difference). The MMVD mode basically introduces a motion vector difference and adds it to a classic merge candidate to generate a certain motion information of the block to be encoded or decoded.

[0061] MMVD improves the decoding efficiency of the VVC decoding system. However, it has been observed that in VVC draft 3, the MMVD tool is only applied to the usual translation merge mode. It may be interesting to combine the MMVD tool with some other inter-frame decoding modes, especially some other motion vector (or motion model in the case of sub-block based motion fields) generation tools included in the VVC coding system.

[0062] Another motion vector decoding tool proposed for VVC standardization is the so-called Symmetric Motion Vector Decoding (SMVD). Like MMVD, SMVD is a new mode to improve the efficiency of motion vector decoding. It is proposed in AMVP mode. However, SMVD is only applied to the case of translation AMVP. As for MMVD, it may be beneficial to extend it to some motion models beyond the simple translation model.

[0063] In the JVET (Joint Video Exploration Team) proposal for a new video compression standard, called the Joint Exploration Model (JEM), it has been proposed to accept a quadtree-binary tree (QTBT) block partition structure due to high compression performance. By partitioning a block in a binary tree (BT) horizontally or vertically in the middle, it can be partitioned into two sub-blocks of equal size. Therefore, a BT block can have a rectangular shape with unequal width and height, unlike blocks in QT, which always have a square shape with equal height and width. In HEVC, the angular intra prediction directions are defined from 45° to -135° over a 180° angle, and they have been maintained in JEM, which has made the definition of the angular direction independent of the target block shape.

[0064] To encode these blocks, intra prediction is used to provide an estimated version of the block using previously reconstructed neighboring samples. The difference between the source block and the prediction is then encoded. In the above classical codec, a single row of reference samples is used to the left and top of the current block.

[0065] In HEVC (High Efficiency Video Coding, H.265), the encoding of frames of a video sequence is based on a quadtree (QT) block partition structure. Frames are divided into square coding tree units (CTUs), all of which are divided into multiple coding units (CUs) based on a rate-distortion (RD) criterion and undergo quadtree-based partitioning. Each CU is intra-predicted (i.e., it is spatially predicted from causally adjacent CUs) or inter-predicted (i.e., it is temporally predicted from a decoded reference frame). In I slices, all CUs are intra-predicted, while in P slices and B slices, CUs can be intra- or inter-predicted. For intra prediction, HEVC defines 35 prediction modes, including one plane mode (indexed as mode 0), one DC mode (indexed as mode 1), and 33 angular modes (indexed as modes 2 to 34). The angular mode is associated with a prediction direction ranging from 45 degrees to -135 degrees in the clockwise direction. Since HEVC supports a quadtree (QT) block partition structure, all prediction units (PUs) have a square shape. Therefore, from the perspective of the PU (prediction unit) shape, the definition of prediction angles from 45 degrees to -135 degrees is reasonable. For a target prediction unit of size N×N pixels, the top reference array and the left reference array each have a size of 2N+1 samples, which is required to cover the above angle range of all target pixels. Considering that the height and width of the PU have equal lengths, it also makes sense that the lengths of the two reference arrays are equal.

[0066] The present invention belongs to the field of video compression. Its purpose is to improve the bidirectional prediction in the inter-frame decoding block compared with the existing video compression system. The present invention also proposes to separate the luminance decoding tree and the chrominance decoding tree for inter-frame slices.

[0067] In the HEVC video compression standard, pictures are divided into so-called coding tree units (CTUs), which are typically 64×64, 128×128 or 256×256 pixels in size. Each CTU is represented by a coding tree in the compressed domain. This is a quadtree partitioning of the CTU, where each leaf is called a coding unit (CU).

[0068] Each CU is then given some intra or inter prediction parameters (prediction information). To this end, it is spatially partitioned into one or more prediction units (PUs), each PU being assigned some prediction information. Intra or inter coding modes are assigned on CU level.

[0069] In order to represent picture data in a more flexible way in the compressed domain, emerging video compression tools have been proposed that include a coding tree unit representation in the compressed domain. The advantage of this more flexible representation of the coding tree is that it provides increased compression efficiency compared to the CU / PU / TU arrangement of the HEVC standard.

[0070] The Quadtree plus Binary Tree (QTBT) decoding tool provides this increased flexibility. It consists in a decoding tree where a decoding unit can be divided into a quadtree and a binary tree form. Such a decoding tree representation of a decoding tree unit is shown.

[0071] The partitioning of the coding units is decided at the encoder side by a rate-distortion optimization process, which involves determining the QTBT representation of the CTU with the minimum rate-distortion cost.

[0072] In QTBT technology, a CU has a square or rectangular shape. The size of a coding unit is always a power of 2 and is usually from 4 to 128.

[0073] In addition to this rectangular shape of the coding unit, this new CTU representation has the following different characteristics compared to HEVC.

[0074] The QTBT decomposition of a CTU consists of two stages: first, the CTU is partitioned in a quadtree manner, and then each quadtree leaf can be further partitioned in a binary tree manner.

[0075] One problem addressed by the present invention is how to extend the use of MMVD and SMVD motion vector decoding tools to all motion model derivation methods and temporal prediction methods supported in currently proposed video standards in order to improve the overall compression performance of these proposed standards.

[0076] The basic principle of this disclosure consists of two aspects:

[0077] - The use of MMVD motion vector coding is extended to all motion models and / or temporal prediction methods supported in VVC Draft 3, as well as other motion models proposed for the VVC standardization process. Specifically, in the present invention, it is described how to combine MMVD with affine motion model, ATMVP motion model, planar motion model, regression motion field, motion model based on triangle partitioning, GBI temporal prediction method, LIC temporal prediction method and multi-hypothesis prediction method. Rich use of MMVD in the case of bidirectional prediction is also provided.

[0078] – Extend the use of the SMVD motion vector decoding tool to all motion models and / or temporal prediction methods supported in VVC Draft 3, as well as other motion model generators proposed for the VVC standardization process. Specifically, in the present invention, it is described how to combine SMVD with affine motion model, ATMVP motion model, planar motion model, regression motion field, triangle partition-based motion model, GBI temporal prediction method, LIC temporal prediction method and multi-hypothesis prediction method.

[0079] In the VVC video standard under development, multiple inter-frame prediction modes are supported or being proposed.

[0080] Figure 5 Depicted is a simplified block diagram of the inter decoding process in version 3 of the VVC draft. The process starts with deriving the MV prediction value (mvp) based on the construction of the MV candidate list (1001), which may be different depending on the inter mode used to code the CU or PU.

[0081] Step 1002 is the motion vector difference (MVd) decoding step. The decoded MVd will be used to reconstruct the motion vector(s) of the considered CU / PU by adding it to the motion vector prediction value obtained in the previous step (1001).

[0082] The next step 1003 consists in deriving a motion model from the motion vector values ​​emanating from the two previous steps. This model can be a simple translation model or a more complex model, as will be described later.

[0083] In step 1004, prediction of samples in the CU is performed using the decoded MV for one reference picture, and in case of bidirectional prediction mode, step 1005 is performed for the other reference picture. In step 1006, the prediction signal is further refined for one reference picture, and in case of bidirectional prediction mode, the prediction signal is further refined for the other reference picture in step 1007. When applicable, a combination of bidirectional prediction, or hybrid intra / inter prediction, or for PUs within a CU is implemented in step 1008. A final refinement step of the prediction signal is performed in step 1009.

[0084] Figure 5 Also shown are the different modes applied to these different steps in VTM3. Tools indicated with this (*) correspond to tools that were being proposed for adoption but were not adopted at the VVC Draft 3 stage.

[0085] In inter frames, two basic modes for deriving MVs are used: merge and AMVP. In both cases, a list of MV candidates is derived. This derivation process is usually different for the two modes, and even for a given mode, variations are applied depending on other settings applied to the CU (e.g., merge or AMVP, translation or affine model).

[0086] In merge mode, the predictor index is signaled to obtain the MV in the merge candidate list. In addition, if skip mode is not applied, the sample prediction residual is signaled. In AMVP mode, for each reference picture (1 reference picture in case of unidirectional prediction and 2 reference pictures in case of bidirectional prediction), the reference frame index, predictor index and motion vector difference (mvd) are signaled. In addition, the sample prediction residual is signaled.

[0087] Table 1 provides an overview of the contents of the different candidate lists in the current VTM.

[0088] Table 1. Summary of candidates in the different MV candidate lists

[0089]

[0090] A summary of the inter modes considered in the present invention is provided in the following table, with some indicative figures on the performance (bit rate change in percentage for psnrY in random access configuration).

[0091] tool describe performance affine Affine motion model -3.8 atmvp Alternative Temporal Motion Vector Prediction -0.7 bio Bidirectional optical flow -1.3 cpr MV reference current picture Specific screen content gb Generalized Dual Prediction -0.7 lic Local illumination compensation -0.5 mh Combined multi-hypothesis merging / inter -0.5 mmvd Merged MVD, also known as the ultimate MV representation -1.3 flat Mv field of each sub-block based on the plane model -0.6 rmvf Mv field of each sub-block based on regression model -0.7 smvd Symmetric MVD, only one MVD is decoded in bidirectional prediction -0.6 triangle Triangle partition -0.6

[0092] The following subsections provide more details on the considered inter modes, where syntax elements are associated with these modes or tools.

[0093] The main aspects described here focus on Figure 5 The object of the present invention is to extend the use of MMVD and SMVD motion vector decoding tools to motion models other than simple translational motion models.

[0094] MMVD Motion Vector Difference Decoding Tool Description

[0095] MMVD is only applied in merge mode. It uses a merge candidate list. The flag "mmvd_skip" indicates whether MMVD mode is applicable. When this mode is applicable, the MV difference (mmvd) is established as:

[0096] – Signaling syntax elements (here mmvd_idx) to construct the correction MV mmvd,

[0097] This syntax element includes the following information:

[0098] o Base MV index, selected by the encoder from among the two first translation merge candidates.

[0099] o is denoted as the index of mmvd_dir_idx, which is related to the direction D (currently 4) in the (x, y) coordinate system (specifying the table dir[] with 4 elements {(0,1),(1,0),(-1,0),(0,-1)})

[0100] o is denoted as the index of mmvd_dist_idx, which is related to the distance step S from the base MV (currently up to 8 distances are possible in the case of specification of table dist[] with 8 elements {1 / 4-pel,1 / 2-pel,1-pel,2-pel,4-pel,8-pel,16-pel,32-pel})

[0101] When the MMVD mode is applied, the MV difference is calculated as

[0102] refinementMV=dir[mmvd_dir_idx]*dist[mmvd_dist_idx]

[0103] Even if the CU is decoded in bidirectional prediction, a single MV difference is signaled. In case of bidirectional prediction, two symmetric MV differences are obtained from a single decoded MV. When the temporal distance between the predicted picture and the reference picture differs between reference picture lists L0 and L1, the decoded mmvd is assigned to the MV difference (mvd) associated with the maximum temporal distance. The mvd associated with the minimum distance is scaled as a function of the POC distance.

[0104] For example, consider the case where reference picture k (k=0 or 1) is the closest to the current picture. Let us define kk=1-k, POCref_0, POCref_1 and POCcur as the picture order counts of reference picture 0, reference picture 1 and the current picture respectively. The scaling factor is derived as follows:

[0105] sc=(POCref_kk–POCcur) / (POCref_k–POCcur)

[0106] Then, the refined MV for each reference image is derived as follows:

[0107] refinementMV_k=refinementMV

[0108] refinementMV_kk=sc*refinementMV

[0109] In VVC draft 3, MMVD is only applied to the translational motion model.

[0110] Symmetrical MVD proposed in JVET-L0370

[0111] The Symmetric MVD (SMVD) tool is being considered for VVC. The principle is to encode some motion vector information under the constraint that the motion information of the CU consists of two symmetric forward and backward motion vector differences in the case of bidirectional prediction. The SMVD mode is only applied to AMVP in VVC draft 3.

[0112] The decoding of a CU under this constraint (called SMVD mode) is signaled by the CU level flag symmetrical_mvd_flag.

[0113] This flag is coded if SMVD mode is feasible, that is, if the prediction mode of the CU is bi-prediction and the two reference pictures of the CU are found as follows:

[0114] – Search the reference pictures of the current CU as the closest forward and backward reference pictures in the (L0 and L1) or (L1 and L0) reference picture lists, respectively. If not found, the SMVD mode is not applicable and symmetrical_mvd_flag is omitted.

[0115] If symmetrical_mvd_flag is signaled and equal to true, then:

[0116] – Signal one mvd for the L0 reference picture, and as a symmetry, derive the mvd for another reference picture list, i.e. the opposite of the first one.

[0117] – As in classic AMVP mode, 2 MV predictor indices are signaled (one for each reference picture list).

[0118] Motion models supported in VVC draft 3 or proposed for VVC

[0119] Translational Model (VVC Draft 3)

[0120] By default, motion in a CU is based on a translation MV applied to all samples in the block.

[0121] ATMVP (VVC Draft 3)

[0122] exist Figure 8 In the Alternative (Advanced) Temporal Motion Vector Prediction (ATMVP) method shown, one or several temporal motion vector prediction values ​​for the current CU are retrieved from its reference picture(s).

[0123] First, a so-called temporal motion vector and an associated reference picture index are obtained as motion data associated with the first candidate in the normal merge candidate list of the current CU.

[0124] Next, the current CU is divided into N×N sub-CUs, where N is usually equal to 4. Figure 8 As shown, for each N×N sub-block (sub-CU), the motion vector (one or more) and the reference picture index (one or more) are identified with the help of the temporal motion vector in the reference picture associated with the temporal MV. Consider the NxN sub-block in the reference picture pointed to by the temporal MV from the current sub-CU position. Its motion data is used as the ATMVP motion data prediction of the current sub-CU. Then it is converted into the motion vector and reference picture index of the current sub-CU through appropriate motion vector scaling.

[0125] Note that in VVC draft 3, the ATMVP motion information predictor is part of the sub-block based merge candidate list.

[0126] Affine motion model (VVC draft 3)

[0127] One of the new motion models introduced in VVC is the affine mode, which basically consists in using an affine motion field to represent the motion vectors in a CU.

[0128] Fig. 9 The used motion model for 2 or 3 control point motion vectors (hereinafter also referred to as control points) is shown. The 4-parameter affine motion field based on 2 control points includes the following motion vector component values ​​for each position (x, y) within the considered block:

[0129]

[0130] Equation 1: Affine model for generating motion fields within a CU for prediction

[0131] Where (v 0x ,v 0y ) and (v 1x ,v 1y ) are the so-called control point motion vectors (CPMVs) used to generate the affine motion field. (v 0x ,v 0y ) is the motion vector of the upper left control point. (v 1x ,v 1y ) is the upper right control point motion vector. A 6-parameter affine field generated from 3 CPMVs is also possible as described in JVET-J0021.

[0132] In practice, to keep the complexity reasonable, affine motion is managed on a 4×4 sub-block basis, i.e., the same motion vector is used for each sample in each 4×4 sub-block (sub-CU) of the considered CU (see Fig.10 ). At the center position of each sub-block, an affine motion vector is calculated according to the CPMV. The obtained MV is represented with 1 / 16-pel accuracy.

[0133] As a result, the entire block (CU) is temporally predicted by motion compensating each 4×4 sub-block (sub-CU) with its own MV. Affine motion compensation can be used in VTM in 2 ways: Affine Inter (AF_INTER) and Affine Merge (or Merge-Affine), as described below.

[0134] - Affine Interval (AF_INTER)

[0135] CUs in AMVP mode, whose size is greater than 8x8, can be predicted in affine inter mode. This is signaled by the flag inter_affine_flag decoded at the CU level. The generation of the affine motion field for the inter CU includes: determining the control point motion vector (CPMV), which is obtained by the decoder by the addition of motion vector difference plus control point motion vector prediction (CPMVP). The CPMVP is a pair (for 4-parameter affine model with 2CPMV) or three (for 6-parameter affine model with 3CPMV) motion vector candidates, which can be inherited from affine neighbors (as in affine merge mode) or taken from Fig.11 The non-affine motion vectors of the lists (A, B, C) and (D, E) and / or (F, G) described above are constructed. This corresponds to the "virtual candidates" mentioned in Table 1.

[0136] - Affine Merge

[0137] In affine merge mode, the CU level flag indicates whether the CU in merge mode uses affine motion compensation. If so, then in JEM (JVC's exploratory reference software developed by JVET before), Fig.12 The first available neighboring CU that has been decoded in affine mode is selected from the ordered list of candidate positions (A, B, C, D, E).

[0138] Once the first neighboring CU in affine mode is obtained, the three motion vectors from the top left, top right and bottom left corners of the neighboring CU are retrieved and (See Fig.13 ). Based on these three vectors, two or three CPMVs of the upper left corner, upper right corner and / or lower left corner of the current CU are derived as follows:

[0139] o

[0140] o

[0141] o

[0142] Equation 2: Derivation of the CPMV of the current CU based on the three corner motion vectors of the neighboring CU

[0143] When the control point motion vector of the current CU is obtained and / or When , the motion field within the current CU is calculated based on the 4×4 sub-CU through the model of Equation 1.

[0144] In parallel application, more candidates for affine merge mode are considered.At the encoder, the best candidate is then selected by a rate-distortion optimization process, and the index of this best candidate is coded in the bitstream by the merge_idx syntax element.

[0145] In contrast to the inherited candidate, the next affine candidate put into the affine merge candidate list is a "constructed" (or "virtual") affine model candidate. The constructed candidate is an affine motion field that is calculated based on the available neighboring motion vectors around the current CU (including motion vectors from neighboring CUs that are not coded in affine mode). 2 or 3 neighboring MVs are derived and used to generate the candidate affine motion field for the current CU.

[0146] The neighboring MVs used to construct affine merge candidates include several spatial MVPs and one temporal MVP.

[0147] Planar motion model (proposed in JVET-L0070)

[0148] In JVET-L0070, Planar Motion Vector Prediction (PMVP) is proposed as an additional merge mode for VVC codec design. Fig.14 The planar motion vector prediction shown is implemented by averaging horizontal and vertical linear interpolations on a 4×4 block basis as follows. These linear interpolations are performed from generators MV located at the corners (AR, AL, BL) of the CU. The principle is Fig.14 Shown in.

[0149] P(x,y)=(H×P h (x,y)+W×P v (x,y)+H×W) / (2×H×W)

[0150] Regression MVF (RMVF) Model

[0151] In JVET-L0171, regression-based motion vector prediction is being proposed as an additional merge mode for VVC codec design.

[0152] Fig.15 The shown regression-based motion vector field (RMVF) includes the following: The principle of RMVF is to use a 6-parameter motion model to calculate the motion vector of a sub-block.

[0153]

[0154] Based on a column and a row of spatially adjacent 4×4 sub-blocks, motion parameters are calculated using their motion vectors and center positions as input to a linear regression method.

[0155] Motion model based on triangle partition

[0156] The triangular motion partitioning tool was adopted in VVC draft 3. It allows flexibility in partitioning blocks into 2 prediction units, such as Fig.16 As shown. Basically, the CU is divided into 2 triangular prediction units in the diagonal or counter-diagonal direction. Each triangular PU is inter-predicted by using its own unidirectional prediction motion vector and reference picture derived from a dedicated unidirectional prediction candidate list.

[0157] After predicting the triangle prediction unit, an adaptive weighting process is performed on the diagonal edges (see Fig.17 ). Then, the transform and quantization process is applied to the entire CU. Triangle PU mode is only applied to skip mode and merge mode.

[0158] Local Illumination Compensation (LIC)

[0159] The purpose of LIC is to compensate for illumination changes that may occur between a predicted block and its reference block used for temporal prediction by motion compensation (MC). In this tool, the decoder calculates some prediction parameters ( Fig.18 a).

[0160] In another method, the prediction samples of the adjacent left / upper reconstructed blocks can be used ( Fig.18 b).

[0161] In the case of bidirectional prediction, a variant of LIC ("bi-dir-IC") consists in estimating the illumination variation between the two current reference blocks in order to derive IC parameters for the current block.

[0162] The LIC parameters are chosen so as to minimize the mean square error difference (MSE) between the samples in Vcur and the corrected samples in Vref (MV).In general, the LIC model is linear: LIC(x) = a.x + b.

[0163]

[0164] s and r correspond to Fig.19 The pixel positions in Vcur and Vref (MV) are shown.

[0165] As a result, when LIC is used for temporal prediction, a linear LIC model is applied to the motion compensated block to provide a temporal prediction of the current block.

[0166] Generalized Bidirectional Inference (GBI)

[0167] GBI is adopted in VVC draft 3. It applies unequal weights to prediction values ​​from L0 and L1 in bi-prediction mode. In inter prediction mode, multiple weight pairs including equal weight pairs (1 / 2, 1 / 2) are evaluated based on rate distortion optimization (RDO), and the GBI index of the selected weight pair is signaled to the decoder.

[0168] In AMVP mode, a GBI index carrying GBI weight information is signaled at CU level.

[0169] In merge mode, the GBI index is inherited from the neighboring CU. The block predicted in GBI mode is calculated as follows:

[0170] P GBi =(w0*P L0 +w1*P L1 )

[0171] Where w0 and w1 are the selected GBI weights. The supported w1 values ​​are typically {-1 / 4, 3 / 8, 1 / 2, 5 / 8, 5 / 4}. The sum of w1 and w0 equals 1, then the corresponding w0 values ​​are {5 / 4, 5 / 8, 1 / 2, 3 / 8, -1 / 4}. The weight pair is selected and signaled at the CU level.

[0172] For non-low latency images, the number of weights is reduced. The values ​​of w1 and w0 are {3 / 8, 1 / 2, 5 / 8} and {5 / 8, 1 / 2, 3 / 8} respectively.

[0173] 1.1.1 Non-sub-block spatiotemporal merged motion vector prediction value (STMVP)

[0174] This section describes a prior art method JVET-L0354 to generate a spatio-temporal merge candidate called STMVP. The method basically consists in retrieving the two top and left spatial neighboring motion vectors of the current PU, and the temporal motion vector predictor of the current CU.

[0175] Spatial neighbors are taken from Fig. 20 The spatial position shown in is called Afar. The spatial position of Afar relative to the top left position of the current PU is given by the coordinates (nbPW*2,-1). Here nbPW is the width of the current block. If no motion vector is available at position Afar (not present or intra-coded), then position B1 is considered to be the top neighboring motion vector of the current block.

[0176] The selection of the left neighboring block is similar. If available, the neighboring motion vector at the relative spatial position (-1, 2*nbPH) is selected, denoted as Lfar. nbPH is the height of the current block. If not available, the left neighboring motion vector at position A1 (-1, nbPH-1) is selected if it is available.

[0177] Next, the TMVP prediction value in the current block is derived as in the temporal motion vector prediction of HEVC.

[0178] Finally, the STMVP merge candidate for the current block is calculated as the average of up to 3 obtained spatial and temporal neighboring motion vectors. Thus, in contrast to the sub-block-based approach of STMVP in JEM, here the STMVP candidate consists of at most one motion vector per reference picture list. For example, some variations of the approach of JVET-L0354 are proposed in JVET-L0207.

[0179] Suggested embodiments for extending the use of MMVD

[0180] Combination of MMVD and bidirectional prediction

[0181] In the current MMVD design, even in the case of bidirectional prediction, a separate MMVD data set consisting of a direction index and a distance index is signaled.

[0182] Embodiment 1 - Inter-frame correlation signaling of mmvd data for two reference pictures

[0183] In an embodiment, when a CU is encoded in bi-directional prediction (bipred) mode and MMVD mode is enabled for the CU, two sets of mmvd data are signaled, one set for each of the 2MMVDs, applied to each of the reference pictures (from lists L0 and L1). In addition, the second set of MMVD data can be derived from the first set of MMVD data, with various possible options.

[0184] This is Fig.21 1 is shown in a simplified block diagram of . In step 1101, mmvd data is decoded for a first reference picture. If bidirectional prediction applies (check of step 1102), conditionally with respect to the mmvd data of the first reference picture, the mmvd data of the second reference picture is decoded (step 1103). Then in step 1104, a bidirectional prediction MV is derived from the mmvd data of the first and second reference pictures. If bidirectional prediction is not applied (check of step 1102), a unidirectional prediction MV is derived from the mmvd data of the first reference picture in step 1105.

[0185] Example 1 - Option 1 - Estimation direction of mmvd of the second reference picture

[0186] In one option, the distance index is decoded only for two mmvds (mmvd0_dist_idx and mmvd1_dist_idx). The direction of the second mmvd is derived from the direction of the first mmvd (mmvd0_dir_idx). If both reference pictures are located on the same temporal side with respect to the current picture, mmvd1_dir_idx is set equal to mmvd0_dir_idx. If the reference pictures are located on both temporal sides of the current picture, mmvd1_dir_idx is set equal to -mmvd0_dir_idx.

[0187] Examples of related simplified syntax are described below.

[0188]

[0189] Example 1 - Option 2 - Differential decoding of the distance of mmvd of the second reference picture

[0190] In another option, the distance of the second reference picture is coded as the difference of the distance to the first reference picture.An index mmvdd1_dist_idx corresponding to the difference to mmvdd1_dist_idx is coded.

[0191] Examples of related simplified syntax are described below.

[0192]

[0193] The distance to refine MV0 is calculated as:

[0194] distMV0=dist[mmvd0_dist_idx]

[0195] And the distance of refined MV1 is calculated as:

[0196] distMV1=distMV0+dist[mmvdd1_dist_idx]

[0197] Embodiment 1 - Option 3 - Restriction of possible distance values ​​of mmvd of the second reference picture

[0198] In one embodiment, the maximum value of the second distance (either the distance signaled by mmvd1_dist_idx, or the distance difference signaled by mmvdd1_dist_idx as in option 2) is reduced compared to a value in mmvd0_dist_idx. Let us consider that for mmvd0_dist_idx, N possible distances are used (e.g. 8, dist[] = {1 / 4-pel, 1 / 2-pel, 1-pel, 2-pel, 4-pel, 8-pel, 16-pel, 32-pel}). For mmvdd1_dist_idx, only the first N' values ​​of dist[] can be used, where N' <N。

[0199] Consider the following variations:

[0200] – N' is calculated from N (e.g., N' = N / 2)

[0201] – N’ is calculated from the value of mmvd0_dist_idx (eg, N’ = mmvd0_dist_idx or N’

[0202] =mmvd0_dist_idx / 2)

[0203] - N' depends on the value of the scaling parameter "sc" (as described in the MMVD decoding tool description section). For example, N' = sc * N

[0204] MMVD-CPR Combo

[0205] In one embodiment, when CPR (Current Picture Reference) mode is applied, MMVD mode is enabled with CPR specific adaptation.

[0206] The MV for CPR refers to the current picture. In VVC draft 3, restrictions are specified to limit the MV to point to a restricted area close to the current CU in order to limit the memory storage requirements. In the current VTM, the restricted area consists of the CTU containing the current CU, such as Fig. 22 Another limitation of CPR MV is that it has integer precision.

[0207] Example 2a - mmvd Limitation on Integer Precision of CPR

[0208] In one embodiment, the enabled distances of mmvd are limited to those distances compatible with the enabled MV precision of CPR. For example, in the current VTM, the CPR MV precision is an integer, and the present invention takes into account that when the CPR mode is used, the distances of mmvd are in the following set:

[0209] {1-pel,2-pel,4-pel,8-pel,16-pel,32-pel}

[0210] Instead of the full set that mmvd usually uses:

[0211] dist[]={1 / 4-pel,1 / 2-pel,1-pel,2-pel,4-pel,8-pel,16-pel,32-pel}

[0212] In the case of CPR, the maximum value N' (eg, 5) of mmvd_dist_idx is smaller than the maximum value N (eg, 8) of mmvd_dist_idx.

[0213] In an embodiment, this solution is achieved using an offset applied to mmvd_dist_idx as follows:

[0214] - If MMVD mode is applied

[0215] oIf CPR mode is applied

[0216] ■distMV=dist[mmvd_dist_idx+offset]

[0217] Otherwise

[0218] ■distMV=dist[mmvd_dist_idx]

[0219] To limit integer precision, offset is equal to 2.

[0220] Example 2b - Limitation of the maximum mmvd value for CPR

[0221] In an embodiment, the maximum distance of mmvd is reduced when CPR mode is activated compared to conventional mmvd use. For example, in current VTM, the CPR MV accuracy is an integer, and the present invention takes into account that the distance of mmvd when CPR mode is activated is in the following set:

[0222] {1-pel, 2-pel, 4-pel, 8-pel}

[0223] Instead of the full set that mmvd usually uses:

[0224] dist[]={1 / 4-pel,1 / 2-pel,1-pel,2-pel,4-pel,8-pel,16-pel,32-pel}

[0225] In the case of CPR, the maximum value N' (eg, 3) of mmvd_dist_idx is smaller than the maximum value N (eg, 8) of mmvd_dist_idx.

[0226] Example 2c-, Trimming MV from mmvd for CPR

[0227] In one embodiment, the MV obtained from mmvd refinement is clipped so that the motion compensated block used for prediction stays within the restricted area. Fig.23 Shown in.

[0228] Example 3: Combination of MMVD and ATMVP

[0229] Example 3.1 - MMVD combined with sub-block based mode ATMVP

[0230] In VVC draft 3, MMVD is only applied to the normal translation merge mode. Therefore, when the sub-block based merge mode is activated for a given CU, MMVD is not used.

[0231] According to this embodiment 3, when the sub-block merge mode is activated, the MMVD motion vector representation tool can be used. In particular, the MMVD can be used in combination with the ATMVP motion vector prediction mode.

[0232] When the merge_flag syntax indicates that the current CU uses merge mode, the use of MMVD is signaled in the same way as in VVC draft 3. In addition, the flag subblock_merge_flag indicating the use of subblock-based merge mode is coded regardless of the value of the syntax element mmvd_merge_flag indicating that MMVD is used for the current block.

[0233] According to the first sub-embodiment 3.0, affine mode is not allowed in combination with MMVD. Therefore, if mmvd_merge_flag is on, it is inferred that the merge mode used is ATMVP mode. Therefore, the merge_idx syntax element is omitted from the decoded bitstream.

[0234] According to another sub-embodiment, affine mode is allowed in combination with MMVD.Therefore, the merge_idx syntax element is decoded as currently done in VVC draft 3.

[0235] Example 3.2 - mmvd is used as a refinement of the inherited sub-block MV of ATMVP

[0236] In one embodiment, when an ATMVP candidate is selected and MMVD is enabled, mmvd refinement is applied to all MVs inherited for each sub-block. Fig.24 . In the first step, the mmv data is decoded. In the next step, the global ATMVP MV is identified. Then, for each sub-block of the CU, the motion data from the corresponding sub-block identified by the global MV in the reference picture is extracted. These motion data are refined using mmvd data, similar to the traditional MMVD refinement process.

[0237] Example 4: Combination of MMVD and Affine Mode

[0238] According to one embodiment, MMVD is used in combination with an affine mode. Several variations can be applied in this embodiment. They are listed below.

[0239] A single mmvd for all CPMV decoding

[0240] According to a first variant, a single motion vector difference mmvd is decoded and applied to all CPMVs used to generate the affine motion field. This results in a constant motion vector being added to the entire affine motion model. This is illustrated in the figure of the simplified decoding block diagram of the affine motion field generation process when the MMVD mode is activated described below.

[0241] o In the case of bidirectional affine prediction, the same concept of symmetric motion vector differences as in the translation case is used.

[0242] mmvd data decoded for each CPMV

[0243] –According to a second variant, one motion vector difference mmvd is allowed per CPMV, which increases the flexibility of generating candidate affine motion fields. Conventional MMVD encoding of MMVD can be used for the first CPMV. Next, for the second and third CPMVs, the differential motion vector differences (mmvdd) can be decoded on top of the mmvd associated with the first CPMV. The advantage of this approach is that it limits the rate overhead of mmvd signaling while allowing the MMVD application some flexibility for affine modes. An example of the relevant simplified syntax is described below in the case of decoding the mmvd information of 3 affine CPMVs. According to this embodiment, the magnitude of the differential motion vector difference (one or more) can be limited to a value less than the mmvd of the first CPMV (for example, limited to dist[] = {1 / 4-pel, 1 / 2-pel, 1-pel}, while the full set of values ​​can be dist[] =

[0244] {1 / 4-pel, 1 / 2-pel, 1-pel, 2-pel, 4-pel, 8-pel, 16-pel, 32-pel}). The process is illustrated in the figure below which describes a simplified decoding block diagram of the affine motion field generation process when MMVD mode is activated.

[0245]

[0246] - According to some variants, in the decoding of the differential mmvd, the range of allowed magnitudes is restricted according to the distance index associated with the mmvd of the first CPMV. For example, mmvd1_dist_idx and, when applicable, mmvd2_dist_idx are restricted to be lower than mmvd0_dist_idx.

[0247] like Fig. 27 As shown exemplarily, the allowable range of the second mmvd in the horizontal and vertical directions may be limited.

[0248] - According to another variant, the same direction index is used for all mmvds, so only the distance information can be decoded for the second and third CPMVs. An example of the relevant simplified syntax is described below in the case of decoding the mmvd information of 3 affine CPMVs.

[0249]

[0250] Limit the list of affine merge candidates by removing dummy candidates

[0251] - According to another variant, since the MMVD mode for affine brings some diversity in the set of affine merge candidates, the affine list of affine merge candidates is reduced compared to the affine merge list of VVC draft 3, resulting in a simplified overall codec design. For example, in an embodiment, some constructed (virtual) affine model candidates are suppressed from the affine merge list. In an embodiment, all constructed (virtual) affine model candidates are suppressed from the affine merge list, and the affine merge list consists only of inherited affine candidates.

[0252] Constrain the distance value of the 2nd (and 3rd) mmvd based on the distance value of the 1st mmvd

[0253] – According to an additional feature, for the second CPMV and the third CPMV (when applicable), the differential decoding (motion vector difference) of the mmvd relative to the mmvd of the first CPMV includes the possibility to decode / decode the distance value 0 of the differential motion vector difference (dist[] = {0-pel, 1 / 4-pel, ...}). In fact, in existing MMVD applied only to the case of translational motion, a distance equal to 0 is not supported because it would provide MV prediction values ​​that are redundant with the usual merge prediction.

[0254] In the case of bidirectional affine, symmetry constraints are imposed on the affine model parameters

[0255] According to another feature, in the bi-directional affine case, the derivation of the second mmvd (attached to the second reference picture) as a function of the first mmvd (attached to the first reference picture) uses a symmetry constraint similar to the symmetry constraint between the two bi-directional prediction mmvds imposed in the translation case. To this end, several approaches are possible.

[0256] o In a first approach, the mmvd associated with the second reference picture list is derived directly from the mmvd associated with the first reference picture list. For example, the second mmvd is derived by scaling the first mmvd, the scaling taking into account the temporal distance between the reference picture and the current picture.

[0257] o In another approach, the MVD of the affine model associated with the second reference picture list is inferred from the MVD of the affine model associated with the first reference picture, by imposing that the affine model parameters (which are typically linked to angles and scaling factors) are symmetric compared to the MVD of the affine model associated with the first reference picture. To this end, the scaling factor and angle values ​​associated with the first affine model are calculated. This is then transformed to satisfy the symmetry constraints, and then the affine model parameters are inversely transformed to provide a second affine model of the current CU for bidirectional affine prediction of the CU. This takes the following form.

[0258] A 4-parameter model is assumed for the considered blocks:

[0259]

[0260] Then, get the rotation angle and scaling parameters as follows:

[0261]

[0262] Then, the angle and scaling factor are transformed as follows:

[0263]

[0264] where k is a scaling factor that depends on the temporal distance between the current picture and its two reference pictures.Finally, from a' and b', the affine model parameters a' and b' of the second reference picture are easily calculated.

[0265] In a simplified version, only the scaling factor is modified, resulting in a simple scaling of the MVd.

[0266] Example 5: Combination of MMVD and planar motion vector prediction

[0267] According to embodiment 5, MMVD is used in combination with the planar motion vector prediction (PMVP) mode.

[0268] This can take one of the following forms.

[0269] A single mmvd that refines each MV of the PMVP playground

[0270] - According to the first basic method, in case PMVP mode is used for the current CU, the MMVD motion vector difference is decoded. The mmvd is applied to the motion field generated by the PMVP, which is originally generated from its generator MV. Therefore, mmvd is used as an additional offset on each MV for the PMVP motion field. This is shown in the figure below which describes a simplified decoding block diagram of the PMVD motion generation process.

[0271] A single mmvd that refines at least one MV used to generate the PMVD motion field

[0272] According to another variant, a motion vector difference is applied to at least one motion vector (generator MV) used to generate a planar motion field. For example, a single MVD may be decoded and applied to the Fig.28 This is shown in the figure below which describes a simplified decoding block diagram of the PMVD motion generation process.

[0273] - According to another variant, the MVD is decoded and applied to the BR motion vectors before generating the planar MV field.

[0274] - According to another variant, the MVD is decoded and applied to the AR motion vectors before generating the planar MV field.

[0275] - According to another variant, the MVD is decoded and applied to the BL motion vectors before generating the planar MV field.

[0276] Several mmvds, which are used to refine several MVs used to generate the PMVD motion field

[0277] - According to another embodiment, multiple MVDs are decoded and applied to one or more of the AL, BL, AR and BR motion vectors before generating a planar motion field. This is shown in the following figure which describes a simplified decoding block diagram of the PMVD motion generation process.

[0278] According to another embodiment, when several MVDs are decoded and applied to several MVs in AL, BL, AR, BR motion vectors, the first decoded MVD is decoded as in MMVD. The next MVD is decoded in a differential manner based on the previous decoded MVD to limit the rate cost associated with the MVD decoding process.

[0279] - According to one variant, the set of allowed distances for MVD in the planar case is modified compared to the MMVD tool currently used in VVC draft 3, e.g. a restricted range of MVD distances is allowed.

[0280] - According to a variant, the number of allowed MVD orientations is modified compared to the existing MMVD system in VVC draft 3, e.g. an enriched set of MVD angles may be supported when MMVD is used in combination with planar MV prediction.

[0281] Example 6: Combination of MMVD and regression-based motion vector field

[0282] According to Example 5, MMVD is used in combination with the regression-based 6-parameter motion field introduced in the part of the regression MVF model.

[0283] This can take one of the following forms.

[0284] Perform a separate mmvd refinement for each MV of the RMVF motion field

[0285] – According to the first basic method, in case RMVF mode is used for the current CU, the MMVD motion vector difference is decoded and applied to the motion field generated by RMVF. Therefore, MVd is used as an additional offset on the RMVF motion field. Fig.29 A similar block diagram as shown in may suitably describe a simplified process for generating an RMVF motion field according to this variation.

[0286] A single mmvd that is used to refine at least one of the MVs used to generate the RMVF motion field

[0287] According to another variant, a motion vector difference is applied to at least one motion vector used to generate the RMVF motion field. For example, a single MVD may be decoded and applied to the motion vector to the left of the current block before generating the RMVF motion field.

[0288] - According to another variant, before generating the regression-based MV field, an MVD is decoded and applied to the top neighboring motion vectors. Fig.30 A similar block diagram as shown in may suitably describe a simplified process for generating an RMVF motion field according to this variation.

[0289] Two mmvds, which are used to refine the two MVs used to generate the RMVF motion field

[0290] - According to another embodiment, before generating the regression-based motion field, two MVDs are decoded and applied to the top and left MVs respectively.

[0291] According to another embodiment, when two MVDs are decoded and applied as described above, the first decoded MVD is decoded as in MMVD. Based on the first decoded MVD, the next MVD is decoded in a differential manner to limit the rate cost associated with the MVD decoding process.

[0292] - According to a variant, the set of allowed distances for MVD based on regression is modified compared to the MMVD tool currently used in VVC draft 3. For example, a restricted range of allowed MVD distances is allowed.

[0293] - According to one variant, the number of allowed MVD orientations is modified compared to the existing MMVD system in VVC Draft 3, e.g., an enriched set of MVD angles may be supported when MMVD is used in combination with RMVF.

[0294] -like Fig.31 A similar block diagram as shown in may suitably describe a simplified process for generating an RMVF motion field according to those variations.

[0295] Example 7: Combination of MMVD and LIC

[0296] In one embodiment, MMVD and LIC can be enabled together. In this case, the decoded motion vector difference is applied to the motion vector of the considered CU, as in VVC draft 3. Therefore, the LIC linear model parameters associated with the temporal prediction of the current CU are derived from the merge candidate used, as in the current LIC tool.

[0297] According to a variation, if the MVD magnitude is above a certain threshold, the LIC flag of the current CU may be set to false, i.e., LIC temporal prediction refinement may be disabled for the current CU. In fact, intuitively, if the motion vector of the current CU is significantly different from the motion vector of the merge candidate used to derive the MV of the current CU, the LIC linear model used for the merge candidate CU may be irrelevant to the current CU.

[0298] For example, if the MVD magnitude is higher than or equal to 16 or 32, the LIC mode may be forced to 0 for the current CU.

[0299] According to another variant, the above threshold on the MVD magnitude may depend on the current CU size.

[0300] Example 8: Combination of MMVD and GBI

[0301] In VVC draft 3, when a CU is coded in translation merge mode, its motion vector is derived from the selected merge candidate and from its GBI index. This means that the GBI weight of the CU used as a reference for deriving the current CU motion data is also used for the current CU without change, including when MMVD is activated for the current CU.

[0302] In this current embodiment, adaptation of the GBI weight of the current CU may be applied based on the magnitude of the MVD for the current merged CU.

[0303] For example, if the MVD magnitude is above a certain threshold, the GBI may be reset to the default GBI weight (1 / 2, 1 / 2) for the current CU. In fact, intuitively, if the motion vector of the current CU is significantly different from the motion vector of the merge candidate used to derive the MV of the current CU, the GBI weight for the merge candidate CU may be irrelevant to the current CU.

[0304] For example, if the MVD magnitude is higher than or equal to 16 or 32, the GBI weight may be forced to (1 / 2, 1 / 2) for the current CU.

[0305] According to another variant, the above threshold on the MVD magnitude may depend on the current CU size.

[0306] Example 9: MMVD and triangular motion partitioning

[0307] According to one embodiment, the MMVD motion vector coding tool is used in conjunction with the triangle motion partitioning tool.

[0308] According to a first variant, following the VVC draft 3 MVD coding system of MMVD, one MVD is coded for each triangle partition.

[0309] According to another variant, a single MVD is coded and used jointly by two triangular motion partitions.

[0310] According to another variant, when two MVDs are decoded, the second MVD is decoded differently from the first MVD. In this case, the second MVD can be restricted to a smaller allowed value range in the same way as in the affine case (see section 4 of embodiment).

[0311] According to a more advanced embodiment, if the MVDs are decoded, one for each partition, the second MVD is decoded in a conditional manner based on the relative values ​​of the motion vectors of the first and second partitions. For example, the second MVD may be constrained so that the second refined MV is not close to the refined MV of the first triangular partition. In fact, if it is close, then the overall prediction of the current CU may behave very close to the usual translational motion compensated prediction of the entire rectangular block.

[0312] Example 10: MMVD and Multiple Hypothesis (MH) Prediction

[0313] In current video standards, such as VVC draft 3, a new prediction mode called multi-hypothesis consists in merging / skipping the combined prediction of temporal prediction blocks and intra prediction blocks.

[0314] However, in VVC draft 3, MMVD and MH prediction cannot be used together.

[0315] In this embodiment, it is allowed to use MMVD and multi-hypothesis prediction mode in combination.

[0316] Basically, this consists in applying the merge motion vector difference to the merge or skip candidate motion vector used for deriving the motion information of the current coding unit.

[0317] The advantage of this embodiment is that the decoding efficiency is further improved.

[0318] Example 11: MMVD and Spatiotemporal Motion Vector Prediction (STMVP)

[0319] In VVC draft 3, a motion vector prediction mode called STMVP is proposed, as described in the section about non-sub-block spatiotemporal merged motion vector prediction values. The STMVP motion vector candidate is an additional merge candidate, which can usually be part of the translation merge candidate list. It is to predict a single motion vector of the current CU. Therefore, MMVD can be applied to the STMVP candidate in a direct way.

[0320] However, MMVD can also be applied to only one of the three spatial and temporal MV prediction values ​​used to calculate the STMVP candidate.

[0321] According to an embodiment, the motion vector difference of MMVD is applied to one or both spatial motion vector predictors, which occurs before calculating the average motion vector between the two spatial motion vector predictors and the temporal motion vector predictor.

[0322] According to an embodiment, the motion vector difference of the MMVD is applied to the temporal motion vector predictor before calculating the average motion vector between this temporal MV predictor and one or two spatial motion vector predictors.

[0323] Suggested implementation of extended use of SMVD

[0324] Example 12: MMVD replaces SMVD in AMVP mode

[0325] As described in the symmetric MVC section, the SMVD motion vector decoding mode is only applicable to the AMVP mode, and the MMVD motion vector representation mode is only applicable to the merge mode. In this embodiment, the codec design is coordinated. For the entire design, only one motion vector decoding mode in MMVD and SMVD is proposed to handle the case of symmetric bidirectional motion and the case of low-magnitude motion vectors in the merge context.

[0326] The following two variations are proposed.

[0327] Use mmvd syntax to decode MV differences in MMVD and SMVD modes

[0328] - In AMVP mode, when symmetric mode is on, the motion vector difference is coded using the MMVD MVD coding syntax described in the MMVD motion vector difference coding tool description section. Thus, if the symmetric MVD mode is valid for the considered CU, the traditional MVD coding method of VVC draft 3 in AMVP is replaced.

[0329] Decoding MV differences in MMVD and SMVD modes using VTM3 MVd syntax

[0330] - In AMVP mode, when symmetric mode is on, the motion vector differences are coded as currently done in VVC draft 3 in AMVP. However, if MMVD mode is on in the merge case, the motion vector differences of MMVD are coded in AMVP fashion.

[0331] Example 13: SMVD combined with affine motion model

[0332] According to one embodiment, SMVD usage is extended to the case of affine AMVP. This may take the following form.

[0333] The first CPMV of the considered affine CU is decoded according to the SMVD mode of the part about the symmetric MVD, as in the case of the classic translational AMVP. Next, the MVDs of the other CPMVs of the considered affine CU are differentially decoded on the basis of the MVD of the first CPMV, as is currently done in the case of affine AMVP. The corresponding simplified block diagram is described in the following figure.

[0334] According to another variant, the differential MVD of the second and optional third CPMV is differentially decoded on the basis of the MVD of the first CPMV, but also differentially decoded under the symmetric mode constraint. This makes it possible to reduce the rate cost of the second and third CPMV.

[0335] Example 14: SMVD combined with triangular motion partitioning

[0336] If triangle partitioning is used in AMVP, then the symmetric MVD pattern can be extended to the triangle partitioning case.

[0337] In this case, in one embodiment, the two unidirectional MVDs of the first and second triangular partitions may be symmetrical to each other.

[0338] Example 15: SMVD combined with multiple hypothesis prediction modes

[0339] In one embodiment, in addition to the merge mode, multiple hypotheses can also be used in AMVP mode. In this case, SMVD mode can be used in conjunction with the multiple hypothesis prediction mode, which can occur when the inter-frame component of this inter-frame / intra-frame composite prediction uses bidirectional prediction.

[0340] Example 16: SMVD combined with planar motion model

[0341] In an embodiment, in addition to the current use in merge mode, the planar motion model can be used in AMVP mode. In the case, the motion difference can be used and applied to the motion vectors around the current CU and used to generate the motion field of the current CU.

[0342] In addition, the SMVD mode can be used in conjunction with this AMVP planar motion model case. In this case, some symmetric bidirectional motion vector differences can be applied to the surrounding motion vectors of the current CU for generating the planar MV field of the current CU. This embodiment is similar to the embodiment covering part of embodiment 5, but in the case of AMVP+symmetric mode.

[0343] Example 17: SMVD combined with regression-based motion model

[0344] In one embodiment, in addition to the current use in merge mode, the regression-based motion model can also be used in AMVP mode. In the case, motion differences can be used, which are applied to the motion vectors around the current CU for generating the regression-based motion field of the current CU.

[0345] In addition, the SMVD mode can be used in conjunction with this AMVP regression-based motion model case. In this case, some symmetric bidirectional motion vector differences can be applied to the surrounding motion vectors of the current CU for generating the regression-based MV field of the current CU. This embodiment is similar to the embodiment covering part of embodiment 5, but in the case of AMVP+symmetric mode.

[0346] Example 18: SMVD combined with ATMVP motion model

[0347] Unique because of AMVP and merger

[0348] In one embodiment, in addition to the current use in the merge mode, the ATMVP motion model can also be used in the AMVP mode. In this case, the motion difference can be used and applied to the motion vector contained in the ATMVP predicted motion vector of the current CU.

[0349] In addition, the SMVD mode can be used in conjunction with this AMVP ATMVP motion model case. In this case, some symmetric bidirectional motion vector differences can be applied to the ATMVP motion vector derived for the current CU. This embodiment is similar to the embodiment covering part of embodiment 3, but in the case of AMVP+ATMVP mode.

[0350] Example 19: SMVD combined with spatiotemporal motion vector prediction (STMVP)

[0351] In current video standard proposals, such as VVC draft 3, STMVP motion vector prediction values ​​are only proposed in the merge case (see the section on STMVP). However, STMVP is allowed in AMVP mode. In this case, SMVD can be applied to STMVP candidates in a direct way.

[0352] However, SMVD can also be applied to only one of the three spatial and temporal MV prediction values ​​used to calculate the STMVP candidate.

[0353] According to an embodiment, the motion vector difference of the SMVD is applied to one or both spatial motion vector predictors, before calculating the average motion vector between the two spatial motion vector predictors and the temporal motion vector predictor.

[0354] According to an embodiment, the motion vector difference of the SMVD is applied to the temporal motion vector predictor, which occurs before calculating the average motion vector between the temporal MV predictor and the one or two spatial motion vector predictors.

[0355] 1.1.2 Example 20: Modified SMVD mode to enforce symmetry of the final bidirectional motion vector pair. Translation case

[0356] This section presents an embodiment in which the SMVD mode is modified so that no symmetry constraint is imposed on the motion vector difference MVd, but a symmetry constraint is imposed on the final motion vector used for bi-directional prediction of a block in the translational AMVP mode.

[0357] In fact, as explained in the section covering symmetric MVD, in SMVD, the MVD is decoded for the first motion vector of a pair of bidirectional MVs, and then the MVD for the second motion vector is derived from the first MVD as an inverse MVD vector, which may be scaled as a function of the temporal distance between the current picture and its reference picture.

[0358] Here, in the proposed embodiment, the modified SMVD mode is such that the MVD of the first vector is decoded as in the existing SMVD. Next, the first motion vector is reconstructed as the sum of the motion vector prediction value and the decoded motion vector difference. Finally, a reconstructed second motion vector is derived from the reconstructed first motion vector as the inverse MV, which may be scaled as a function of the temporal distance between the current picture and its reference picture.

[0359] Thus, the proposed new SMVD mode ensures that the prediction block is located on the same row as its two backward and forward reference blocks. This approach is expected to provide improved decoding efficiency compared to the existing SMVD covering part of the symmetric MVD.

[0360] Example 21: Modified SMVD mode to impose symmetry of the final affine model (rotation, scaling / zoom and translation)

[0361] According to an embodiment, the SMVD mode is used in combination with the affine AMVP. In this case, the symmetry constraints can be imposed on the affine MVD in a straightforward manner.

[0362] According to another approach, the symmetry constraints may be imposed on the affine model parameters (angles and scaling factors) in a manner similar to the corresponding characteristics of the portion of Example 4 with respect to the combination of MMVD and affine.

[0363] Fig.35 One embodiment of a method 3500 according to the main aspects described herein is shown. The method begins at start block 3501 and control passes to block 3510 for indicating a first motion mode by syntax in a video bitstream. Control passes from block 3510 to block 3520 to indicate the use of a second motion mode by the presence of syntax in the video bitstream and, if present, to include information related to the second motion mode. Control passes from block 3520 to block 3530 to encode a video block using motion information corresponding to the first and second motion modes.

[0364] Fig.36Another embodiment of a method 3600 under the main aspects described herein is shown. The method starts at block 3601 and control passes to block 3610 for parsing a video bitstream for syntax indicating a first motion mode. Control passes from block 3610 to block 3620 for parsing the video bitstream for syntax indicating the presence of a second motion mode and, if present, determining information related to the second motion mode. Control passes from block 3620 to block 3630 to obtain motion information corresponding to the first motion mode. Control passes from block 3630 to block 3640 to decode the block using the motion information.

[0365] Fig.37 An embodiment of an apparatus 3700 for encoding, decoding, compressing or decompressing video data using a simplification of a coding mode based on an adjacent sample dependent parameter model is shown. The apparatus includes a processor 3710 and can be interconnected to a memory 3720 through at least one port. Both the processor 3710 and the memory 3720 may also have one or more additional interconnections to external connections.

[0366] The processor 3710 is also configured to insert or receive information in a bitstream and to compress, encode, or decode using any of the described aspects.

[0367] The present application describes a number of aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described as having specificity, and at least in order to illustrate individual characteristics, are usually described in a manner that may sound limited. However, this is for the purpose of describing clearly, and does not limit the application or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide other aspects. In addition, these aspects can also be combined and interchanged with the aspects described in earlier documents.

[0368] The aspects described and contemplated in this application can be implemented in many different forms. Figure 3 , 4 and 34 provide some embodiments, but other embodiments are contemplated and are Figure 3 , 4 The discussion of 34 does not limit the breadth of implementation. At least one of the aspects is generally related to video encoding and decoding, and at least one other aspect is generally related to transmitting the generated or encoded bitstream. These and other aspects can be implemented as methods, devices, computer-readable storage media having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having stored thereon bitstreams generated according to any of the described methods.

[0369] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image", "picture" and "frame" are used interchangeably. Usually, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0370] Various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for the correct operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0371] Various methods and other aspects described in this application can be used to modify modules, e.g. Figure 3 and Figure 4 Intra-frame prediction, entropy coding, and / or decoding modules (160, 360, 145) of the video encoder 100 and the video decoder 200 shown. In addition, the present invention is not limited to VVC or HEVC, and can be applied to, for example, other standards and proposals (whether pre-existing or developed in the future) and extensions of any such standards and proposals (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this application can be used alone or in combination.

[0372] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes, and the described aspects are not limited to these specific values.

[0373] Figure 3 An encoder 100 is shown. Variations of the encoder 100 are contemplated, but for the sake of clarity, the encoder 100 is described below without describing all contemplated variations.

[0374] Before being encoded, the video sequence may undergo a pre-encoding process (101), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0375] In encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is partitioned (102) and processed in units such as CUs. Each unit is encoded using, for example, intra or inter mode. When the unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of intra mode or inter mode to use to encode the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. For example, a prediction residual is calculated by subtracting (110) the prediction block from the original image block.

[0376] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy encoded (145) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass both transform and quantization, i.e., directly encode the residual without applying the transform or quantization process.

[0377] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. An in-loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0378] Figure 4 2 shows a block diagram of a video decoder 200. In the decoder 200, a bitstream is decoded by decoder elements as described below. The video decoder 200 generally performs the same operations as described below. Figure 3 The encoding process described in 1 is a decoding process that is the inverse of the encoding process described in 1. The encoder 100 also typically performs video decoding as part of encoding the video data.

[0379] In particular, the input to the decoder comprises a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors and other encoding information. Picture partition information indicates how the picture is partitioned. The decoder may therefore partition (235) the picture according to the decoded picture partition information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined (255) with a prediction block to reconstruct the image block. The prediction block may be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0380] The decoded picture may further undergo post-decoding processing (285), such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or performing an inverse remapping of the remapping process performed in the pre-encoding process (101). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0381] Fig.34 A block diagram of an example of a system in which various aspects and embodiments are implemented is shown. System 1000 can be implemented as a device including various components described below, and is configured to perform one or more aspects described herein. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances and servers. The elements of system 1000 can be implemented in a single integrated circuit (IC), multiple ICs and / or discrete components individually or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed on multiple ICs and / or discrete components. In various embodiments, the system 1000 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or by dedicated input and / or output ports. In various embodiments, the system 1000 is configured to implement one or more aspects described herein.

[0382] The system 1000 includes at least one processor 1010, which is configured to execute instructions loaded therein, for implementing various aspects described herein, for example. The processor 1010 may include embedded memory, input and output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include a non-volatile memory and / or a volatile memory, including but not limited to an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a disk drive, and / or an optical disk drive. As a non-limiting example, the storage device 1040 may include an internal storage device, an attached storage device (including a removable storage device and a non-removable storage device) and / or a network accessible storage device.

[0383] The system 1000 includes an encoder / decoder module 1030, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module (one or more) that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding module and the decoding module. In addition, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000 or may be incorporated into the processor 1010 as a combination of hardware and software as known to those skilled in the art.

[0384] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of the various items during the execution of the processes described herein. These stored items may include, but are not limited to, input video, decoded video or portions of the decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0385] In some embodiments, memory within the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or a storage device 1040, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, an operating system for a television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as working memory for MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by Joint Video Team experts JVET).

[0386] As shown in block 1130, input to the elements of system 1000 may be provided through various input devices. Such input devices include, but are not limited to: (i) an RF portion that receives a radio frequency (RF) signal transmitted, for example, over the air by a broadcaster, (ii) a component (COMP) input terminal (or a set of component input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Fig.34 Other examples not shown include composite video.

[0387] In various embodiments, the input device of block 1130 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band limiting the frequency band to a narrower frequency band to select a signal frequency band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements to perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or baseband. In a set-top box embodiment, the RF part and its relevant input processing element receive the RF signal that sends by wired (for example, cable) medium, and carry out the frequency selection to the desired frequency band by filtering, down-conversion and filtering again.Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements that perform similar or different functions.Adding element can include and insert element between existing element, for example insert amplifier and analog-to-digital converter.In various embodiments, the RF part comprises antenna.

[0388] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Solomon error correction) may be implemented as needed, for example, in a separate input processing IC or processor 1010. Similarly, various aspects of USB or HDMI interface processing may be implemented as needed in a separate interface IC or in processor 1010. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 1010 and encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on output devices.

[0389] The various components of system 1000 may be disposed in an integrated housing in which the various components may be interconnected and transmit data using a suitable connection arrangement (e.g., an internal bus known in the art, including an inter-IC (I2C) bus, wiring, and a printed circuit board).

[0390] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data through the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0391] In various embodiments, data is streamed or otherwise provided to the system 1000 using a wireless network (e.g., a Wi-Fi network, such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)). The Wi-Fi signals of these embodiments are received through a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or router that provides access to an external network including the Internet to allow streaming applications and other cloud communications. Other embodiments use a set-top box that transmits data through an HDMI connection of an input box 1130 to provide streaming data to the system 1000. Still other embodiments use an RF connection of an input box 1130 to provide streaming data to the system 1000. As described above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use a wireless network other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0392] The system 1000 can provide output signals to various output devices (including a display 1100, a speaker 1110, and other peripheral devices 1120). The display 1100 of various embodiments includes one or more of the following: for example, a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 can be used for a television, a tablet computer, a laptop computer, a cellular phone (mobile phone), or other devices. The display 1100 can also be integrated with other components (for example, as in a smart phone), or it can be separate (for example, an external monitor for a laptop computer). In various examples of various embodiments, the other peripheral devices 1120 include one or more of the following: an independent digital video disk (or digital versatile disk) (DVR, for both), a disk player, a stereo system, and / or an illumination system. Various embodiments use one or more peripheral devices 1120 that provide functions based on the output of the system 1000. For example, a disk player performs the function of playing the output of the system 1000.

[0393] In various embodiments, control signals are transmitted between the system 1000 and the display 1100, speaker 1110, or other peripheral device 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. The output devices may be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, the output devices may be connected to the system 1000 via a communication interface 1050 using a communication channel 1060. The display 1100 and speaker 1110 may be integrated into a single unit in an electronic device (e.g., a television) along with other components of the system 1000. In various embodiments, the display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0394] For example, if the RF portion of the input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may alternatively be separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signals may be provided via dedicated output connections, including, for example, an HDMI port, a USB port, or a COMP output.

[0395] These embodiments may be implemented by the processor 1010 or by computer software implemented by hardware or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, as a non-limiting example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 1010 may be of any type suitable for the technical environment and may include, as a non-limiting example, one or more of the following: a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0396] Various implementations relate to decoding. As used in the present application, "decoding" may include, for example, all or part of the processing performed on the received coded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by the decoders of the various embodiments described in the present application.

[0397] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or to generally refer to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0398] Various implementations involve encoding. In a manner similar to the above discussion of "decoding", "encoding" as used in this application may include, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such processes also or alternatively include processes performed by an encoder of the various embodiments described in this application.

[0399] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will become clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0400] Note that the syntax elements as used herein are descriptive terms. Therefore, they do not exclude the use of other syntax element names.

[0401] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0402] Various embodiments may relate to parameter models or rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is usually considered, usually given a constraint on computational complexity. It can be measured by rate-distortion optimization (RDO) metric, or by least mean square (LMS), mean absolute error (MAE) or other such measurements. Rate-distortion optimization is usually formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different methods to solve the rate-distortion optimization problem. For example, these methods can be based on extensive testing of all coding options, including all considered modes or coding parameter values, and a complete evaluation of their coding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to save coding complexity, in particular, to calculate approximate distortion based on prediction or prediction residual signals rather than reconstructed signals. A mixture of these two methods can also be used, for example by using approximate distortion only for some possible coding options and full distortion for other coding options. Other methods only evaluate a subset of possible coding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a complete assessment of both the coding cost and the associated distortion.

[0403] The implementations and aspects described herein can be implemented in, for example, methods or processes, devices, software programs, data streams, or signals. Even if only discussed in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the features discussed can also be implemented in other forms (e.g., devices or programs). For example, the device can be implemented with appropriate hardware, software, and firmware. The method can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0404] References to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations mean that a particular feature, structure, characteristic, etc. described in conjunction with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations in various places in this application do not necessarily all refer to the same embodiment.

[0405] Additionally, the present application may refer to “determining” various information. Determining the information may include, for example, one or more of: estimating the information, calculating the information, predicting the information, or retrieving the information from a memory.

[0406] In addition, the present application may involve "accessing" various information. Accessing the information may include, for example, one or more of: receiving the information, retrieving the information (e.g., retrieving the information from a memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0407] Additionally, the present application may refer to "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving the information may include, for example, one or more of: accessing the information or retrieving the information (e.g., from a memory). Furthermore, during operations such as storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is generally involved in one way or another.

[0408] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selection of only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to include selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to multiple items listed, as will be apparent to one of ordinary skill in this and related arts.

[0409] In addition, as used herein, the word "signal" especially refers to a corresponding decoder indicating something. For example, in some embodiments, the encoder sends a signal to send a specific one of multiple transformations, decoding modes, or signs. In this way, in one embodiment, the same transformation, parameter or mode is used on the encoder side and the decoder side. Therefore, for example, the encoder can send (explicitly send with a signal) a specific parameter to the decoder so that the decoder can use the same specific parameter. On the contrary, if the decoder already has the specific parameter and other parameters, signaling can be used without sending (implicitly sending with a signal) to simply allow the decoder to know and select the specific parameter. By avoiding the transmission of any actual function, bit saving is achieved in various embodiments. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more grammatical elements, signs, etc. are used to send information to the corresponding decoder with a signal. Although the foregoing relates to the verb form of the word "signal", the word "signal" can also be used as a noun in this article.

[0410] As will be apparent to one of ordinary skill in the art, implementations may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier wave using the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0411] We have described multiple embodiments across various claim categories and types. Features of these embodiments may be provided individually or in any combination. In addition, across various claim categories and types, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination:

[0412] ●A process or apparatus using MMVD and / or SMVD with an affine motion model.

[0413] • A process or apparatus using MMVD and / or SMVD with alternative temporal motion vector prediction.

[0414] A process or apparatus for using MMVD and / or SMVD with bidirectional optical flow.

[0415] A method or apparatus using MMVD and / or SMVD with a motion vector referring to a current picture.

[0416] A process or apparatus using MMVD and / or SMVD with generalized bidirectional prediction.

[0417] A process or apparatus using MMVD and / or SMVD with local illumination compensation.

[0418] • A process or apparatus for merge / intra mode using MMVD and / or SMVD in combination with multiple hypotheses.

[0419] A process or apparatus using MMVD and / or SMVD with a combined MVD or final MV expression.

[0420] A method using MMVD and / or SMVD with a regression-based model for each sub-block’s motion

[0421] The process or device of measuring a field.

[0422] A process or apparatus using MMVD and / or SMVD with symmetric MVD, where only

[0423] There is 1 MVD that is decoded with bidirectional prediction.

[0424] A process or apparatus using MMVD and / or SMVD with triangle partitioning.

[0425] A process or apparatus combining MMVD and bidirectional prediction.

[0426] A procedure or device that combines MMVD and CPR.

[0427] A process or device that combines MMVD and ATMVP.

[0428] A process or apparatus that combines MMVD and affine modeling.

[0429] A process or apparatus that combines MMVD and planar motion vector prediction.

[0430] A process or apparatus that combines MMVD and regression-based motion vector fields.

[0431] A process or apparatus that combines MMVD with GBI.

[0432] A process or apparatus that combines MMVD and spatiotemporal motion vector prediction.

[0433] A process or device that replaces SMVD with MMVD in AMVP mode.

[0434] A process or apparatus that combines SMVD with an affine motion model.

[0435] A process or apparatus that combines SMVD with a planar motion model.

[0436] A process or apparatus that combines SMVD with a regression-based motion model.

[0437] A process or apparatus that combines the SMVD with the ATMVP motion model.

[0438] A process or apparatus that combines SMVD with spatiotemporal motion vector prediction.

[0439] • A process or apparatus that modifies the SMVD pattern to impose symmetry on bi-directional motion vectors in the case of translation.

[0440] A process or apparatus that modifies the SMVD pattern to impose symmetries on the final affine model including rotation, scaling, scaling or translation.

[0441] • A bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0442] A bitstream or signal comprising a syntax conveying information generated according to any of the described embodiments

[0443] information.

[0444] • Creating and / or sending and / or receiving and / or decoding according to any of the embodiments described.

[0445] A method, process, apparatus, medium storing instructions according to any of the described embodiments,

[0446] A medium or signal that stores data.

[0447] • Inserting syntax elements in the signaling that enable the decoder to determine the coding mode in a manner corresponding to that used by the encoder.

[0448] • Creating and / or sending and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0449] • A TV, set-top box, cell phone, tablet or other electronic device that performs the transformation method(s) according to any of the described embodiments.

[0450] • A TV, set-top box, cell phone, tablet or other electronic device that performs the transformation method(s) determined according to any of the described embodiments and displays (e.g., using a monitor, screen or other type of display) the resulting image.

[0451] a TV, set-top box, cellular phone, tablet computer, or other electronic device that selects, band-limits, or tunes (e.g., using a tuner) a channel to receive a signal including an encoded image,

[0452] and performing the transformation method(s) according to any of the described embodiments. A TV, set-top box, cellular phone, tablet or other electronic device that receives over the air (e.g. using an antenna) a signal comprising an encoded image and performs the transformation method(s).

Claims

1. A method, include: Parsing syntax indicating use of a motion vector coding mode for a current video block, the motion vector coding mode being a merge mode with motion vector difference (MMVD) coding mode or a symmetric motion vector difference (SMVD) coding mode; Obtaining motion information of a video block based on the motion vector decoding mode combined with another decoding tool; as well as The current video block is decoded using the motion information.

2. The method according to claim 1, wherein the motion vector decoding mode is the MMVD decoding mode, and the other decoding tool is any one of the following: current picture reference (CPR) mode, alternative temporal motion vector prediction (ATMVP) mode, planar motion vector prediction (PMPV) mode, regression-based motion vector field, triangular motion partition mode, generalized bi-prediction (GBI) temporal prediction mode, local illumination compensation (LIC) temporal prediction mode, multiple hypothesis prediction mode and spatial-temporal motion vector prediction (STMVP) mode.

3. The method according to claim 1, wherein the motion vector decoding mode is the SMVD decoding mode, and the other decoding tool is any one of the following: affine prediction mode, alternative temporal motion vector prediction mode, planar motion vector prediction mode, regression-based motion vector field mode, triangular motion partition mode, multiple hypothesis prediction mode and spatial-temporal motion vector prediction mode.

4. The method of claim 3, wherein the SMVD coding mode is modified so that the symmetry constraint is on the final motion vector or on the final affine model for bidirectional prediction of blocks in the translation adaptive motion vector prediction mode.

5. The method of claim 3, wherein SMVD is replaced by MMVD when used in combination with adaptive motion vector prediction mode.

6. A method, include: encoding syntax indicating use of a motion vector coding mode for a current video block, the motion vector coding mode being a merge mode with motion vector difference (MMVD) coding mode or a symmetric motion vector difference (SMVD) coding mode; Obtaining motion information of a video block based on the motion vector coding mode combined with another coding tool; and The current video block is encoded using the motion information.

7. The method of claim 6, wherein the motion vector coding mode is the MMVD coding mode, and the other coding tool is any one of: a current picture reference (CPR) mode, an alternative temporal motion vector prediction (ATMVP) mode, a planar motion vector prediction (PMPV) mode, a regression-based motion vector field, a triangular motion partitioning mode, a generalized bi-prediction (GBI) temporal prediction mode, a local illumination compensation (LIC) temporal prediction mode, a multiple hypothesis prediction mode, and a spatial-temporal motion vector prediction (STMVP) mode.

8. The method of claim 6, wherein the motion vector coding mode is the SMVD coding mode, and the other coding tool is any one of: an affine prediction mode, an alternative temporal motion vector prediction mode, a planar motion vector prediction mode, a regression-based motion vector field mode, a triangular motion partition mode, a multiple hypothesis prediction mode, and a spatial-temporal motion vector prediction mode.

9. The method of claim 8, wherein the SMVD coding mode is modified so that the symmetry constraint is on the final motion vector or on the final affine model for bidirectional prediction of blocks in the translation adaptive motion vector prediction mode.

10. The method of claim 8, wherein SMVD is replaced by MMVD when used in combination with adaptive motion vector prediction mode.

11. An apparatus comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform: Parsing syntax indicating use of a motion vector coding mode for a current video block, the motion vector coding mode being a merge mode with motion vector difference (MMVD) coding mode or a symmetric motion vector difference (SMVD) coding mode; Obtaining motion information of a video block based on the motion vector decoding mode combined with another decoding tool; as well as The current video block is decoded using the motion information.

12. The device of claim 11, wherein the motion vector coding mode is the MMVD coding mode, and the other coding tool is any one of: a current picture reference (CPR) mode, an alternative temporal motion vector prediction (ATMVP) mode, a planar motion vector prediction (PMPV) mode, a regression-based motion vector field, a triangular motion partitioning mode, a generalized bi-prediction (GBI) temporal prediction mode, a local illumination compensation (LIC) temporal prediction mode, a multiple hypothesis prediction mode, and a spatial-temporal motion vector prediction (STMVP) mode.

13. A device according to claim 11, wherein the motion vector decoding mode is the SMVD decoding mode, and the other decoding tool is any one of the following: affine prediction mode, alternative temporal motion vector prediction mode, planar motion vector prediction mode, regression-based motion vector field mode, triangular motion partition mode, multiple hypothesis prediction mode and spatial-temporal motion vector prediction mode.

14. The apparatus of claim 13, wherein the SMVD coding mode is modified so that the symmetric constraint is on a final motion vector or on a final affine model for bi-directional prediction of blocks in the translation adaptive motion vector prediction mode.

15. The apparatus of claim 13, wherein the SMVD is replaced by the MMVD when used in combination with the adaptive motion vector prediction mode.

16. An apparatus comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform: encoding syntax indicating use of a motion vector coding mode for a current video block, the motion vector coding mode being a merge mode with motion vector difference (MMVD) coding mode or a symmetric motion vector difference (SMVD) coding mode; Obtaining motion information of a video block based on the motion vector coding mode combined with another coding tool; and The current video block is encoded using the motion information.

17. The device of claim 16, wherein the motion vector coding mode is the MMVD coding mode, and the other coding tool is any one of: a current picture reference (CPR) mode, an alternative temporal motion vector prediction (ATMVP) mode, a planar motion vector prediction (PMPV) mode, a regression-based motion vector field, a triangular motion partitioning mode, a generalized bi-prediction (GBI) temporal prediction mode, a local illumination compensation (LIC) temporal prediction mode, a multiple hypothesis prediction mode, and a spatial-temporal motion vector prediction (STMVP) mode.

18. A device according to claim 16, wherein the motion vector decoding mode is the SMVD decoding mode, and the other decoding tool is any one of the following: affine prediction mode, alternative temporal motion vector prediction mode, planar motion vector prediction mode, regression-based motion vector field mode, triangular motion partition mode, multiple hypothesis prediction mode and spatial-temporal motion vector prediction mode.

19. The apparatus of claim 18, wherein the SMVD coding mode is modified so that the symmetric constraint is on a final motion vector or on a final affine model for bi-directional prediction of blocks in a translation adaptive motion vector prediction mode.

20. The apparatus of claim 18, wherein SMVD is replaced by MMVD when used in combination with adaptive motion vector prediction mode.

Images