Affine motion-based prediction in video coding

Abstract

The present invention provides a method for performing an affine motion compensation prediction (MCP for short) in a video decoder, and a method for performing an affine motion compensation prediction (MCP) in a video decoder. The method includes selecting an affine MCP pattern from a candidate set of affine MCP patterns. The method further includes performing affine motion compensated prediction in the video decoder based on the selected affine MCP mode. A first affine MCP mode candidate set is applied for a codec unit whose width and height are both greater than a first threshold. A second affine MCP mode candidate set is applied for a codec unit having at least one of a width and a height not greater than a first threshold. The first set of affine MCP mode candidates has more affine MCP mode candidates than the second set of affine MCP mode candidates.

Description

[0001] Cross-references

[0002] The present invention claims priority to U.S. Provisional Patent Application No. 63 / 378,245 filed on October 4, 2022, and claims priority to U.S. Provisional Patent Application No. 63 / 387,530 filed on December 15, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present invention relates to video coding and decoding. In particular, the present disclosure relates to the utilization of affine-based motion compensation in video encoding and decoding systems. Background Art

[0004] In the High Efficiency Video Coding (HEVC) standard, motion compensation prediction (MCP) only uses a translation motion model. However, in actual scenes, various types of motion may occur, such as zooming in / out, rotation, perspective motion, and other irregular motions. To solve this problem, the Versatile Video Coding (VVC) standard applies block-based affine transformation motion compensation prediction. By introducing affine prediction, 2D translation (two degrees of freedom) can be extended to more degrees of freedom. Summary of the invention

[0005] Aspects of the present disclosure provide a method for performing affine motion compensated prediction (MCP) in a video decoder. The method includes selecting an affine MCP mode from an affine MCP mode candidate set. The method also includes performing affine motion compensated prediction in the video decoder based on the selected affine MCP mode. For codec units whose width and height are both greater than a first threshold, a first affine MCP mode candidate set is applied. For codec units whose width and height are at least one not greater than the first threshold, a second affine MCP mode candidate set is applied. The first affine MCP mode candidate set has more affine MCP mode candidates than the second affine MCP mode candidate set.

[0006] Aspects of the present disclosure provide another method for performing affine motion compensated prediction (MCP) in a video encoder. The method includes selecting an affine MCP mode from an affine MCP mode candidate set. The method also includes performing affine motion compensated prediction in the video encoder based on the selected affine MCP mode. For codec units whose width and height are both greater than a first threshold, a first affine MCP mode candidate set is applied. For codec units whose width and height are at least one not greater than the first threshold, a second affine MCP mode candidate set is applied. The first affine MCP mode candidate set has more affine MCP mode candidates than the second affine MCP mode candidate set.

[0007] Aspects of the present disclosure provide an apparatus for performing affine motion compensated prediction (MCP) in a video decoder. The apparatus includes a circuit configured to select an affine MCP mode from an affine MCP mode candidate set, and to perform affine motion compensated prediction in the video decoder based on the selected affine MCP mode. For codec units whose width and height are both greater than a first threshold, a first affine MCP mode candidate set is applied. For codec units whose width and height are at least one not greater than the first threshold, a second affine MCP mode candidate set is applied. The first affine MCP mode candidate set has more affine MCP mode candidates than the second affine MCP mode candidate set.

[0008] A device for performing affine motion compensated prediction (MCP) in a video decoder. The device includes a circuit configured to select an affine MCP mode from an affine MCP mode candidate set; and perform affine motion compensated prediction in the video decoder based on the selected affine MCP mode. For codec units whose width and height are both greater than a first threshold, a first affine MCP mode candidate set is applied; for codec units whose width and height are at least one not greater than the first threshold and whose width is greater than or equal to the height, a second affine MCP mode candidate set is applied; for codec units whose width and height are at least one not greater than the first threshold and whose width is less than the height, a third affine MCP mode candidate set is applied. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Various embodiments of the present disclosure, presented as examples, will be described in detail with reference to the following drawings, wherein like numerals indicate like elements, and wherein:

[0010] Figure 1A-1B A 4-parameter affine motion model and a 6-parameter affine motion model based on two control points and three control points are shown respectively;

[0011] Figure 2 The affine motion vector field (MVF) of each sub-block is shown;

[0012] Figure 3 Different scenarios with codec unit sizes of 8x8 and 4x4 are shown;

[0013] Figure 4 The following illustrates a scenario where the codec unit size is 8x8 and 4x4 according to an embodiment of the present disclosure;

[0014] Figure 5 A flowchart showing a process of performing affine motion compensated prediction (MCP) in a video encoder or a video decoder according to an embodiment of the present disclosure;

[0015] Figure 6 Shows multiple types of tree segmentation patterns;

[0016] Figure 7 An example of a quadtree with a nested multi-type tree codec block structure is shown;

[0017] Fig. 8A and 8B FIG. 4 shows the search point layout in merge mode with motion vector difference (MMVD for short);

[0018] Fig. 9 shows the position of the inherited affine motion predictor;

[0019] Fig.10 An example showing inheritance of control point motion vectors;

[0020] Fig.11 showing the positions of candidate positions of the constructed affine merge pattern;

[0021] Fig.12 Shows sub-block MV V SB and pixel Δv(i,j) (dashed arrow);

[0022] Fig.13 An example of decoding-side motion vector refinement is shown;

[0023] Fig.14 An example of geometric partition mode (GPM) segmentation grouped by the same angle is shown;

[0024] Fig.15 Shows the top and left neighboring blocks used in the derivation of combined inter-intra prediction (CIIP) weights;

[0025] Fig.16 shows the template matching performed on the search area around the initial MV;

[0026] Fig.17 A diamond-shaped area in the search area is shown;

[0027] Fig.18 Shows the theoretical vertical and horizontal positions of 4:2:0 luma and chroma samples in a picture;

[0028] Fig.19 Shows the theoretical vertical and horizontal positions of 4:2:2 luminance and chrominance samples in a picture;

[0029] Fig. 20 Shows the theoretical vertical and horizontal positions of 4:4:4 luma and chroma samples in a picture;

[0030] Fig.21 showing the template in the reference picture and the reference sample of the template;

[0031] Fig. 22 showing a template of a block with sub-block motion using motion information of a sub-block of a current block and a reference sample of the template;

[0032] Fig.23 Additional directions along the k×π / 8 diagonal angles are shown (the positions filled with the darkest texture are for anchor points);

[0033] Fig.24 shows 4x8 luma blocks and 2x4 chroma blocks when the input video is 4:2:0 chroma sampling;

[0034] Fig.25 A block diagram showing a video encoder according to an embodiment of the present disclosure; and

[0035] Fig.26 A block diagram of a video decoder according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0036] The following disclosure provides different embodiments or examples for implementing the different features of the provided subject matter. Specific examples of components and arrangements are described below to simplify the disclosure. Of course, these are merely examples and are not intended to be limiting.

[0037] For example, for the sake of clarity, the order of discussion of different steps is given herein. Generally, these steps can be performed in any suitable order. In addition, although each different feature, technology, configuration, etc. may be discussed in different locations of the present disclosure herein, it is intended that each concept can be performed independently of each other or in combination with each other. Therefore, the present disclosure can be embodied and viewed in many different ways.

[0038] Additionally, as used herein, the words "a," "an," etc. generally have the meaning of "one or more" unless otherwise indicated.

[0039] In the present disclosure, unless otherwise explicitly stated, the size of a coding unit (CU) is expressed in terms of the luma samples contained in the CU. In addition, the size of a luma (or chroma) block is given in units of luma (or chroma) samples within the block. In addition, the size of a coding unit or block can be expressed as WxH, where W represents the number of samples in the width direction and H represents the number of samples in the height direction.

[0040] The present disclosure relates to enhanced affine motion compensated prediction (MCP) used in video encoding and decoding systems. The enhanced affine-based MCP is applicable to small CU sizes and can simplify the parsing of related syntax elements.

[0041] Figure 1A-1B A 4-parameter affine motion model and a 6-parameter affine motion model are shown, which are based on two control points and three control points respectively. The affine motion field of the current block can be characterized by motion information derived from motion vectors of two control points (for a 4-parameter model) or three control points (for a 6-parameter model).

[0042] According to the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the current block can be calculated as:

[0043]

[0044] According to the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the current block can be calculated as:

[0045]

[0046] In these equations, (mv 0x ,mv 0y ) is the motion vector of the upper left control point v0, (mv 1x ,mv 1y ) is the motion vector of the upper right control point v1, (mv 2x ,mv 2y ) is the motion vector of the lower left control point v2.

[0047] Similar to translational motion inter prediction, two basic affine motion inter prediction modes, affine merge mode and affine Advanced Motion Vector Prediction (AMVP) mode, can be used to generate motion vectors for affine motion compensation. Affine merge mode uses motion information from spatially neighboring blocks to generate the control point motion vector (CPMV) of the current CU. In affine AMVP mode, the difference between the vector of the current CU and the vector of its predictor is transmitted in the bitstream.

[0048] In addition, sub-block based affine transformation prediction can be used to simplify motion compensated prediction. Figure 2 The affine motion vector field (MVF) of each sub-block is shown. Figure 2 As shown, to obtain the motion vector for each luminance subblock (e.g., 4×4 size), the motion vector of the center sample of each subblock is calculated using the above formula and rounded to 1 / 16 fractional precision. Then, a motion compensated interpolation filter can be applied to generate a prediction for each subblock using the derived motion vector. The subblock size of the chrominance component can also be set to 4×4. The motion vector of the 4×4 chrominance subblock is calculated as the average of the motion vectors of the upper left and lower right luminance subblocks in the same 8×8 luminance region.

[0049] Typically, existing video codecs define the minimum size of the codec block. For example, in Enhanced Compression Model (ECM) 6.0, the minimum block size of the regular inter-frame mode is defined as a 4x4 luminance block. In the case of VVC, when using the 4:2:0 chroma sampling format, the minimum chroma block size of the regular inter-frame mode is set to 2x2. For affine motion compensation based on sub-block luminance and chroma components, the predefined minimum sub-block size in ECM 6.0 is 4x4.

[0050] In some examples, affine merging / AMVP mode is limited to codec units with width and height both equal to or greater than 8. Figure 3 Different scenarios are shown for codec unit sizes of 8x8 and 4x4. For larger blocks (e.g., the 8x8 luma block shown), the corresponding two chroma blocks are both 4x4 in size, satisfying the requirement of 4x4 as the minimum sub-block size for applying affine motion compensation as defined in the affine merge / AMVP mode. However, for smaller blocks (e.g., the 4x4 luma block shown), the corresponding chroma blocks become too small (i.e., 2x2). Affine motion compensation as defined in the affine merge / AMVP mode is not applied.

[0051] According to an embodiment of the present disclosure, the minimum block size of the affine merging and / or affine AMVP mode may be aligned with the minimum block size allowed by the regular merging and / or regular AMVP mode. Figure 4 A scenario is shown in which the codec unit according to an embodiment of the present disclosure has 8x8 and 4x4 sizes. Based on the chroma sampling format adopted, a smaller sub-block size is allowed to be used instead of using a 4x4 sub-block size for the chroma component. For example, the chroma component can have a minimum sub-block size of 2x2. This allows affine merging and / or AMVP mode to be enabled on smaller codec units with a width and / or height less than 8.

[0052] It should be noted that in existing codecs, the smallest CU for regular inter mode is usually 4x4. If regular inter mode is allowed to use smaller CU sizes, such as 2x2 and 1x1, in the future, the method described in this disclosure can be adjusted accordingly to adapt to the new development of CU size.

[0053] According to one embodiment, affine merging and / or affine AMVP mode is allowed on small blocks such as 4xN or Nx4 CUs, and 4:2:0 chroma sampling format is adopted, where N is a power of 2 and N≥4. In this case, the corresponding chroma block size is 2x(N / 2) or (N / 2)x2. Therefore, the size of the chroma sub-block for affine motion compensation needs to be modified.

[0054] As an example, in sub-block based affine motion compensation, 4x4 luma sub-block size and 2x2 chroma sub-block size can be always used. Alternatively, different chroma sub-block sizes can be used for different CU sizes.

[0055] For example, for a codec unit of size 4xN or Nx4, the sub-block size of the luma component can be 4x4, while the sub-block size of the chroma component can be 2x2. In addition, for a codec unit with a width greater than 4 and a height greater than 4, both the luma component and the chroma component can use a sub-block size of 4x4.

[0056] In another example, when the size of the codec unit is 4x4, the luma subblock size may be 4x4 and the chroma subblock size may be 2x2. If the size of the codec unit is equal to 4xN, a subblock size of 2x4 may be used for the chroma components. If the size of the codec unit is equal to Nx4, a subblock size of 4x2 may be used for the chroma components. For codec units with a width greater than 4 and a height greater than 4, a subblock size of 4x4 may be used for both the luma and chroma components.

[0057] According to another embodiment, affine merging and / or affine AMVP mode is allowed on small blocks such as 4xN or Nx4 CUs, and a 4:2:2 chroma sampling format is adopted, where N is a power of 2, and N≥4. In this case, the corresponding chroma block size is 2x(N) and (N / 2)x4. As mentioned above, since the 4x4 sub-block size is too large for the chroma component, the adjusted minimum chroma sub-block size can be used to enable affine motion compensation.

[0058] In one example, in sub-block based affine motion compensation, 4x4 luma sub-block size and 2x4 chroma sub-block size can be always used. Alternatively, different chroma sub-block sizes can be used for different CU sizes.

[0059] For example, for a codec unit of size 4xN or Nx4, the sub-block size of the luma component can be 4x4, while the sub-block size of the chroma component can be 2x4. In addition, for a codec unit with a width greater than 4 and a height greater than 4, both the luma component and the chroma component can use a sub-block size of 4x4.

[0060] In another example, when the size of the codec unit is 4xN, the luma subblock size may be 4x4 and the chroma subblock size may be 2x4. If the size of the codec unit is equal to Nx4, where N>4, the luma subblock size may be 4x4 and the chroma subblock size may also be 4x4. For codec units with a width greater than 4 and a height greater than 4, both the luma component and the chroma component may use a subblock size of 4x4.

[0061] According to another embodiment, affine merging and / or affine SMVP mode is allowed on small blocks such as 4xN or Nx4 CUs, and a 4:4:4 chroma sampling format is adopted, where N is a power of 2 and N≥ 4. In this case, the corresponding chroma block size is 4xN or Nx4.

[0062] As an example, in sub-block based affine motion compensation, 4x4 luma sub-block size and 2x2 chroma sub-block size can be always used. Alternatively, different chroma sub-block sizes can be used for different CU sizes.

[0063] For example, for a codec unit of size 4xN or Nx4, the sub-block size of the luminance component can be 4x4, while the sub-block size of the chrominance component can be 2x2. In addition, for a codec unit with a width greater than 4 and a height greater than 4, both the luminance component and the chrominance component can use a sub-block size of 4x4.

[0064] In another example, when the size of the codec unit is 4x4, the luma subblock size may be 4x4 and the chroma subblock size may be 2x2. If the size of the codec unit is equal to 4xN, where N>4, the chroma component may use a subblock size of 2x4. If the size of the codec unit is equal to Nx4, where N>4, the chroma component may use a subblock size of 4x2. For codec units with a width greater than 4 and a height greater than 4, both the luma component and the chroma component may use a subblock size of 4x4.

[0065] As mentioned above, Versatile Video Coding (VVC) and ECM 6.0 provide flexibility, and the affine AMVP codec block can adaptively choose between 6-parameter and 4-parameter affine models. Compared with the 4-parameter affine AMVP mode, the 6-parameter affine AMVP mode requires more bits to represent motion information. However, for small blocks with relatively simple motion, it may not be necessary to use a more complex 6-parameter affine motion model.

[0066] Therefore, by relaxing the restrictions on the sub-block size, the parsing of the relevant syntax elements can be simplified. Since the simpler 4-parameter model is sufficient to characterize the motion pattern of a small block, there is no need to include syntax elements indicating the selection between the 4-parameter and 6-parameter affine motion models. In addition, fewer bits are required to send information related to the 4-parameter model compared to the 6-parameter model.

[0067] According to an embodiment of the present disclosure, when the affine motion model in the affine AMVP mode is applied to a codec unit (e.g., a codec unit of size 4xN or Nx4, where N is a power of 2, and N≥4), it is automatically inferred that the current CU uses a 4-parameter affine motion model. Therefore, specific syntax elements for indicating whether a 4-parameter or 6-parameter affine motion model is selected can be omitted. In addition, only two control point motion vectors need to be sent, thereby reducing the number of bits transmitted.

[0068] Those skilled in the art may identify various variations, modifications, and alternatives to simplify the syntax element transmission. For example, for a small CU, a 6-parameter affine motion model may be automatically inferred instead of a 4-parameter affine motion model, and only two control point motion vector signals need to be sent. The third control point motion vector may then be derived from the transmitted control point motion vector. For example, in the case of an Nx4 CU, only the upper left and upper right control point motion vectors need to be sent, while for a 4xN CU, only the upper left and lower left control point motion vectors need to be sent.

[0069] As another example, for a small decoding unit whose width is greater than or equal to the height (e.g., Nx4 CU), a 4-parameter affine motion model can be used, and motion vectors relative to the upper-left and upper-right control points can be sent; for a small decoding unit whose width is less than the height (e.g., 4xN CU), a 4-parameter affine motion model can be used, and motion vectors relative to the upper-left and lower-left control points can be sent.

[0070] It should be noted that these examples are merely illustrative and other implementations are possible without departing from the spirit and scope of the present disclosure.

[0071] For example, a small block can be defined as a codec unit with a width less than or equal to L and a height less than or equal to M, where L and M are non-negative integers. The values ​​of L and M can be pre-defined or sent to the bitstream as syntax elements of the video codec standard. The syntax elements can be sent at various levels, such as sequence level (e.g., in a sequence parameter set), picture level (e.g., in a picture parameter set), slice level (e.g., in a slice header), CTU level, CU level, or block level.

[0072] For example, for affine codec CU / blocks of size 4x4, 4x2 or 2x4, 6-parameter affine AMVP mode and / or affine MMVD mode may not be allowed. However, for CUs of size 4xN or Nx4, where N is a power of 2 and N>4, 6-parameter affine AMVP mode and affine MMVD mode may be allowed.

[0073] Optionally or additionally, for the edit decoding unit, the affine merge with motion vector difference (MMVD) mode may not be allowed to be used, thereby further simplifying the syntax element transmission.

[0074] Note that for the decoder unit, it is still possible to use the 6-parameter affine motion model in affine merge mode, since it does not require more bits compared to the 4-parameter affine merge mode.

[0075] Figure 5 A flow chart of a process for performing affine motion compensated prediction (MCP) in a video encoder according to an embodiment of the present disclosure is shown. Process 500 begins at step 510, where a decision is made as to whether affine motion compensated prediction is allowed based on the size of the codec unit. When both the width and height of the codec unit are equal to or greater than a threshold value (e.g., 4), step S510 decides to allow affine motion compensated prediction to be performed on the codec unit. When at least one of the width and height of the codec unit is less than a second threshold value, step S510 decides not to allow affine motion compensated prediction to be performed on the codec unit.

[0076] In step S520, an affine MCP mode is selected from the affine MCP mode candidate set. For codecs whose width and height are both greater than a threshold (e.g., 4), the first affine MCP mode candidate set is applied. For codecs whose width and height are at least one not greater than the threshold, the second affine MCP mode candidate set is applied. The first affine MCP mode candidate set may have more affine MCP mode candidates than the second affine MCP mode candidate set. For example, the 6-parameter affine AMVP mode and / or the affine MMVD mode may be included in the first affine MCP mode candidate set, but not in the second affine MCP mode candidate set.

[0077] In step S530, affine motion compensation prediction may be performed on the coding unit based on the selected affine MCP mode.

[0078] Any of the above-mentioned methods and means can be implemented in an encoder and / or a decoder. For example, any of the above-mentioned methods and means can be implemented in an inter-frame prediction module of an encoder and / or a decoder. Alternatively, any of the above-mentioned methods and means can be implemented as a processing circuit coupled to the inter-frame prediction module of an encoder and / or a decoder.

[0079] Although aspects of the present disclosure have been described in conjunction with specific embodiments presented as examples, these examples may be substituted, modified, and varied. Therefore, the embodiments described herein are intended to be illustrative rather than restrictive. Changes may be made without departing from the scope of the claims described below.

[0080] Aspects of the present disclosure can be further described as follows.

[0081] I. Video Coding and Decoding Methods

[0082] 1. Partition CTU using tree structure

[0083] In the High-Efficient Video Coding standard (HEVC), pictures are divided into a series of coding tree units (CTUs). For pictures with three sample arrays, a CTU consists of an NxN luma sample block and two corresponding chroma sample blocks, or for pictures encoded using three independent color planes, a CTU consists of an NxN monochrome plane sample block. The CTU concept is roughly similar to the macroblock concept in previous standards such as Advanced Video Coding (AVC). In the main profile, the maximum allowed size of a luma block in a CTU is specified as 64x64. A quadtree structure (called a codec tree) is used to divide a CTU into CUs to accommodate various local features. At the leaf CU level, it is decided whether to use inter (temporal) or intra (spatial) prediction to encode and decode the picture area. Depending on the PU division type, each leaf CU can be further divided into one, two, or four prediction units (PUs). Within a PU, the same prediction process is applied, and the relevant information is transmitted to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU partition type, the leaf CU can be divided into transform units (TUs) according to another quadtree structure similar to the codec tree of the CU. A key feature of the HEVC structure is that it has multiple partition concepts including CU, PU and TU.

[0084] The Versatile Video Coding (VVC) standard is the successor of HEVC. In VVC, a nested multi-type tree quadtree using binary and ternary tree partitioning structures replaces the concept of multiple partition unit types, that is, it eliminates the separation of the concepts of CU, PU, ​​and TU unless the size of the CU is too large for the maximum transform length, and supports more flexible CU partition shapes. In the codec tree structure, the CU can be square or rectangular. The coding tree unit (CTU) is first divided by a quadtree (also known as a quadtree) structure. Then, the quadtree leaf nodes can be further divided by a multi-type tree structure.

[0085] Figure 6 The partitioning pattern of multiple types of trees is shown in Figure 2. Figure 6As shown, there are four types of splits in the multi-type tree structure, vertical binary split (SPLIT_BT_VER), horizontal binary split (SPLIT_BT_HOR), vertical ternary split (SPLIT_TT_VER) and horizontal ternary split (SPLIT_TT_HOR). The leaf nodes of the multi-type tree are called coding units (CUs), and this split is used for prediction and transform processing without further division unless the CU is too large for the maximum transform length. This means that in most cases, in a quadtree with a nested multi-type tree codec block structure, the CU, PU, ​​and TU have the same block size. An exception occurs when the maximum supported transform length is less than the width or height of the CU color component.

[0086] exist Figure 7 In FIG. 4 , an example of a quadtree with a nested multi-type tree codec block structure is shown. Figure 7 The CTU is divided into multiple CUs using a quadtree and nested multi-type tree codec block structure, where the bold block edges represent quadtree partitions and the remaining edges represent multi-type tree partitions. The quadtree with nested multi-type tree partitions provides a content-adaptive codec tree structure consisting of CUs. The size of the CU can be as large as the CTU or as small as 4×4 (in units of luma samples). For the case of 4:2:0 chroma sampling format, the maximum chroma CB size is 64×64 and the minimum chroma CB size consists of 16 chroma samples.

[0087] In VVC, the maximum supported luma transform size is 64 × 64, and the maximum supported chroma transform size is 32 × 32. When the width or height of the CB is larger than the maximum transform width or height, the CB is automatically split horizontally and / or vertically to meet the transform size limit in that direction.

[0088] In VVC, the codec tree scheme supports luma and chroma to have separate block tree structures. For P and B slices, the luma and chroma CTBs in one CTU must share the same codec tree structure. However, for I slices, luma and chroma can have separate block tree structures. When separate block tree mode is applied, the luma CTB is divided into CUs by one codec tree structure, and the chroma CTB is divided into chroma CUs by another codec tree structure. This means that a CU in an I slice can consist of codec blocks for the luma component or codec blocks for two chroma components, while a CU in a P or B slice always consists of codec blocks for all three color components unless the video is monochrome.

[0089] For each inter-predicted CU, the motion parameters consist of a motion vector, a reference picture index and a reference picture list usage index, as well as additional information required by the new VVC codec feature for inter-prediction sample generation. The motion parameters can be signaled explicitly or implicitly. When a CU is encoded and decoded using skip mode, the CU is associated with one PU and has no significant residual coefficients, no encoded motion vector increments or reference picture indices. A merge mode is specified, in which the motion parameters of the current CU are obtained from neighboring CUs (including spatial and temporal candidates) and the additional scheduling introduced in VVC. Merge mode can be applied to any inter-predicted CU, not just skip mode. An alternative to merge mode is to explicitly transmit motion parameters, in which motion vectors, corresponding reference picture indices for each reference picture list and reference picture list usage flags, as well as other required information, are explicitly sent per CU.

[0090] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 5) are investigating the potential need for standardization of future video codecs that will have compression capabilities that significantly exceed the current VVC standard. The Enhanced Compression Model (ECM) reference software is provided to demonstrate a reference implementation of the encoding techniques and decoding processes of the JVET enhanced compression exploration work beyond VVC capabilities. ECM is essentially the successor to VVC, so it has many things in common with VVC.

[0091] 2. Overview of Inter-frame Prediction

[0092] In HEVC, for each inter PU, one of three prediction modes can be selected, including inter, skip and merge. In general, a motion vector competition (MVC) scheme is introduced to select motion candidates from a given candidate set containing spatial and temporal motion candidates. Multiple references to motion estimation allow the best reference to be found in 2 possible reconstructed reference picture lists (i.e., list 0 and list 1). For inter mode (informally known as AMVP mode, where AMVP stands for advanced motion vector prediction), inter prediction indicators (list 0, list 1 or bidirectional prediction), reference index, motion candidate index, motion vector difference (MVD) and prediction residual are sent. For skip mode and merge mode, only the merge index is transmitted, and the current PU inherits the inter prediction indicator, reference index and motion vector from the neighboring PU referenced by the codec merge index. In the case where the CU is encoded and decoded by skip mode, the residual signal is also omitted.

[0093] In VVC, AMVP mode is further improved with new modes such as symmetric motion vector difference (SMVD) mode, adaptive motion vector resolution (AMVR), and affine AMVP mode; merge / skip mode is further improved with enhanced merge candidates, combined inter-intra prediction (CIIP), affine merge mode, subblock temporal motion vector predictor (SbTMVP), merge mode with motion vector difference (MMVD), and geometric partition mode (GPM). In VVC, decoder-side motion vector refinement (DMVR), bi-directional optical flow (BDOF), and prediction refinement with optical flow (PROF) are used to refine motion vectors or motion compensation predictors at the decoder.

[0094] In ECM, several new codec tools are used to further improve AMVP, merge and skip modes, such as bilateral matching AMVP-merge mode, multi-hypothesis prediction (MHP), overlapped block motion compensation (OBMC), etc. In addition, decoder-side motion vector refinement based on template matching is also proposed to improve the codec efficiency of inter-frame prediction.

[0095] In addition to the inter-frame coding and decoding functions in HEVC, VVC also includes many new and improved inter-frame prediction coding and decoding tools, as shown below:

[0096] – Extended Consolidated Forecast

[0097] – Merge mode with MVD (MMVD for short)

[0098] –Symmetric MVD (SMVD) signal transmission

[0099] – Affine motion compensated prediction

[0100] – Subblock-based temporal motion vector prediction (SbTMVP)

[0101] – Adaptive motion vector resolution (AMVR)

[0102] – Motion field storage: 1 / 16 luminance sample MV storage and 8x8 motion field compression

[0103] – Bi-prediction with CU-level weight (BCW)

[0104] – Bi-direction optical flow (BDOF)

[0105] –Decoder side motion vector refinement (DMVR)

[0106] – Geometric partitioning mode (GPM)

[0107] – Combined inter and intra prediction (CIIP)

[0108] After VVC was finalized, the Enhanced Compression Model (ECM) reference software was developed to study the potential needs for future video codec standardization. In the current ECM, several inter-frame prediction codec tools are included to further save BD rate:

[0109] – Local illumination compensation (LIC)

[0110] – Non-adjacent spatial candidates

[0111] – Template Matching (TM)

[0112] – Overlapped Block Motion Compensation (OBMC)

[0113] –Multi-hypothesis prediction (MHP)

[0114] – Bilateral matching AMVP merge mode

[0115] – and several other tools under development

[0116] The following text details some selected inter prediction methods specified in VVC and ECM.

[0117] 3. Extended Merge Forecast

[0118] In VVC, the merge candidate list consists of the following five types of candidates in order:

[0119] 1) Spatial MVP from spatial neighbor CU

[0120] 2) Time MVP from the same CU

[0121] 3) History-based MVP from FIFO table

[0122] 4) Paired Average MVP

[0123] 5) Zero MV.

[0124] The size of the merge list is indicated in the sequence parameter set header, and the maximum allowed size of the merge list is 6. For each CU encoded or decoded in merge mode, the best merge candidate index is encoded using truncated unary binarization (TU). The first bin of the merge index is encoded or decoded using context, and the other bins are encoded or decoded using bypass coding.

[0125] This section provides the derivation process for each merge candidate category. As in HEVC, VVC also supports parallel derivation of merge candidate lists (or merge candidate lists) for all CUs in a certain size region.

[0126] 4. History-based Merge Candidate Derivation

[0127] History-based MVP (HMVP) merge candidates are added to the merge list after spatial MVP and TMVP. In this method, the motion information of the previously coded blocks is stored in a table and used as the MVP of the current CU. A table with multiple HMVP candidates is maintained during the encoding / decoding process. The table is reset (cleared) when a new CTU row is encountered. Whenever there is a non-inter-subblock coded CU, the relevant motion information is added to the last entry of the table as a new HMVP candidate.

[0128] The HMVP table size S is set to 6, which means that up to 5 history-based MVP (HMVP) candidates can be added to the table. When a new motion candidate is inserted into the table, a constrained first-in-first-out (FIFO) rule is used, where a redundancy check is first used to find out whether there is an identical HMVP in the table. If found, the identical HMVP is deleted from the table, and all subsequent HMVP candidates are moved forward, and the identical HMVP is inserted into the last entry of the table.

[0129] HMVP candidates can be used in the merge candidate list construction process. Check the latest few HMVP candidates in the table in order and insert them into the candidate list after the TMVP candidate. Apply redundancy check to the HMVP candidates of spatial or temporal merge candidates.

[0130] To reduce the number of redundant check operations, the following simplifications are introduced:

[0131] 1) The last two entries in the table perform redundancy checks on the A1 and B1 space candidates respectively.

[0132] 2) Once the total number of available merge candidates reaches the maximum allowed merge candidate minus 1, the merge candidate list construction process of HMVP will terminate.

[0133] 5. Pairwise Average Merging Candidate Derivation

[0134] Paired average candidates are generated by averaging predefined candidate pairs in the existing merge candidate list, using the first two merge candidates. The first merge candidate is defined as p0Cand, and the second merge candidate can be defined as p1Cand. The average motion vector is calculated based on the p0Cand and p1Cand motion vectors available for each reference list respectively. If both motion vectors in one list are available, even if they point to different reference pictures, the two motion vectors are averaged and their reference pictures are set to the one of p0Cand; if only one motion vector is available, the motion vector is used directly; if no motion vector is available, this list is kept invalid. In addition, if the half-pixel interpolation filter index of p0Cand and p1Cand is different, it is set to 0.

[0135] When the merge list is not full after adding pairwise average merge candidates, zero MVPs are inserted at the end until the maximum number of merge candidates is encountered.

[0136] 6. Merge mode with MVD (MMVD)

[0137] In addition to the merge mode (in which the implicitly derived motion information is directly used for prediction sample generation of the current CU), VVC also introduces a merge mode with motion vector difference (MMVD). After sending the regular merge flag, the MMVD flag signal is immediately sent to specify whether to use the MMVD mode for the CU.

[0138] In MMVD, after a merge candidate is selected, it is further refined by signaling MVD information. Further information includes a merge candidate flag, an index specifying the magnitude of motion, and an index indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected as the MV basis. The MMVD candidate flag is sent to specify which one to use between the first and second merge candidates.

[0139] Fig. 8A and 8B Displays the MMVD search point. The distance index specifies the motion magnitude information and indicates the predetermined offset from the starting point. Figures 8A-8B , the offset is added to the horizontal component or the vertical component of the starting MV. The relationship between the distance index and the predetermined offset is specified in Table 1.

[0140] Table 1 Relationship between distance index and predetermined offset

[0141]

[0142] The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions as shown in Table 2. It should be noted that the meaning of the MVD symbol may vary depending on the information of the starting MV. When the starting MV is a unidirectional prediction MV or a bidirectional prediction MV and both lists point to the same side of the current picture (i.e., the POCs of both reference pictures are greater than the POC of the current picture, or are less than the POC of the current picture), the symbol in Table 2 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectional prediction MV, and the two MVs point to different sides of the current picture (i.e., the POC of one reference picture is greater than the POC of the current picture, and the POC of the other reference picture is less than the POC of the current picture), and the POC difference in list 0 is greater than the POC difference in list 1, the symbol in Table 2 specifies the sign of the MV offset added to the list 0 MV component of the starting MV, and the sign of the list 1 MV has the opposite value. Otherwise, if the POC difference in list 1 is greater than list 0, the symbol in Table 2 specifies the sign of the MV offset added to the list 1 MV component of the starting MV, and the sign of the list 0 MV has the opposite value.

[0143] The MVD is scaled according to the POC difference in each direction. If the POC difference in both lists is the same, no scaling is required. Otherwise, if the POC difference in list 0 is greater than the POC difference in list 1, the MVD of list 1 is scaled, as described by defining the POC difference for L0 as td and the POC difference for L1 as tb. If the POC difference for L1 is greater than L0, the MVD of list 0 is scaled in the same way. If the starting MV is unidirectionally predicted, the MVD is added to the available MVs.

[0144] Table 2 – Sign of MV offset specified by direction index

[0145] Directions to IDX 0 1 0 1 x-axis / A / A y-axis / A / A

[0146] 7. Affine Motion Compensated Prediction

[0147] In HEVC, motion compensation prediction (MCP) only uses translational motion models. In the real world, there are many types of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensation prediction is used. Figure 1A-1B The affine motion model based on 4 parameter control points and the affine motion model based on 6 parameter control points are shown respectively. Figure 1A-1B As shown, the affine motion field of a block is described by the motion information of two control points (4 parameters) or three control point motion vectors (6 parameters).

[0148] For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as follows:

[0149]

[0150] For the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as follows:

[0151]

[0152] Where (mv 0x ,mv 0y ) is the motion vector of the upper left control point, (mv 1x ,mv 1y ) is the motion vector of the upper right control point, (mv 2x ,mv 2y ) is the motion vector of the lower left control point.

[0153] To simplify motion compensated prediction, block-based affine transform prediction is applied. Figure 2The affine MVF of each sub-block is displayed. In order to derive the motion vector of each 4×4 luminance sub-block, the motion vector of the center sample of each sub-block is calculated according to the above formula (such as Figure 2 The MV of a 4×4 chroma subblock is calculated as the average of the MVs of the top left and bottom right luminance subblocks in the same 8×8 luminance region.

[0154] Like translational motion inter prediction, there are two affine motion inter prediction modes: affine merge mode and affine AMVP mode.

[0155] 8. Affine Merge Prediction

[0156] AF_MERGE mode can be applied to CUs with width and height greater than or equal to 8. In this mode, the CPMV of the current CU is generated based on the motion information of spatially adjacent CUs. There can be up to five CPMVP candidates, and an index is issued to indicate which candidate to use for the current CU. The following three types of CPVM candidates are used to form the affine merge candidate list:

[0157] – Inherited affine merge candidates inferred from the CPMV of neighboring CUs

[0158] – Affine merge candidate CPMVP constructed using the translation MV of the adjacent CU

[0159] – Zero MV

[0160] In VVC, there are at most two inherited affine candidates, which come from the affine motion models of neighboring blocks, one from the left neighboring CU and one from the above neighboring CU. Fig. 9 The positions of the inherited affine motion predictors are shown. The candidate blocks are Fig. 9 As shown. For the left predictor, the scanning order is A0->A1, and for the upper predictor, the scanning order is B0->B1->B2. Only the first inheritance candidate on each side is selected. No pruning check is performed between two inheritance candidates. When a neighboring affine CU is identified, its control point motion vectors are used to derive the CPMVP candidate in the affine merge list of the current CU. Fig.10 Display control point motion vector inheritance. As shown in the figure, if the adjacent lower left block A is encoded and decoded using the affine mode, the motion vectors v2, v3, and v4 of the upper left, upper right, and lower left corners of the CU containing block A can be obtained. If block A is encoded and decoded using the 4-parameter affine model, the two CPMVs of the current CU are calculated based on v_2 and v_3. If block A is encoded and decoded using the 6-parameter affine model, the three CPMVs of the current CU are calculated based on v2, v3, and v4.

[0161] Fig.11 Displays the candidate positions of the constructed affine merge mode. The constructed affine candidate means that the candidate is constructed by combining the translation motion information of the neighbors of each control point. The motion information of the control point comes from Fig.11 The specified spatial neighbors and temporal neighbors shown in CPMV k (k=1, 2, 3, 4) indicates the kth control point. For CPMV1, check B2->B3->A2 blocks and use the MV of the first available block. For CPMV2, check B1->B0 blocks, and for CPMV3, check A1->A0 blocks. For TMVP, if available, use it as CPMV4.

[0162] After obtaining the MVs of the four control points, construct the affine merge candidate based on the motion information. Use the following control point MV combinations in order to construct:

[0163] {CPMV1,CPMV2,CPMV3},{CPMV1,CPMV2,CPMV4},

[0164] {CPMV1,CPMV3,CPMV4},{CPMV2,CPMV3,CPMV4},

[0165] {CPMV1,CPMV2},{CPMV1,CPMV3}

[0166] The combination of 3 CPMVs constitutes a 6-parameter affine merge candidate, and the combination of 2 CPMVs constitutes a 4-parameter affine merge candidate. To avoid the motion scaling process, if the reference index of the control point is different, the related control point MV combination is discarded.

[0167] After checking inherited affine merge candidates and constructed affine merge candidates, if the list is still not full, a zero MV is inserted at the end of the list.

[0168] 9. Prediction refinement with optical flow foraffine mode (PROF)

[0169] Compared with pixel-based motion compensation, sub-block-based affine motion compensation can save memory access bandwidth and reduce computational complexity, but the cost is that the prediction accuracy will be reduced. In order to achieve finer-grained motion compensation, prediction refinement with optical flow (PROF) is used to refine sub-block-based affine motion compensation predictions without increasing the memory access bandwidth of motion compensation. In VVC, after performing sub-block-based affine motion compensation, the brightness prediction samples are refined by adding the difference derived from the optical flow equation. PROF is described as the following four steps:

[0170] Step 1) Perform sub-block based affine motion compensation to generate sub-block prediction I(i,j).

[0171] Step 2) Use a 3-tap filter [-1, 0, 1] to calculate the spatial gradient g of the sub-block prediction at each sampling position x (i,j) and g y (i,j). The gradient calculation is exactly the same as that in BDOF.

[0172] g x (i,j)=(I(i+1,j)>>shift1)-(I(i-1,j)>>shift1) (3-17)

[0173] g y (i,j)=(I(i,j+1)>>shift1)-(I(i,j-1)>>shift1) (3-18)

[0174] shift1 is used to control the accuracy of the gradient. The sub-block (i.e. 4x4) prediction is extended by one sample on each side for gradient calculation. To avoid extra memory bandwidth and extra interpolation calculations, the extended samples on the extended boundary are copied from the nearest integer pixel position in the reference picture.

[0175] Step 3) Brightness prediction refinement is calculated using the following optical flow equation.

[0176] ΔI(i,j)= g x (i,j)*Δv x (i,j)+g y (i,j)*Δv y (i,j) (3-19)

[0177] Where Δv(i,j) is the difference between the sample MV calculated for the sample position (i,j) (denoted as v(i,j)) and the sub-block MV of the sub-block to which the sample (i,j) belongs, such as Fig.12 As shown in Figure 2, Δv(i,j) is quantized in units of 1 / 32 chroma sample precision.

[0178] Fig.12 The sub-block MV VSB and pixel Δv(i,j) are shown (dashed arrows). Since the affine model parameters and the sample positions relative to the sub-block center do not change between sub-blocks, Δv(i,j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. Let dx(i,j) and dy(i,j) be the distance from the sample position (i,j) to the sub-block center (x SB ,y SB ), Δv(x,y) can be obtained by the following equations,

[0179]

[0180] In order to ensure accuracy, the sub-block (x SB ,y SB ) is calculated as ((W SB -1) / 2, (H SB -1) / 2), where W SB and H SB are the width and height of the sub-block respectively.

[0181] For the 4-parameter affine model,

[0182]

[0183] For the 6-parameter affine model,

[0184]

[0185] Where (v 0x ,v 0y )、(v 1x ,v 1y )、(v 2x ,v 2y ) are the motion vectors of the upper left, upper right and lower left control points respectively, and w and h are the width and height of the CU respectively.

[0186] Step 4) Finally, the luma prediction refinement ΔI(i,j) is added to the sub-block prediction I(i,j). The final prediction I' is generated according to the following formula.

[0187] I′(i,j)=I(i,j)+ΔI(i,j)

[0188] For affine codec CUs, PROF is not applicable in two cases: 1) all control point MVs are the same, indicating that the CU has only translational motion; 2) the affine motion parameters are larger than the specified limit, because the sub-block based affine MC degenerates to the CU based MC to avoid the larger memory access bandwidth requirement.

[0189] A fast coding method is used to reduce the coding complexity of PROF-based affine motion estimation. PROF is not applied in the affine motion estimation stage in the following two cases: a) If the CU is not a root block and its parent block does not select the affine mode as its best mode, PROF is not applied because the possibility that the current CU selects the affine mode as the best mode is low; b) If the magnitudes of the four affine parameters (C, D, E, F) are all less than a predetermined threshold and the current image is not a low-latency image, PROF is not applied because the improvement introduced by PROF is small for this case. In this way, PROF-based affine motion estimation can be accelerated.

[0190] 10.Decoder side motion vector refinement (DMVR)

[0191] In order to improve the accuracy of the MV in merge mode, decoder-side motion vector refinement based on bilateral matching (BM) is applied in VVC. In the bidirectional prediction operation, the refined MV is searched around the initial MV in the reference picture list L0 and the reference picture list L1. The BM method calculates the distortion between two candidate blocks in the reference picture list L0 and list L1. Fig.13 Displays the decoder side motion vector refinement. Fig.13 As shown, the SAD between blocks surrounding each MV candidate of the initial MV is calculated, and these blocks are filled with sparser diagonal stripes. The MV candidate with the lowest SAD becomes the refined MV and is used to generate the bidirectional prediction signal.

[0192] In VVC, the application of DMVR is limited and only applies to CUs using the following modes and feature codecs:

[0193] –CU level merge mode and bidirectional prediction MV

[0194] – Relative to the current image, one reference image is in the past and the other reference image is in the future

[0195] – The distance between the two reference images and the current image (i.e., POC difference) is the same

[0196] – Both reference images are short-term reference images

[0197] –CU has more than 64 luma samples

[0198] – CU height and CU width are both greater than or equal to 8 luma samples

[0199] – BCW weighted index means equal weighting

[0200] – The current block is not WP enabled

[0201] – The current block does not use CIIP mode

[0202] The improved MV obtained by DMVR processing is used to generate inter-frame prediction samples and is also used for temporal motion vector prediction of future picture encoding and decoding. The original MV is used in the deblocking process and is also used for spatial motion vector prediction of future CU encoding and decoding.

[0203] Additional features of DMVR are mentioned in the following sub-clauses.

[0204] 11. Geometric partitioning mode (GPM)

[0205] In VVC, the geometric partition mode supports inter-frame prediction. The geometric partition mode is represented by a CU-level flag as a merge mode. Other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. For each possible CU size w×h=2 m ×2 n , the geometric partitioning mode supports a total of 64 partitions, where m,n∈{3…6} (except 8x64 and 64x8).

[0206] Fig.14 An example of GPM segmentation showing groups by the same angle. When this mode is used, the CU is split into two parts by a geometrically positioned straight line ( Fig.14 ). The position of the partitioning line is mathematically derived from the angle and offset parameters of the specific partition. Each part of the geometric partition in the CU is inter-predicted using its own motion; only unidirectional prediction is allowed for each partition, i.e. each part has a motion vector and a reference index. Unidirectional prediction motion constraints are applied to ensure that, as with traditional bidirectional prediction, only two motion compensated predictions are required per CU. The unidirectional prediction motion for each partition is derived using the process described in 3.4.11.1.

[0207] If the current CU uses geometric partitioning mode, a geometric partition index is also emitted, indicating the partitioning mode (angle and offset) of the geometric partition and two merge indices (one for each partition). The number of maximum GPM candidate sizes is sent explicitly in the SPS, and the syntax for the GPM merge indices is specified in binarization. After predicting each part of the geometric partition, the sample values ​​at the edges of the geometric partition are adjusted using a blending process with adaptive weights, as described in 3.4.11.2. This is the prediction signal for the entire CU, and like other prediction modes, the transform and quantization process will be applied to the entire CU. Finally, the motion field of the CU predicted using the geometric partition mode is stored, as described in 3.4.11.3.

[0208] 12. Combined inter and intra prediction (CIIP)

[0209] In VVC, when a CU is encoded and decoded in merge mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and both the CU width and the CU height are less than 128 luma samples, an additional flag is sent to indicate whether the combined inter / intra prediction (CIIP) mode is applied to the current CU. Fig.15 The top and left neighboring blocks used for CIIP weight derivation are shown. As the name implies, CIIP prediction combines the inter prediction signal with the intra prediction signal. The inter prediction signal P in CIIP mode inter The intra prediction signal P is derived using the same inter prediction process applied to the normal merge mode; and intra The intra-frame and inter-frame prediction signals are then combined using a weighted average, where the weights are calculated based on the coding mode of the top and left neighboring blocks (e.g. Fig.15 ), as shown below:

[0210] – If the top neighbor is available and intra-coded, isIntraTop is set to 1, otherwise isIntraTop is set to 0;

[0211] – If the left neighbor is available and is intra-coded, isIntraLeft is set to 1, otherwise isIntraLeft is set to 0;

[0212] – If (isIntraLeft+isIntraTop) is equal to 2, wt is set to 3;

[0213] – Otherwise, if (isIntraLeft+isIntraTop) is equal to 1, wt is set to 2;

[0214] – Otherwise, wt is set to 1.

[0215] CIIP forecasts the following:

[0216] P CIIP =((4-wt)*P inter +wt*P intra +2)>>2 (3-43)

[0217] 13. Template matching (TM)

[0218] Template matching (TM) is a decoder-side MV derivation method that refines the motion information of the current CU by finding the closest match between a template in the current picture (i.e., the top and / or left neighboring blocks of the current CU) and a block in a reference picture (i.e., the same size as the template). Fig.16 Shows how template matching is performed on the search area around the initial MV. Fig.16 As shown, in the [-8, +8] pixel search range, a better MV is searched around the initial motion of the current CU. The template matching method in JVET-J0021 is used with the following modifications: the search step size is determined based on the AMVR mode, and the TM can be cascaded with the bilateral matching process in the merge mode.

[0219] In AMVP mode, the MVP candidate is determined based on the template matching error to select the MVP candidate with the smallest difference between the current block template and the reference block template, and then TM is performed only on this specific MVP candidate for MV refinement. TM refines this MVP candidate starting from full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode) within the [-8, +8] pixel search range using an iterative diamond search. The AMVP candidate can be further refined using a cross search of full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode), followed by half-pixels and quarter-pixels in turn according to the AMVR mode, as described in Table 3. This search process ensures that the MVP candidate still maintains the same MV accuracy as indicated by the AMVR mode after the TM process. During the search process, if the difference between the previous minimum cost and the current minimum cost in the iteration is less than or equal to the threshold of the block area, the search process terminates.

[0220] Table 3. Search patterns of AMVR and merge patterns with AMVR.

[0221]

[0222]

[0223] In merge mode, a similar search method is applied to the merge candidates indicated by the merge index. As shown in Table 3, TM may be performed all the way to 1 / 8 pixel MVD accuracy or skip accuracy beyond half-pixel MVD accuracy, depending on whether an optional interpolation filter is used based on the merged motion information (used when AMVR is half-pixel mode). In addition, when TM mode is enabled, template matching can be processed as an independent process between block-based and sub-block-based bilateral matching (BM) methods or as an additional MV refinement, depending on whether BM can be enabled according to its enabling condition check.

[0224] 14. Multi-pass decoder-side motion vector refinement

[0225] Multiple passes of decoder-side motion vector refinement are applied. In the first pass, bilateral matching (BM) is applied to the codec block. In the second pass, BM is applied to each 16x16 sub-block within the codec block. In the third pass, the MV in each 8x8 sub-block is refined by applying bi-directional optical flow (BDOF). The refined MVs are stored for spatial and temporal motion vector prediction.

[0226] (1) First pass: Block-based bidirectional matching MV refinement

[0227] In the first pass, the refined MV is derived by applying BM to the codec block. Similar to decoder-side motion vector refinement (DMVR), in the bidirectional prediction operation, the refined MV is searched around the two initial MVs (MV0 and MV1) in the reference picture lists L0 and L1. The refined MVs (MV0_pass1 and MV1_pass1) are derived around the initial MV based on the minimum bilateral matching cost between the two reference blocks in L0 and L1.

[0228] BM performs local search and obtains integer sample precision intDeltaMV. The local search adopts a 3×3 square search pattern, with a circular search range of [–sHor, sHor] in the horizontal direction and a circular search range of [–sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block dimension, and the maximum values ​​of sHor and sVer are both 8.

[0229] The bilateral matching cost is calculated as: bilCost = mvDistanceCost + sadCost. When the block size cbW*cbH is greater than 64, the MRSAD cost function is used to eliminate the DC effect of the distortion between reference blocks. When the bilCost at the center point of the 3×3 search pattern has the minimum cost, the intDeltaMV local search terminates. Otherwise, the current minimum cost search point becomes the new center point of the 3×3 search pattern, and the search for the minimum cost continues until the end of the search range is reached.

[0230] The existing fractional sample refinement is further applied to get the final deltaMV. The refined MV after the first pass is as follows:

[0231] MV0_pass1 = MV0 + deltaMV

[0232] MV1_pass1 = MV1 – deltaMV

[0233] (2) Second pass: Sub-block-based bilateral matching MV refinement

[0234] In the second pass, the refined MV is derived by applying BM to the 16×16 grid sub-blocks. For each sub-block, the refined MV is searched in the reference picture lists L0 and L1 around the two MVs (MV0_pass1 and MV1_pass1) obtained in the first pass. The refined MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the two reference sub-blocks in L0 and L1.

[0235] For each sub-block, BM performs a full search to obtain integer sample precision intDeltaMV. The horizontal search range of the full search is [-sHor, sHor], and the vertical search range is [-sVer, sVer], where the values ​​of sHor and sVer are determined by the block dimension, and the maximum values ​​of sHor and sVer are both 8.

[0236] The bilateral matching cost is calculated by applying a cost factor to the SATD cost between two reference sub-blocks as follows: bilCost = satdCost * costFactor. Fig.17 Displays the diamond-shaped areas in the search area. The search area (2*sHor+1)*(2*sVer+1) is divided into a maximum of 5 diamond-shaped search areas, such as Fig.17 As shown. Each search area is assigned a costFactor, which is determined by the distance (intDeltaMV) between each search point and the starting MV, and each diamond area is processed in order starting from the center of the search area. In each area, the search points are processed in a raster scan order from the upper left corner to the lower right corner of the area. When the minimum bilCost in the current search area is less than or equal to the threshold of sbW*sbH, the integer-pixel full search is terminated, otherwise, the integer-pixel full search continues to the next search area until all search points are checked. In addition, if the difference between the previous minimum cost and the current minimum cost in the iteration is less than or equal to the threshold of the block area, the search process is terminated.

[0237] The existing VVC DMVR fractional sample refinement is further applied to derive the final deltaMV (sbIdx2). Then the second pass refined MV is derived as follows:

[0238] ●MV0_pass2(sbIdx2)=MV0_pass1+deltaMV(sbIdx2)

[0239] ●MV1_pass2(sbIdx2)=MV1_pass1–deltaMV(sbIdx2)

[0240] (3) The third pass: sub-block-based bidirectional optical flow MV refinement

[0241] In the third pass, the refined MV is derived by applying BDOF to the 8×8 grid sub-blocks. For each 8×8 sub-block, BDOF refinement is used to derive scaled Vx and Vy without clipping from the refined MV of the second pass parent sub-block. The derived bioMv(Vx,Vy) is rounded to 1 / 16 sample precision and clipped between -32 and 32.

[0242] The refined MVs of the third pass (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) are derived as follows:

[0243] ●MV0_pass3(sbIdx3)=MV0_pass2(sbIdx2)+bioMv

[0244] ●MV1_pass3(sbIdx3)=MV0_pass2(sbIdx2)–bioMv

[0245] 14. Bidirectional matching AMVP merge mode

[0246] The bidirectional predictor consists of an AMVP predictor in one direction and a merged predictor in another direction. This mode can be enabled for a codec block when the selected merged predictor and AMVP predictor meet the DMVR condition, where there is at least one reference picture from the past and one reference picture from the future relative to the current picture, and the distances of the two reference pictures to the current picture are the same, and bidirectional matching MV refinement will be applied to the merged MV candidate and the AMVP MVP as a starting point. Otherwise, if the template matching function is enabled, the template matching MV refinement is applied to the merged predictor or the AMVP predictor with a higher template matching cost.

[0247] The AMVP part of the pattern is sent as regular unidirectional AMVP, ie the reference index and MVD are sent, and if template matching is used, it has the derived MVP index, or the MVP index is sent when template matching is disabled.

[0248] For AMVP direction LX, X can be 0 or 1, and the merged part of the other direction (1-LX) is implicitly derived by minimizing the bilateral matching cost between the AMVP predictor and the merge predictor, that is, for a pair of AMVP and merge motion vectors. For each merge candidate with another direction (1-LX) motion vector in the merge candidate list, the bilateral matching cost is calculated using the merge candidate MV and the AMVP MV. The merge candidate with the minimum cost is selected. Bilateral matching refinement is applied to the codec block starting from the selected merge candidate MV and AMVP MV.

[0249] The third pass of multi-pass DMVR is the 8x8 sub-PU BDOF refinement of multi-pass DMVR, which has been enabled by the AMVP merge mode codec block.

[0250] The mode is indicated by a flag, and if the mode is enabled, the AMVP direction LX is also indicated by a flag.

[0251] When the current block uses bilateral matching (BM) AMVP merge mode and template matching is enabled, the MVD signal is not sent. An additional pair of AMVP merge MVP is introduced. The merge candidate list is sorted in ascending order according to the BM cost. An index (0 or 1) is sent to indicate which merge candidate in the sorted merge candidate list to use. When there is only one candidate in the merge candidate list, the AMVP MVP and merge MVP pair that is not processed by bilateral matching MV refinement is filled.

[0252] 15. Chroma format sampling structure

[0253] In monochrome sampling, there is only one sample array, nominally considered to be the brightness array.

[0254] In 4:2:0 sampling, each of the two chroma arrays has half the height and half the width of the luma array.

[0255] In 4:2:2 sampling, each of the two chroma arrays has the same height and half the width as the luma array.

[0256] In 4:4:4 sampling, each of the two chroma arrays has the same height and width as the luma array.

[0257] The number of bits required for the representation of each sample in the luminance and chrominance arrays of a video sequence is in the range of 8 to 16 (inclusive).

[0258] Fig.18 Displays the nominal vertical and horizontal positions of 4:2:0 luma and chroma samples in the picture. Fig.19 Displays the nominal vertical and horizontal positions of 4:2:2 luma and chroma samples in the picture. Fig. 20Displays the nominal vertical and horizontal positions of 4:4:4 luma and chroma samples in the picture.

[0259] When the value of sps_chroma_format_idc is equal to 1, the nominal vertical and horizontal relative positions of luma and chroma samples in the picture are as follows: Fig.18 Optional chroma sample relative positions may be indicated in the VUI parameters as described in Rec.ITU-T H.274 | ISO / IEC 23002-7.

[0260] When the value of sps_chroma_format_idc is equal to 2, the chroma samples are co-located with the corresponding luma samples, and the nominal positions in the picture are as follows: Fig.19 shown.

[0261] When the value of sps_chroma_format_idc is equal to 3, all array samples are co-located in all picture cases, and the nominal positions in the picture are as follows Fig. 20 shown.

[0262] 16. Adaptive reordering of merge candidates with template matching (ARMC-TM)

[0263] Adaptively re-rank the merge candidates using template matching (TM). The re-ranking method is applicable to regular merge mode, TM merge mode, and affine merge mode (excluding SbTMVP candidates). For TM merge mode, merge candidates are re-ranked before the refinement process.

[0264] First, an initial merge candidate list is constructed according to a given inspection order, such as spatial, TMVP, non-adjacent, HMVP, paired, and virtual merge candidates. The candidates in the initial list are then divided into several subgroups. For template matching (TM) merge mode and adaptive DMVR mode, each merge candidate in the initial list is first refined using TM / multi-pass DMVR. The merge candidates in each subgroup are reordered to generate a reordered merge candidate list, and the reordering is performed according to the cost value based on template matching. The index of the selected merge candidate in the reordered merge candidate list is sent to the decoder. For simplicity, the merge candidates in the last subgroup (but not the first subgroup) are not reordered. During the construction of the merge motion vector candidate list, all zero candidates from the ARM reordering process are excluded. For the normal merge mode and the TM merge mode, the subgroup size is set to 5. For the affine merge mode, the subgroup size is set to 3.

[0265] (1) Cost calculation

[0266] Fig.21 Displays the template in the reference image and a reference sample of the template.

[0267] The template matching cost of the merge candidate during the reordering process is measured by the SAD between the template samples of the current block and its corresponding reference samples. The template includes a set of reconstructed samples adjacent to the current block. The reference samples of the template are located by the motion information of the merge candidate. When the merge candidate uses bidirectional prediction, the reference samples of the template of the merge candidate are also generated by bidirectional prediction, such as Fig.21 shown.

[0268] (2) Refinement of the initial merge candidate list

[0269] When multi-pass DMVR is used to derive refined motion to the initial merge candidate list, only the first pass of multi-pass DMVR (i.e. PU level) is applied for reordering. When template matching is used to derive refined motion, the template size is set to 1. When the block is a flat block and the block width is greater than 2 times the height or a narrow block and the height is greater than 2 times the width, only the upper or left template is used during motion refinement of the TM. The TM is expanded to perform 1 / 16 pixel MVD accuracy. In TM merge mode, the first four merge candidates are reordered using refined motion.

[0270] Fig. 22 A template of a block having sub-block motion using motion information of a sub-block of a current block and a reference sample of the template are displayed.

[0271] For a sub-block-based merge candidate with a sub-block size equal to Wsub×Hsub, the upper template includes multiple sub-templates of size Wsub×1, and the left template includes multiple sub-templates of size 1×Hsub. Fig. 22 As shown, the reference samples of each sub-template are derived using the motion information of the sub-block in the first row and first column of the current block.

[0272] (3) Re-ranking criteria

[0273] During the re-ranking process, if the cost difference between a candidate and its predecessor is less than the lambda value, such as |D1-D2|<λ, the candidate is considered redundant, where D1 and D2 are the costs obtained during the first ARMC ranking and λ is the Lagrangian parameter used in the encoder-side RD criterion.

[0274] The proposed algorithm is defined as follows:

[0275] – Determine the minimum cost difference between a candidate and its predecessor among all candidates in the list

[0276] ●If the minimum cost difference is greater than or equal to λ, the list is considered to be diverse enough and re-ranking stops.

[0277] If this minimum cost difference is less than λ, the candidate is considered redundant and is moved to another position in the list. This further position is the first position where the candidate has sufficient diversity compared to its predecessor.

[0278] – The algorithm stops after a finite number of iterations (if the minimum cost difference is not less than λ).

[0279] The algorithm works for the regular, TM, BM, and affine merge modes. Similar algorithms work for the merged MMVD and signed MVD prediction methods, which also use ARMC for reordering.

[0280] The value of λ is set equal to the λ of the rate-distortion criterion used to select the best merge candidate at the encoder side (for low-delay configurations) and the λ corresponding to another QP (for random access configurations). A set of λ values ​​corresponding to each signal QP offset is provided in the SPS or slice header for QP offsets not present in the SPS.

[0281] (4) Expand to AMVP mode

[0282] The ARMC design also applies to AMVP mode, where AMVP candidates are reordered based on TM cost. For template matching in advanced motion vector prediction (TM-AMVP) mode, an initial AMVP candidate list is constructed and then refined from the TM to construct a refined AMVP candidate list. In addition, if the TM cost of an MVP candidate is greater than a threshold (equal to five times the cost of the first MVP candidate), it will be skipped.

[0283] Note that when surround motion compensation is enabled, MV candidates should be clipped taking the surround offset into account.

[0284] 17. ARMC based on MV candidate type

[0285] The merge candidates of a single candidate type, such as TMVP or non-adjacent MVP (NA-MVP), are reordered based on the ARMC TM cost value. The reordered candidates are then added to the merge candidate list. The TMVP candidate type adds more TMVP candidates with more temporal positions and different inter-frame prediction directions to perform reordering and selection. In addition, the NA-MVP candidate type is further extended with more spatial non-adjacent positions. The target reference picture of the TMVP candidate can be selected from any reference picture in the list according to the scaling factor. The selected reference picture is the picture with the scaling factor closest to 1.

[0286] 18. TM-based MMVD and affine MMVD reordering

[0287] Fig.23 Additional directions along the k×π / 8 diagonal angle are shown (red positions are used in the anchors).

[0288] The MMVD offset extends the MMVD and affine MMVD modes. Fig.23 As shown, additional refinement positions along the k×π / 8 diagonal are added, increasing the number of directions from 4 to 16. Secondly, all possible MMVD refinement positions (16×6) for each basis candidate are reordered based on the SAD cost between the template (one row above the current block and one column to the left) of each refinement position and its reference. Finally, the first 1 / 8 refinement positions with the smallest template SAD cost are retained as available positions for MMVD index encoding. The MMVD index is binarized by a rice code with parameter equal to 2. The affine MMVD reordering is extended, where additional refinement positions along the k×π / 4 diagonal are added. After reordering, the first 1 / 2 refinement positions with the smallest template SAD cost are retained.

[0289] The first N motion candidates in the candidate list before reordering are used as the base candidates for MMVD and affine MMVD. For MMVD, N is equal to 3; for affine MMVD, N is equal to [1,3], depending on the neighbor block affine flag. Two ways of adding MMVD offsets are allowed, including "both sides" and "one side", depending on whether the offset of the other reference picture list is mirrored or directly set to zero. Which way is applied to a block depends on the TM cost.

[0290] II. Proposed Method

[0291] In this document, when not specified, the size is given in units of luma samples in coding units (CUs). The size of a block is given in units of samples in that block. Also, in this document, block sizes are expressed as WxH, where W is width and H is height.

[0292] Typically, the minimum decoding block is defined in the existing video codec; for example, when the input video uses 4:2:0 chroma sampling format, a 4x4 luma block is the minimum block size for regular inter mode in ECM6.0, and a 2x2 chroma block is the minimum block size for regular inter mode in VVC.

[0293] In ECM6.0, only CUs with width and height greater than or equal to 8 are allowed to be used for affine merging, and only CUs with width and height greater than or equal to 8 are allowed to be used for affine AMVP mode. It should be noted that in order to perform affine motion compensation for each affine AMVP and affine merge codec block, each block is divided into sub-blocks (e.g. 4x4), and the motion vector of each sub-block is derived according to the affine motion model associated with the current CU. All sub-blocks of the current CU perform motion compensation to obtain the predictor. It should also be noted that the chroma block is also divided into 4x4 chroma sub-blocks.

[0294] In the first embodiment, the block size of the affine merge and / or affine AMVP mode is consistent with the block size allowed by the conventional merge and / or conventional AMVP mode. By eliminating the size restriction, the parsing of the related syntax elements can be simplified.

[0295] To enable affine merging and / or AMVP mode on small blocks (e.g., 4x4 luma blocks), the corresponding 2x2 chroma blocks may be smaller than a predefined subblock size (e.g., 4x4). In the event that the chroma block is smaller than the predetermined chroma subblock size, a smaller chroma block subblock size (e.g., 2x2) is used instead.

[0296] In the proposed scheme, when affine merging and / or AMVP mode is enabled / allowed on a small block (e.g., 4xN / Nx4 CU) and the chroma sampling format is 4:2:0, the corresponding chroma block sizes are 2x(N / 2) and (N / 2)x2. Since the original sub-block size of 4x4 for the chroma component is too large for this case, the size of the chroma sub-block for which affine motion compensation is performed should also be modified accordingly. In one embodiment, 2x2 blocks are always used instead of 4x4 blocks as sub-blocks for affine sub-block based motion compensation.

[0297] In yet another embodiment, when the affine motion model is enabled on a 4xN / Nx4 CU, the 2x2 block is used as a chroma sub-block for motion compensation based on an affine sub-block. Otherwise, when the affine motion model is enabled on a CU whose width and height are both greater than 4, the 4x4 block is still used as a chroma sub-block for motion compensation based on an affine sub-block. In one example, when the affine motion model is enabled on a 4xN CU (width = 4, height = N) AMVP mode, the current CU / block is inferred to use a 4-parameter affine model (no syntax needs to be sent to indicate which affine model is used). Only the upper left and lower left control point MVs (CPMVs) need to be sent. In another example, when the affine motion model is enabled in an Nx4 CU (width = N, height = 4) AMVP mode, the current CU / block is inferred to use a 4-parameter affine model. Only the upper left and upper right control point MVs (CPMVs) need to be sent. In another example, when the affine motion model is enabled in 4xN CU (width=4, height=N) AMVP mode, the current CU / block is inferred to use a 6-parameter affine model, but only the top-left and bottom-left control point MVs (CPMVs) need to be sent. In another example, when the affine motion model is enabled in Nx4 or 4xN CUs AMVP mode, the current CU / block can choose to use a 4-parameter or 6-parameter affine model, but only two CPMVs need to be sent (e.g., the top-left and top-right corners of Nx4 CUs, the top-left and bottom-left corners of 4xN CUs). If the 6-parameter affine model is selected, the third CPMV will be derived from the two sent CPMVs.

[0298] In yet another embodiment, when the affine motion model is enabled on a 4x4 CU, a 2x2 block is used as the chroma sub-block for motion compensation based on the affine sub-block. Otherwise, when the affine motion model is enabled on a 4xN CU and N is greater than 4, a 2x4 block is used as the chroma sub-block for motion compensation based on the affine sub-block. Otherwise, when the affine motion model is enabled on an Nx4 CU with N greater than 4, a 4x2 block is used as the chroma sub-block for motion compensation based on the affine sub-block.

[0299] In the proposed scheme, when affine merging and / or AMVP mode is enabled / allowed on small blocks such as 4xN / Nx4 CUs and the chroma sampling format is 4:2:2, the corresponding chroma block sizes are 2x(N) and (N / 2)x4. Since the original sub-block size of 4x4 for the chroma component is too large for this case, the size of the chroma sub-block for which affine motion compensation is performed should also be modified accordingly. In one embodiment, 2x4 blocks are always used instead of 4x4 blocks as sub-blocks for affine sub-block based motion compensation.

[0300] In another embodiment, when the affine motion model is enabled on a 4xN / Nx4 CU, the 2x4 block is used as a chroma sub-block for motion compensation based on the affine sub-block. Otherwise, when the affine motion model is enabled on a CU with a width and height greater than 4, the 4x4 block is still used as a chroma sub-block for motion compensation based on the affine sub-block.

[0301] In yet another embodiment, when the affine motion model is enabled on a 4x4 CU, a 2x4 block is used as the chroma sub-block for motion compensation based on the affine sub-block. Otherwise, when the affine motion model is enabled on a 4xN CU with N greater than 4, a 2x4 block is used as the chroma sub-block for motion compensation based on the affine sub-block. Otherwise, when the affine motion model is enabled on an Nx4 CU and N is greater than 4, a 4x4 block is used as the chroma sub-block for motion compensation based on the affine sub-block.

[0302] In the proposed scheme, when affine merging and / or AMVP mode is enabled / allowed on small blocks such as 4xN / Nx4 CUs and the chroma sampling format is 4:4:4, the corresponding chroma block sizes are 4xN and Nx4. Since the original sub-block size of 4x4 for the chroma component is too large for this case, the size of the chroma sub-block used to perform affine motion compensation should also be modified accordingly. In one embodiment, 2x2 blocks are always used instead of 4x4 blocks as sub-blocks for affine sub-block based motion compensation.

[0303] In another embodiment, when the affine motion model is enabled on a 4xN / Nx4 CU, a 2x2 block is used as the chroma sub-block for motion compensation based on the affine sub-block. Otherwise, when the affine motion model is enabled on a CU with a width and height greater than 4, a 4x4 block is still used as the chroma sub-block for motion compensation based on the affine sub-block.

[0304] In yet another embodiment, when the affine motion model is enabled on a 4x4 CU, a 2x2 block is used as the chroma sub-block for motion compensation based on the affine sub-block. Otherwise, when the affine motion model is enabled on a 4xN CU with N greater than 4, a 2x4 block is used as the chroma sub-block for motion compensation based on the affine sub-block. Otherwise, when the affine motion model is enabled on an Nx4 CU and N is greater than 4, a 4x2 block is used as the chroma sub-block for motion compensation based on the affine sub-block. In another embodiment, the chroma sub-block size may be min(4,N / 2) x min(4,N / 2).

[0305] In one embodiment, the sub-block size depends on the affine motion model. That is, the MV difference between two sub-blocks cannot be greater than a predetermined threshold. In the supported sub-block size set, the largest sub-block size that can meet this constraint is selected as the unit of motion compensation based on affine sub-blocks. For example, the supported sub-block size set includes 16x16, 8x8, 4x4 and 2x2. If the MV difference between two 16x16 sub-blocks is greater than a predetermined threshold and the MV difference between two 8x8 sub-blocks is less than or equal to a predetermined threshold, 8x8 is selected as the sub-block size. In another case, if the MV difference between two 4x4 sub-blocks is greater than a predetermined threshold and the MV difference between two 2x2 sub-blocks is less than or equal to a predetermined threshold, 2x2 is selected as the sub-block size. In another embodiment, if the MV difference between the two smallest sub-blocks is greater than a predetermined threshold, a fallback mechanism is used instead of affine motion compensation. The method can be applied only to the luminance component, only to the chrominance component, or to both the luminance and chrominance components. The constraint can be the MV difference between two sub-blocks, the difference between two control point MVs considering the CU width / height, or the difference between two control point MVs not considering the CU width / height.

[0306] In another embodiment, the sub-block size depends on the CU size. For example, the sub-block size will be max(p, min(height, width) / q), where p and q are non-zero integers. In another example, it can be max(p, width / q) x max(p, height / q), where p and q are non-zero integers.

[0307] In yet another embodiment, the block size depends on the number of affine parameters. For example, when the CU is encoded and decoded with a 4-parameter affine model, the block size is 4x4. Otherwise (the CU is encoded and decoded with a 6-parameter affine model), the block size is 2x2. In another example, when the CU is encoded and decoded with a 6-parameter affine model, the block size is 4x4. Otherwise (the CU is encoded and decoded with a 4-parameter affine model), the block size is 4x4. In another embodiment, if a 6-parameter affine model is selected and the CU size is 4xN or Nx4, a 2x4 or 4x2 luma block is used instead of a 4x4 luma block. In another embodiment, if a 6-parameter affine model is selected and the CU size is 4xN or Nx4, a 2x2 luma block is used instead of a 4x4 luma block.

[0308] In VVC and ECM, the mv of each chroma subblock is directly derived from the corresponding luma subblock. In the proposed scheme, when the subblock size of the luma block is 4x4 and the subblock size of the chroma block is 2x4 / 4x2, the mv of each chroma subblock is derived from the corresponding luma subblock. In one embodiment, the mv of each chroma subblock is derived from the average of the mvs of the two luma subblocks. Fig.24 As shown, when the input video is 4:2:0 chroma sampling, the 2x4 chroma sub-block corresponds to two 4x4 luminance sub-blocks, and the MV of the 2x4 chroma sub-block is derived from the average value of the MVs of the two corresponding luminance sub-blocks.

[0309] In VVC and ECM-6.0, a 6-parameter or 4-parameter affine model can be adaptively selected for the affine AMVP codec block. Since the 6-parameter affine AMVP mode requires more bits to represent motion information, and small blocks may not require complex motion models (such as 6-parameter affine models), it is proposed not to allow small blocks to use the 6-parameter affine AMVP mode. It should be noted that the small block size affine merge mode still allows the use of the 6-parameter affine motion model, because the 6-parameter affine merge mode does not require additional bits compared to the 4-parameter affine merge mode. Small blocks can be predefined as CU / PU / blocks with a width less than or equal to L and a height less than or equal to M, where L and M can be any non-negative integers. L and M can be predefined or sent in the bitstream as syntax elements of the video codec standard; and syntax elements can be sent at the sequence level (such as sequence parameter set), picture level (picture parameter set), slice level (slice header), CTU level, CU level, or block level.

[0310] It should be noted that in existing codecs, the minimum CU of the conventional inter-frame mode is 4x4. When the conventional inter-frame mode size allows a smaller size, such as 2x2 or 1x1, all the methods mentioned in the present invention can also be applied and the corresponding block size is adjusted accordingly.

[0311] In order to achieve better affine motion mode prediction efficiency, the present invention also proposes to enable PROF for chroma blocks.

[0312] Since in the proposed scheme, small blocks are used for affine motion mode, it is recommended to use 4-tap and / or 2-tap interpolation filters to compensate for the luminance and chrominance sub-blocks of the affine motion mode. In one embodiment, the 2-tap filter can be a bilinear interpolation filter. In another embodiment, the 2-tap interpolation filter and / or the 4-tap interpolation filter are only used for smaller CUs, while other CUs can still use the interpolation filter used in the conventional inter-frame mode. A smaller CU can be defined as a CU with a height <= M and a width <= N, where M and N are non-negative integers.

[0313] In another embodiment, ARMC is not allowed for affine coded CU / blocks whose size, width and / or height is equal to or less than a predetermined threshold. For example, ARMC is not allowed for affine coded CU / blocks whose size is equal to 4x4, 4x2 or 2x4.

[0314] In another embodiment, for affine-coded CU / blocks whose size, width and / or height is equal to or less than a predetermined threshold, affine MMVD is not allowed. For example, for affine-coded CU / blocks whose size is equal to 4x4, 4x2 or 2x4, affine MMVD is not allowed.

[0315] Any of the above-mentioned methods can be implemented in an encoder and / or a decoder. For example, any of the above-mentioned methods can be implemented in an inter-frame prediction module of an encoder and / or a decoder. Alternatively, any of the above-mentioned methods can be implemented as a circuit coupled to an inter-frame prediction module of an encoder and / or a decoder.

[0316] Fig.25 A block diagram of a video encoder is shown, which may include or be coupled to a module or circuit that implements the methods and techniques described in the present disclosure. The video encoder may be implemented based on the VVC standard, the HEVC standard, or any other video coding standard. When the inter-frame mode is used, the intra / inter-frame prediction unit 110 generates an inter-frame prediction based on motion estimation (ME) / motion compensation (MC). When the intra-frame mode is used, the intra / inter-frame prediction unit 110 generates an intra-frame prediction. The intra / inter-frame prediction data (i.e., the intra / inter-frame prediction signal) is provided to the subtractor 115 to form a prediction error, also known as a "residual" or "residual", by subtracting the intra / inter-frame prediction signal from the signal associated with the input frame. The process of generating intra / inter-frame prediction data is referred to as a prediction process in the present disclosure. The prediction error (i.e., the residual) is then processed by a transform (T) and then quantized (Q) (T+Q, 120). The entropy coding unit 125 then encodes and decodes the transformed and quantized residual for inclusion in a video bitstream corresponding to the compressed video data.

[0317] The bitstream associated with the transform coefficients is then packaged together with side information such as motion, codec mode, and other information associated with the image area. The side information can also be compressed by entropy coding to reduce the required bandwidth. Since the reconstructed frame can be used as a reference frame for inter-frame prediction, one or more reference frames must be reconstructed at the encoder end. Therefore, the transformed and quantized residuals are processed by inverse quantization (IQ) and inverse transformation (IT) (IQ+IT, 130) to recover the residuals. The reconstructed residuals are then added back to the intra / inter prediction data at the reconstruction unit (REC) 135 to reconstruct the video data. The process of adding the reconstructed residuals to the intra / inter prediction signal is referred to as a reconstruction process in the present disclosure. The output frame of the reconstruction process is referred to as a reconstructed frame.

[0318] In order to reduce artifacts in the reconstructed frame, loop filters are used, including but not limited to a deblocking filter (DF) 140, a sample adaptive offset (SAO) 145, and an adaptive loop filter (ALF) 150. In the present disclosure, DF, SAO, and ALF are all labeled as filtering processes. The filtered reconstructed frame at the output of all filtering processes is referred to as a decoded frame in the present disclosure. The decoded frame is stored in a frame buffer 155 and is used to predict other frames.

[0319] Fig.26 A block diagram of a video decoder is shown, which may include or be coupled to a module or circuit that implements the methods and techniques described in the present disclosure. The video decoder may be implemented based on the VVC standard, the HEVC standard, or any other video coding standard. Since the encoder includes a local decoder for reconstructing video data, many decoder components have been used in the encoder in addition to the entropy decoder. On the decoder side, the entropy decoding unit 226 is used to recover the encoded symbols or syntax from the bitstream. The process of generating a reconstructed residual from an input bitstream is referred to as a residual decoding process in the present disclosure. The prediction process for generating intra / inter prediction data is also applied to the decoder side, but the intra / inter prediction unit 211 is different from the intra / inter prediction unit 110 on the encoder side because inter prediction only requires the use of motion information derived from the bitstream to perform motion compensation. In addition, the adder 215 is used to add the reconstructed residual to the intra / inter prediction data.

Claims

1. A method for performing affine motion compensated prediction (MCP) in a video decoder, comprising: Selecting an affine MCP pattern from the affine MCP pattern candidate set; as well as Based on the selected affine MCP mode, affine motion compensated prediction is performed in the video decoder, wherein For codec units whose width and height are both greater than the first threshold, the first affine MCP mode candidate set is applied, For coding units whose width and height are at least one of which is not greater than the first threshold, applying a second affine MCP mode candidate set, and The first affine MCP mode candidate set has more affine MCP mode candidates than the second affine MCP mode candidate set.

2. The method for performing affine motion compensated prediction (MCP) in a video decoder according to claim 1, characterized in that The first affine MCP pattern candidate set includes: 6-parameter affine advanced motion vector prediction (AMVP) mode, which means using a 6-parameter affine motion model in affine AMVP mode, and 4-parameter affine AMVP mode, indicating that a 4-parameter affine motion model is used in the affine AMVP mode, and The second affine MCP mode candidate set includes only one of the 6-parameter affine AMVP mode and the 4-parameter affine AMVP mode.

3. The method for performing affine motion compensated prediction (MCP) in a video decoder according to claim 2, characterized in that When the second affine MCP mode candidate set includes the 6-parameter affine AMVP mode, The motion vectors of the two control points relative to the 6-parameter affine motion model are sent, and The motion vector of a third control point relative to the 6-parameter affine motion model is derived from the transmitted motion vectors relative to the two control points.

4. The method for performing affine motion compensated prediction (MCP) in a video decoder according to claim 1, characterized in that The first affine MCP mode candidate set includes an affine merged motion vector difference (MMVD) mode, The second affine MCP mode candidate set does not include the affine MMVD mode.

5. The method for performing affine motion compensated prediction (MCP) in a video decoder according to claim 1, characterized in that The first threshold is a predetermined value, or is sent in the form of a syntax element at a sequence level, a picture level, a slice level, a codec tree unit level, a codec unit level, or a block level.

6. The method for performing affine motion compensated prediction (MCP) in a video decoder according to claim 1, characterized in that The first threshold is set to 4.

7. The method for performing affine motion compensated prediction (MCP) in a video decoder according to claim 1, characterized in that When the 4:2:0 chroma sampling format is adopted, the affine motion compensation prediction is performed on the encoding and decoding unit with a 4x4 sub-block size of the luminance component and a 2x2 sub-block size of the chroma component.

8. The method for performing affine motion compensated prediction (MCP) in a video decoder according to claim 1, characterized in that When using the 4:2:0 chroma sampling format, For a codec unit of size equal to 4xN or Nx4, where N is a power of 2 and N≥4, performing the affine motion compensation prediction on the codec unit with a sub-block size of 4x4 for multiple luminance components and a sub-block size of 2x2 for multiple chrominance components; as well as For a codec unit whose width is greater than 4 and whose height is greater than 4, the affine motion compensation prediction is performed on the codec unit with a sub-block size of 4x4 for multiple luminance components and multiple chrominance components.

9. The method for performing affine motion compensated prediction (MCP) in a video decoder as claimed in claim 1, characterized in that in, When using the 4:2:0 chroma sampling format, For a codec unit of size 4x4, the affine motion compensation prediction is performed on the codec unit with a sub-block size of multiple luminance components of 4x4 and a sub-block size of multiple chrominance components of 2x2; For a codec unit of size 4xN, where 4 is the width, N is the height, N is a power of 2, and N>4, performing the affine motion compensation prediction on the codec unit with a sub-block size of 4x4 for multiple luminance components and a sub-block size of 2x4 for multiple chrominance components; For a codec unit of size Nx4, where N is width, 4 is height, N is a power of 2, and N>4, performing the affine motion compensation prediction on the codec unit with a sub-block size of 4x4 for multiple luminance components and a sub-block size of 4x2 for multiple chrominance components; as well as For a codec unit whose width is greater than 4 and whose height is greater than 4, the affine motion compensation prediction is performed on the codec unit with a sub-block size of 4x4 for multiple luminance components and multiple chrominance components.

10. The method for performing affine motion compensated prediction (MCP) in a video decoder according to claim 1, characterized in that When the 4:2:2 chroma sampling format is adopted, the affine motion compensation prediction is performed on the encoding and decoding unit with a sub-block size of 4x4 for multiple luminance components and a sub-block size of 2x4 for multiple chroma components.

11. The method for performing affine motion compensated prediction (MCP) in a video decoder according to claim 1, characterized in that When using 4:2:2 chroma sampling format, For a codec unit of size 4xN or Nx4, where N is a power of 2 and N≥4, the codec unit is subjected to the affine motion compensation prediction with a sub-block size of 4x4 for multiple luminance components and a sub-block size of 2x4 for multiple chrominance components; For a codec unit whose width is greater than 4 and whose height is greater than 4, the affine motion compensation prediction is performed on the codec unit with the sub-block size of multiple luminance components and multiple chrominance components both being 4x4.

12. The method for performing affine motion compensated prediction (MCP) in a video decoder according to claim 1, characterized in that When using 4:2:2 chroma sampling format, For a codec unit of size 4xN, where 4 is the width, N is the height, N is a power of 2, and N≥4, performing the affine motion compensation prediction on the codec unit with a sub-block size of 4x4 for multiple luminance components and a sub-block size of 2x4 for multiple chrominance components; For a codec unit of size Nx4, where N is width, 4 is height, N is a power of 2, and N>4, performing the affine motion compensation prediction on the codec unit with a sub-block size of 4x4 for multiple luminance components and a sub-block size of 4x4 for multiple chrominance components; For a codec unit with a width greater than 4 and a height greater than 4, the affine motion compensation prediction unit is performed on the codec unit with a sub-block size of multiple luminance components and multiple chrominance components both being 4x4.

13. The method for performing affine motion compensated prediction (MCP) in a video decoder according to claim 1, characterized in that When the 4:4:4 chroma sampling format is adopted, the affine motion compensation prediction is performed on the encoding and decoding unit with a sub-block size of 4x4 for multiple luminance components and a sub-block size of 2x2 for multiple chroma components.

14. The method for performing affine motion compensated prediction (MCP) in a video decoder according to claim 1, characterized in that When using 4:4:4 chroma sampling format, For a codec unit of size 4xN or Nx4, where N is a power of 2 and N≥4, performing the affine motion compensation prediction on the codec unit with a sub-block size of 4x4 for multiple luminance components and a sub-block size of 2x2 for multiple chrominance components; For a codec unit whose width is greater than 4 and whose height is greater than 4, the affine motion compensation prediction is performed on the codec unit with the sub-block size of multiple luminance components and multiple chrominance components both being 4x4.

15. The method for performing affine motion compensated prediction (MCP) in a video decoder according to claim 1, characterized in that When using 4:4:4 chroma sampling format, For a codec unit of size 4x4, the affine motion compensation prediction is performed on the codec unit with a sub-block size of multiple luminance components of 4x4 and a sub-block size of multiple chrominance components of 2x2; For a codec unit of size 4xN, where 4 is the width, N is the height, N is a power of 2, and N>4, performing the affine motion compensation prediction on the codec unit with a sub-block size of 4x4 for multiple luminance components and a sub-block size of 2x4 for multiple chrominance components; For a codec unit of size Nx4, where N is width, 4 is height, N is a power of 2, and N>4, the codec unit is subjected to the affine motion compensation prediction coding unit with a sub-block size of 4x4 for multiple luminance components and a sub-block size of 4x2 for multiple chrominance components; For a codec unit whose width is greater than 4 and whose height is greater than 4, the affine motion compensation prediction is performed on the codec unit with the sub-block size of multiple luminance components and multiple chrominance components both being 4x4.

16. The method for performing affine motion compensated prediction (MCP) in a video decoder according to claim 1, characterized in that The method further includes: determining whether to allow the affine motion compensation prediction for the codec unit based on the size of the codec unit, wherein When the width and the height of the codec unit are both equal to or greater than a second threshold, it is determined that the affine motion compensation prediction is allowed for the codec unit. When at least one of the width and the height of the codec unit is smaller than the second threshold, it is determined that the affine motion compensation prediction for the codec unit is not allowed.

17. The method for performing affine motion compensated prediction (MCP) in a video decoder according to claim 16, characterized in that The second threshold is a predetermined value, or is represented by a syntax element in a bitstream at a sequence level, a picture level, a slice level, a codec tree unit level, a codec unit level, or a block level.

18. The method for performing affine motion compensated prediction (MCP) in a video decoder according to claim 16, characterized in that The second threshold is set to 4, 2 or 1.

19. A method for performing affine motion compensated prediction (MCP) in a video encoder, comprising: Selecting an affine MCP pattern from the affine MCP pattern candidate set; as well as Based on the selected affine MCP mode, the affine motion compensated prediction is performed in the video encoder, wherein For codec units whose width and height are both greater than the first threshold, the first affine MCP mode candidate set is applied, For codecs whose width and height are at least one less than the first threshold, applying the second affine MCP mode candidate set, and The first affine MCP mode candidate set has more affine MCP mode candidates than the second affine MCP mode candidate set.

20. An apparatus for performing affine motion compensated prediction (MCP) in a video decoder, comprising circuitry configured to: Select an affine MCP pattern from the affine MCP pattern candidate set; as well as Affine motion compensated prediction is performed in a video decoder based on the selected affine MCP mode, wherein: For codec units whose width and height are both greater than the first threshold, applying the first affine MCP mode candidate set; For a codec unit whose width and height are at least one not greater than the first threshold, applying a second affine MCP mode candidate set; and The first affine MCP mode candidate set has more affine MCP mode candidates than the second affine MCP mode candidate set.

21. An apparatus for performing affine motion compensated prediction (MCP) in a video decoder, comprising circuitry configured to: Selecting an affine MCP pattern from the affine MCP pattern candidate set; and The affine motion compensated prediction is performed in the video decoder based on the selected affine MCP mode, wherein: For codec units whose width and height are both greater than the first threshold, applying the first affine MCP mode candidate set; For a codec unit whose at least one of width and height is not greater than the first threshold and whose width is greater than or equal to the height, applying a second affine MCP mode candidate set; and For a codec unit whose at least one of width and height is not greater than the first threshold and whose width is smaller than the height, a third affine MCP mode candidate set is applied.

22. The apparatus for performing affine motion compensated prediction (MCP) in a video decoder according to claim 21, characterized in that: The first affine MCP pattern candidate set includes: 6-parameter affine advanced motion vector prediction (AMVP) mode, indicating the use of a 6-parameter affine motion model in affine AMVP mode, and 4-parameter affine AMVP mode, indicating that a 4-parameter affine motion model is used in the affine AMVP mode, where motion vectors relative to the top left and top right control points are sent, The second affine MCP pattern candidate set includes: 4-parameter affine AMVP mode, indicating that a 4-parameter affine motion model is used in the affine AMVP mode, where motion vectors relative to the top left and top right control points are sent, and The third affine MCP pattern candidate set includes: 4-parameter affine AMVP mode, indicating that a 4-parameter affine motion model is used in the affine AMVP mode, in which motion vectors relative to the upper left and upper right control points are sent.

Images