Method and apparatus for chroma prediction in video coding

Abstract

A method for video encoding and decoding includes: receiving data of a current block of pixels to be encoded or decoded as a current picture of a video, where the current block includes at least one chroma block; and encoding or decoding the current block by a target inter prediction mode, comprising: obtaining a motion compensation prediction of a chroma sample contained in the at least one chroma block; obtaining at least one cross-component prediction of the chroma sample; and determining an inter prediction of the chroma sample by jointly considering the motion compensated prediction and the at least one cross-component prediction.

Description

[0001]

Cross-reference

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 439,257, filed on January 16, 2023.

Technical field

[0003] The present invention relates to video coding and decoding, and more particularly, to a video coding and decoding method for determining inter-frame prediction of chroma samples by jointly considering (e.g., mixing) motion compensated prediction (i.e., inter-frame predicted chroma samples) and at least one cross-component prediction (i.e., one or more cross-component intra-frame predicted chroma samples) and related devices. [Background Technology]

[0004] Traditional video codec standards typically use block-based coding techniques to exploit spatial and temporal redundancy. For example, the basic approach is to divide the entire source image into multiple blocks, perform intra-frame and inter-frame prediction on each block, transform the residual of each block, and then quantize and entropy encode it. Furthermore, a reconstructed image is generated during the codec loop to provide reference data for encoding and decoding subsequent blocks. For some video codecs, loop filters may be used to enhance the image quality of the reconstructed frames.

[0005] A video decoder is used to perform the inverse of the video encoding operations performed by a video encoder. For example, a video decoder may have multiple processing circuits, such as entropy decoding circuitry, intra-frame prediction circuitry, motion compensation circuitry, inverse quantization circuitry, inverse transform circuitry, reconstruction circuitry, and a loop filter.

[0006] Typically, for inter-frame blocks, chroma prediction may not be as accurate as luma prediction. For example, the motion vectors of chroma samples are inherited from luma samples in the same codec unit. In other words, chroma samples may not have their own motion vectors. Therefore, an innovative inter-frame chroma prediction scheme is needed to improve inter-frame chroma prediction accuracy and codec efficiency. [Summary of the invention]

[0007] One of the objects of the present invention is to provide a video coding method and related devices for determining inter-frame prediction of chroma samples by jointly considering (e.g., mixing) motion compensated prediction (i.e., chroma samples predicted inter-frame) and at least one cross-component prediction (i.e., chroma samples predicted intra-frame across one or more components).

[0008] According to a first aspect of the present invention, an exemplary method for video encoding and decoding is disclosed. The exemplary method includes: receiving data of a current pixel block to be encoded or decoded as a current picture of a video, wherein the current block includes at least one chroma block; and encoding or decoding the current block using a target inter-frame prediction mode, including: obtaining a motion compensated prediction of chroma samples contained in the at least one chroma block; obtaining at least one cross-component prediction of the chroma samples; and determining an inter-frame prediction for the chroma samples by jointly considering the motion compensated prediction and the at least one cross-component prediction.

[0009] According to a second aspect of the present invention, an exemplary video encoder is disclosed. The exemplary video encoder includes a video data memory and encoding circuitry. The video data memory is configured to receive data of a current pixel block to be encoded as a current picture of a video, wherein the current block includes at least one chroma block. The encoding circuitry is configured to encode the current block using a target inter-frame prediction mode. The encoding circuitry includes motion compensation circuitry and inter-frame chroma prediction circuitry. The motion compensation circuitry is configured to obtain a motion compensated prediction of chroma samples contained in the at least one chroma block. The inter-frame chroma prediction circuitry is configured to obtain at least one cross-component prediction of the chroma sample and determine an inter-frame prediction for the chroma sample by jointly considering the motion compensated prediction and the at least one cross-component prediction.

[0010] According to a third aspect of the present invention, an exemplary video decoder is disclosed. The exemplary video decoder includes a video data memory and a decoding circuit. The video data memory is configured to receive data of a current pixel block to be decoded as a current picture of a video, wherein the current block includes at least one chroma block. The decoding circuit is configured to decode the current block using a target inter-frame prediction mode. The decoding circuit includes motion compensation circuitry and inter-frame chroma prediction circuitry. The motion compensation circuitry is configured to obtain a motion compensated prediction of chroma samples contained in the at least one chroma block. The inter-frame chroma prediction circuitry is configured to obtain at least one cross-component prediction of the chroma sample and determine an inter-frame prediction for the chroma sample by jointly considering the motion compensated prediction and the at least one cross-component prediction.

[0011] These and other objects of the present invention will become apparent to those of ordinary skill in the art after reading the following detailed description of the preferred embodiments.

Brief Description of the Drawings

[0012] Figure 1 FIG. 4 is a schematic diagram illustrating four reference lines adjacent to a prediction block according to an embodiment of the present invention.

[0013] Figure 2 FIG. 4 is a schematic diagram illustrating sample positions for deriving α and β according to an embodiment of the present invention.

[0014] Figure 3 FIG. 2 is a schematic diagram illustrating classifying neighboring samples into two groups according to an embodiment of the present invention.

[0015] Figure 4 FIG. 4 is a schematic diagram illustrating the effect of the slope adjustment parameter “u” according to an embodiment of the present invention.

[0016] Figure 5 is a schematic diagram illustrating the spatial portion of a convolution filter according to an embodiment of the present invention.

[0017] Figure 6 FIG. 4 is a schematic diagram illustrating a reference area (and its filling) for deriving filter coefficients according to an embodiment of the present invention.

[0018] Figure 7 Schematic diagram illustrating 16 gradient modes of GLM according to an embodiment of the present invention.

[0019] Figure 8 This is a schematic diagram illustrating that IBC in VVC only allows the reconstruction of a predefined area including the current CTU area and some left CTU areas according to an embodiment of the present invention.

[0020] Figure 9 FIG. 4 is a schematic diagram illustrating HoG calculation from a template with a width of 3 pixels according to an embodiment of the present invention.

[0021] Figure 10 3 is a schematic diagram illustrating prediction fusion by weighted averaging two HoG modes and a plane according to an embodiment of the present invention.

[0022] Figure 11 FIG. 4 is a schematic diagram illustrating a template and a reference sample used in TIMD according to an embodiment of the present invention.

[0023] Figure 12 FIG. 4 is a schematic diagram illustrating spatial merging candidate positions according to an embodiment of the present invention.

[0024] Figure 13 FIG. 2 is a schematic diagram illustrating candidate pairs considered for spatial merging candidate redundancy check according to an embodiment of the present invention.

[0025] Figure 14 FIG. 4 is a schematic diagram illustrating spatially neighboring blocks for deriving spatial merging candidates according to an embodiment of the present invention.

[0026] Figure 15 is a diagram illustrating motion vector scaling of temporal merging candidates.

[0027] Figure 16 2 is a schematic diagram illustrating candidate positions of temporal merging candidates, C0 and C1, according to an embodiment of the present invention.

[0028] Figure 17FIG. 2 is a schematic diagram illustrating top and left neighboring blocks for CIIP weight derivation according to an embodiment of the present invention.

[0029] Figure 18 FIG. 2 is a schematic diagram illustrating an example of GPM partitions grouped at the same angle according to an embodiment of the present invention.

[0030] Figure 19 FIG. 1 is a schematic diagram illustrating single-prediction MV selection in a geometric partitioning mode according to an embodiment of the present invention.

[0031] Figure 20 FIG. 1 is a schematic diagram illustrating an example of generating a mixing weight W0 of a GPM according to an embodiment of the present invention.

[0032] Figure 21 FIG. 4 is a schematic diagram illustrating a ramp function of a GPM weight derived based on a displacement from a sample position to a GPM partition boundary and a fixed threshold τ according to an embodiment of the present invention.

[0033] Figure 22 FIG. 4 is a schematic diagram illustrating GPM mixing in the current ECM 4.0 according to an embodiment of the present invention.

[0034] Figure 23 FIG. 2 is a schematic diagram illustrating an example of generating a warp weight w0 using a geometric partitioning mode according to an embodiment of the present invention.

[0035] Figure 24 FIG. 1 is a diagram illustrating a GPM with inter-frame and intra-frame prediction (or GPM intra) according to an embodiment of the present invention.

[0036] Figure 25 is a diagram illustrating a spatial GPM (SGPM) consisting of one partition mode and two associated intra prediction modes according to an embodiment of the present invention.

[0037] Figure 26 FIG. 4 is a schematic diagram illustrating SGPM signaling according to an embodiment of the present invention.

[0038] Figure 27 FIG. 4 is a schematic diagram illustrating a template shape and an expansion weight according to an embodiment of the present invention.

[0039] Figure 28 FIG. 2 is a schematic diagram illustrating a linear prediction method for linearly predicting chrominance samples from luma samples according to an embodiment of the present invention.

[0040] Figure 29 FIG. 2 is a schematic diagram illustrating an example of sub-block decision making according to an embodiment of the present invention.

[0041] Figure 30 The conventional merge mode inter-frame predictor uses the above R x,-1 and left R -1,ySchematic diagram of the reconstruction sample for optimization.

[0042] Figure 31 It is a schematic diagram illustrating that the boundary matching cost of a candidate mode refers to a discontinuity measurement (including top boundary matching and / or left boundary matching) between a current prediction (prediction samples within a current block generated by the candidate mode) and a neighboring reconstruction (reconstructed samples within one or more neighboring blocks) according to an embodiment of the present invention.

[0043] Figure 32 FIG. 4 is a schematic diagram illustrating a scanning order for adding information of spatial neighbor blocks to a list according to an embodiment of the present invention.

[0044] Figure 33 FIG. 4 is a schematic diagram illustrating an example of the lower right area of ​​the current chroma block according to an embodiment of the present invention.

[0045] Figure 34 is a block diagram illustrating a video encoder supporting the proposed inter-frame chroma prediction mode according to an embodiment of the present invention.

[0046] Figure 35 is a block diagram illustrating a video decoder supporting the proposed inter-frame chroma prediction mode according to an embodiment of the present invention.

[0047] Figure 36 is a flowchart illustrating a video encoding and decoding method according to an embodiment of the present invention. [Specific implementation method]

[0048] Certain terms are used in the following description and claims to refer to specific components. As those skilled in the art will appreciate, electronic device manufacturers may refer to a component by different names. This document does not intend to distinguish between components that differ in name but function identically. In the following description and claims, the terms "include" and "comprising" are used in an open-ended manner and should, therefore, be interpreted as meaning "including, but not limited to..." Furthermore, the term "connected" is intended to mean either an indirect or direct electrical connection. Thus, if one device is coupled to another, that connection may be through a direct electrical connection or an indirect electrical connection via other devices and connections.

[0049] abbreviation

[0050] CCLM: Cross Component Linear Model

[0051] MMLM: Multiple model CCLM

[0052] LM: Linear Model

[0053] MV: Motion Vector

[0054] CU: Coding Unit

[0055] CB: Coding Block

[0056] CTU: Coding Tree Unit

[0057] VVC: Versatile Video Coding

[0058] SAO: Sample Adaptive Offset

[0059] ALF: Adaptive Loop Filtering

[0060] IBC: Intra Block Copy

[0061] MRL: Multiple reference line

[0062] ISP: Intra sub-partitions

[0063] MIP: Matrix weighted intra prediction

[0064] JCCR: Joint Coding of Chroma Residual

[0065] Intra-mode codec with 67 intra-prediction modes

[0066] To capture arbitrary edge directions present in natural videos, the number of directional intra modes in VVC is extended from 33 used in HEVC to 65. To keep the complexity of the most probable mode (MPM) list generation low, an intra mode encoding and decoding method with 6 MPMs is used by considering two available neighboring intra modes. The following three aspects are considered when constructing the MPM list:

[0067] – Default intra mode

[0068] – Neighboring intra mode

[0069] – Derived intra mode

[0070] Multi-reference intra prediction

[0071] Multiple reference line (MRL) intra prediction uses more reference lines for intra prediction. Figure 1 In

[15] , an example of 4 reference lines is depicted, where the samples of segments A and F are not taken from the reconstructed neighboring samples, but are filled with the closest samples from segments B and E. HEVC intra prediction uses the nearest reference line (i.e., reference line 0). In MRL, 2 additional lines are used (reference line 1 and reference line 3).

[0072] The index of the selected reference line (mrl_idx) is signaled and used to generate the intra predictor. For reference line indices greater than 0, only additional reference line modes are included in the MPM list, and only the MPM index is signaled without the remaining modes. The reference line index is signaled before the intra prediction mode, and if a non-zero reference line index is signaled, planar mode is excluded from the intra prediction mode.

[0073] MRL is disabled for the first row of a block within a CTU to prevent the use of extended reference samples outside the current CTU row. In addition, PDPC is disabled when additional lines are used. For MRL mode, the derivation of the DC value in DC intra prediction mode for non-zero reference line index is consistent with the derivation for reference line index 0. MRL requires the storage of 3 neighboring luma reference lines with the CTU to generate the prediction. The Cross Component Linear Model (CCLM) tool also requires 3 neighboring luma reference lines for its downsampling filter. The definition of MRL uses the same 3 lines as CCLM to reduce the memory requirements of the decoder.

[0074] Cross-component linear model prediction

[0075] In order to reduce cross-component redundancy, a cross-component linear model (CCLM) prediction mode (or LM) is used in VVC, in which the chroma samples are predicted by a linear model based on the reconstructed luminance samples of the same or corresponding CU, as shown below:

[0076] pred C (i,j)=α·rec L ′(i,j)+ β (1)

[0077] where pred C (i, j) represents the predicted chroma sample in CU, rec L ′(i, j) represents the downsampled reconstructed luma sample of the same or corresponding CU.

[0078] As an example, CCLM parameters (α and β) are derived from up to four neighboring chroma samples and their corresponding downsampled luma samples. Assuming the size of the current chroma block is W×H, W'' and H' are set to:

[0079] – When LM_LA mode is applied, W'=W, H'=H;

[0080] – When LM_A mode is applied, W'=W+H;

[0081] – When LM_L mode is applied, H'=H+W;

[0082] The adjacent positions above are represented as S[0,-1]…S[W'-1,-1], and the adjacent positions on the left are represented as S[-1,0]…S[-1,H'-1]. Then four samples are selected as follows:

[0083] – When LM_LA mode is applied and both the above and left neighboring samples are available, S[W' / 4,-1], S[3*W' / 4,-1], S[-1,H' / 4], S[-1,3*H' / 4];

[0084] – When LM_A mode is applied or only the above adjacent samples are available, S[W' / 8,-1], S[3*W' / 8,-1], S[5*W' / 8,-1], S[7*W' / 8,-1];

[0085] – When LM_L mode is applied or only left neighbor samples are available, S[-1,H' / 8], S[-1,3*H' / 8], S[-1,5*H' / 8], S[-1,7*H' / 8];

[0086] The four neighboring brightness samples at the selected position are downsampled and compared four times to find the two larger values: x 0 A and x 1 A , and two smaller values: x 0 B and x 1 B Their corresponding chrominance sample values ​​are represented as y 0 A 、y 1 A 、y 0 B and y 1 B Then X a 、X b 、Y a and Y b is derived as:

[0087] X a =(x 0 A +x 1 A +1)>>1;Xb =(x 0 B +x 1 B +1)>>1;Y a =(y 0 A +y 1 A +1)>>1;Y b =(y 0 B +y 1 B +1)>>1(2)

[0088] Finally, the linear model parameters α and β are obtained according to the following equations.

[0089]

[0090] β=Y b -α·X b (4)

[0091] Figure 2 Examples of left and upper samples involved in the LM_LA mode and sample positions of the current block are shown. The cross-component linear model (CCLM) in the present invention may use the examples and / or the CCLM in the present invention is not limited to the examples.

[0092] The division operation to calculate the parameter α is implemented using a lookup table. To reduce the memory required to store the table, the difference (the difference between the maximum and minimum values) and the parameter α are expressed as exponents. For example, the difference is approximated using a 4-bit significand and an exponent. Therefore, the table for 1 / diff is reduced to 16 elements, corresponding to the 16 values ​​of the significand, as shown below:

[0093] DivTable[]={0,7,6,5,5,4,4,3,3,2,2,1,1,1,1,0} (5)

[0094] This will help reduce the computational complexity as well as the memory size of the required tables. This memory reduction method can be used for CCLM and / or is not limited to CCLM, but can be used for any cross-component pattern to reduce computational complexity.

[0095] In addition to the templates above and on the left being used together to calculate linear model coefficients, they can also be used alternately in two other LM modes, called LM_A and LM_L modes.

[0096] In LM_A mode, only the upper template is used to calculate the linear model coefficients. To obtain more samples, the upper template is expanded to (W+H) samples. In LM_L mode, only the left template is used to calculate the linear model coefficients. To obtain more samples, the left template is expanded to (H+W) samples.

[0097] In LM_LA mode, the left and upper templates are used to calculate the linear model coefficients.

[0098] To match the chroma sample positions of a 4:2:0 video sequence, in some examples, two types of downsampling filters are applied to the luma samples to achieve a 2 to 1 downsampling ratio in both the horizontal and vertical directions. The choice of downsampling filter is specified by a sequence parameter set (SPS) level flag. The two downsampling filters are as follows, corresponding to "type-0" and "type-2" content, respectively.

[0099]

[0100] Note that when the upper reference line is at a coding tree unit (CTU) boundary, only one luma line (generic line buffer in intra prediction) is used to generate downsampled luma samples. The downsampling filter may be from the examples described and / or is not limited to the examples described.

[0101] This parameter calculation is performed as part of the decoding process, not just the encoder search operation. Therefore, no syntax is used to pass the α and β values ​​to the decoder.

[0102] For chroma intra mode encoding and decoding, in one example, a total of 8 intra modes are allowed for chroma intra mode encoding and decoding. These modes include five traditional intra modes and three cross-component linear model modes (LM_LA, LM_A, and LM_L). The chroma mode signaling and derivation process are shown in Table 1-1. Chroma mode encoding and decoding directly depends on the intra prediction mode of the corresponding luminance block. Since the independent block partition structure of luminance and chrominance components is enabled in I slices, one chroma block may correspond to multiple luminance blocks. Therefore, for the chroma DM mode, the intra prediction mode of the corresponding luminance block covering the center position of the current chroma block is directly inherited.

[0103] Table 1-1 – Chroma prediction modes derived from luma modes when ccm_is is enabled

[0104]

[0105]

[0106] Regardless of the value of sps_cclm_enabled_flag, a single binarization table is used, as shown in Table 1-2.

[0107] Table 1-2 – Unified binarization table for chroma prediction mode

[0108] The value of intra_chroma_pred_mode Binary string 4 00 0 0100 1 0101 2 0110 3 0111 5 10 6 110 7 111

[0109] In Table 1-2, the first binary bit indicates whether it is normal mode (0) or CCLM mode (1). If it is CCLM mode, the next binary bit indicates whether it is LM_LA (0). If it is not LM_LA, the next 1 binary bit indicates whether it is LM_L (0) or LM_A (1). In this case, when sps_cclm_enabled_flag is 0, the first binary bit of the corresponding intra_chroma_pred_mode binarization table can be discarded before entropy coding. In other words, the first binary bit is inferred to be 0 and therefore not encoded. This single binarization table is used for the case where sps_cclm_enabled_flag is equal to 0 and 1. The first two binary bits in Table 1-2 are context coded using their own context model, and the remaining binary bits are bypass coded. Chroma intra mode coding and decoding can use this example and / or chroma intra mode coding and decoding are not limited to this example.

[0110] Additionally, in one example, to reduce luma-chroma latency in a dual tree, when a 64x64 luma codec tree node uses no partitioning (and ISP is not used for 64x64 CUs) or QT partitioning, the chroma CUs in a 32x32 / 32x16 chroma codec tree node are allowed to use CCLM, as follows:

[0111] – If a 32x32 chroma node is not partitioned or partitioned as a QT partition, all chroma CUs in the 32x32 node can use CCLM

[0112] – If the 32x32 chroma node partition is horizontal BT, and the 32x16 child node is not partitioned or uses vertical BT partitioning, all chroma CUs in the 32x16 chroma node can use CCLM.

[0113] Under all other luma and chroma codec tree partitioning conditions, chroma CUs are not allowed to use CCLM.

[0114] Multi-model CCLM

[0115] In JEM, a multi-model CCLM (MMLM) scheme is proposed for predicting the chroma samples of the entire chroma CU / CB from the luma samples using two models. In MMLM, the neighboring luma samples and chroma samples of the current block are classified into two groups, each of which is used as a training set to derive a linear model (i.e., a specific α and β are derived for a specific group). In addition, the samples of the current luma block are also classified according to the classification rules of the neighboring luma samples.

[0116] Figure 3 An example of classifying neighboring samples into two groups is shown. The threshold is calculated as the average value of neighboring reconstructed brightness samples or a predefined value. Neighboring samples Rec′ L [x,y]<=threshold is classified as group 1; and the neighboring sample Rec′ L [x,y]>threshold is classified as group 2.

[0117]

[0118] Slope adjustment of CCLM

[0119] CCLM uses a two-parameter model to map luma values ​​to chroma values. The slope parameter "a" and the offset parameter "b" define the mapping as follows:

[0120] chromaVal=a*lumaVal+b

[0121] The adjustment of the slope parameter "u" is signaled to update the model in the following form:

[0122] chromaVal=a'*lumaVal+b'

[0123] in

[0124] a'=a+u

[0125] b'=bu*y r .

[0126] With this choice, the mapping function is centered around the brightness value y r The average value of the reference brightness samples is used as the y r , to provide meaningful modifications to the model. Figure 4 The process is illustrated, where sub-figure (A) shows the model created using the current cross-component linear model (CCLM) and sub-figure (B) shows the proposed updated model.

[0127] accomplish

[0128] The slope adjustment parameter is provided as an integer between -4 and 4 and is transmitted in the bitstream. The unit of the slope adjustment parameter is 1 / 8 of the value of each luma sample. thChroma sample values ​​(for 10-bit content).

[0129] In one example, the adjustment is applied to CCLM models that use reference samples above and to the left of the block ("LM_CHROMA_IDX" and "MMLM_CHROMA_IDX"), but not to "one-sided" mode. This choice is based on the consideration of the codec efficiency and complexity trade-off.

[0130] When slope adjustment is applied to a multi-mode CCLM model (or MMLM), both models can be adjusted, so at most two slope updates are transmitted for a single chroma block.

[0131] Encoder Method

[0132] The proposed encoder method performs a SATD-based search to find the optimal value for the slope update of Cr, and a similar SATD-based search to find the optimal value for Cb. If either result is a non-zero slope adjustment parameter, the combined slope adjustment pair (SATD-based update of Cr, SATD-based update of Cb) is included in the RD check list of the TU.

[0133] Convolutional Cross-Component Model (CCCM)

[0134] In CCCM, a convolution model is applied to improve chroma prediction performance. In one example, the convolution model has a 7-tap filter consisting of a 5-tap plus-shaped spatial component, a nonlinear term, and a bias term. The input to the spatial 5-tap component of the filter includes the center (C) luma sample corresponding to the chroma sample to be predicted and its neighbors above / north (N), below / south (S), left / west (W), and right / east (E), such as Figure 5 shown.

[0135] The nonlinear term (denoted as P) is expressed as the square of the center luminance sample C, scaled to the sample value range of the content:

[0136] P=(C*C+midVal)>>bitDepth

[0137] That is, for 10 bits of content, the calculation is:

[0138] P=(C*C+512)>>10

[0139] The bias term (denoted as B) represents a scalar offset between the input and output (similar to the offset term in CCLM) and is set to the middle chroma value (512 for 10-bit content).

[0140] The output of the filter is calculated as the filter coefficient c i Convolve with the input value and clip to the range of valid chroma samples:

[0141] predChromaVal=c0C+c1N+c2S+c3E+c4W+c5P+c6B

[0142] Filter coefficient c i It is computed by applying any predefined regression method or by minimizing the mean squared error (MSE) between the predicted and reconstructed chroma samples in a reference region. Figure 6 The reference region is shown, consisting of six rows of chroma samples above and to the left of the PU. The reference region extends one PU width to the right and one PU height downward to the PU boundary. The region is adjusted to only include available samples. The region extension shown as a slash in the reference region requires "side samples" that support the plus-shaped spatial filter and fills in the unavailable area.

[0143] MSE minimization or a predefined regression method is performed by calculating the autocorrelation matrix of the luma input and the cross-correlation vector between the luma input and the chroma output. In one example, the autocorrelation matrix is ​​subjected to LDL decomposition and the final filter coefficients are calculated using back substitution. This process roughly follows the calculation of the ALF filter coefficients in the Enhanced Compression Model (ECM), however, LDL decomposition is selected instead of Cholesky decomposition to avoid the use of square root operations. The autocorrelation matrix can be the method mentioned in this example and / or the regression method is not limited to the method mentioned in this example.

[0144] Gradient Linear Model (GLM)

[0145] Compared to CCLM, GLM utilizes the gradient of luma samples to infer the linear model instead of the downsampled luma values. Specifically, in one example, when applying GLM, the input of the CCLM process, i.e., the downsampled luma samples L, is replaced by the luma sample gradient G. The other parts of CCLM (e.g., parameter inference, linear transformation of prediction samples) remain unchanged.

[0146] C=α·G+β

[0147] For signaling, when CCLM mode is enabled for the current CU, two flags are transmitted for the Cb and Cr components respectively to indicate whether GLM is enabled for each component; if GLM is enabled for a component, a syntax element is further transmitted to select one of the 16 gradient filters for gradient calculation, such as Figure 7 As shown in Figure 2. GLM can be combined with the existing CCLM by transmitting an additional flag in the bitstream. When this combination is applied, the filter coefficients used to derive the linear model input luma samples are calculated as a combination of the gradient filter selected by GLM and the downsampling filter of CCLM.

[0148] Intra-block replication

[0149] Intra block copy (IBC) is a tool used in the HEVC extension for screen content coding (SCC). It is well known that it significantly improves the encoding and decoding efficiency of screen content material. Since the IBC mode is implemented as a block-level codec mode, block matching (BM) is performed in the encoder to find the optimal block vector (or motion vector) for each coding unit (CU). Here, the block vector is used to indicate the displacement from the current block to the reference block, which has been reconstructed within the current picture. The luma block vector of the IBC-encoded CU is integer precision. The chroma block vector is also rounded to integer precision. When combined with adaptive motion vector resolution (AMVR), the IBC mode can switch between 1-pixel and 4-pixel motion vector precision. The IBC-encoded CU is treated as a third prediction mode in addition to the intra or inter prediction mode. The IBC mode is applicable to CUs with a width and height that are less than or equal to 64 luma samples.

[0150] On the encoder side, IBC performs hash-based motion estimation. The encoder performs rate-distortion (RD) checks on blocks with a width or height no greater than 16 luma samples. For non-merge mode, a block vector search is first performed using a hash-based search. If the hash search does not return a valid candidate, a local search based on block matching is performed.

[0151] In hash-based search, hash key matching (32-bit CRC) between the current block and the reference blocks is extended to all allowed block sizes. The hash key calculation for each position in the current picture is based on a 4x4 sub-block. For larger current block sizes, a hash key is determined to match the hash key of the reference block when the hash keys of all 4x4 sub-blocks match the hash keys of the corresponding reference positions. If multiple reference blocks are found whose hash keys match the hash key of the current block, the block vector cost of each matching reference is calculated and the one with the lowest cost is selected.

[0152] In the block matching search, the search range is set to cover the previous and current coding tree units (CTUs).

[0153] At the CU level, the IBC mode is signaled via a flag and can be signaled as either IBC Adaptive Motion Vector Prediction (AMVP) mode or IBC Skip / Merge mode as shown below:

[0154] - IBC skip / merge mode: Use the merge candidate index to indicate which block vector to use from the list of neighboring candidate IBC coded blocks to predict the current block. The merge list consists of spatial, historical motion vector prediction (HMVP) and / or pairwise candidates.

[0155] -IBC AMVP mode: Block vector differences are coded in the same way as motion vector differences. The block vector prediction method uses two candidates as predictors, one from the left neighbor and one from the upper neighbor (if IBC coding). When either neighbor is unavailable, the default block vector is used as the predictor. A flag is signaled to indicate the block vector predictor index.

[0156] IBC Reference Area

[0157] To reduce memory consumption and decoder complexity, IBC in VVC only allows the reconstructed portion of a predefined area to contain the area of ​​the current CTU and some areas of the left CTU. Figure 8 Illustrate the reference area of ​​the IBC mode, where each block represents a 64x64 unit of luma samples.

[0158] Depending on the position of the current coding CU in the current CTU, the following rules apply:

[0159] If the current block is located in the upper-left 64x64 block of the current CTU, in addition to the reconstructed samples in the current CTU, the current picture reference (CPR) mode may be used to reference the reference samples in the lower-right 64x64 block of the left CTU. (More details on CPR can be found in JVET-T2002.) The current block may also use the CPR mode to reference the reference samples in the lower-left 64x64 block of the left CTU and the reference samples in the upper-right 64x64 block of the left CTU.

[0160] -If the current block is located in the upper right 64x64 block of the current CTU, in addition to the reconstructed samples in the current CTU, if the luma position (0, 64) relative to the current CTU has not been reconstructed, the current block can also use the CPR mode to refer to the reference samples in the lower left 64x64 block and the lower right 64x64 block of the left CTU; otherwise, the current block can also refer to the reference samples in the lower right 64x64 block of the left CTU.

[0161] -If the current block is located in the lower left 64x64 block of the current CTU, in addition to the reconstructed samples in the current CTU, if the luma position (64,0) relative to the current CTU has not been reconstructed, the current block can also use the CPR mode to refer to the reference samples in the upper right 64x64 block and the lower right 64x64 block of the left CTU. Otherwise, the current block can also use the CPR mode to refer to the reference samples in the lower right 64x64 block of the left CTU.

[0162] - If the current block is located in the lower right 64x64 block of the current CTU, only the reconstructed samples in the current CTU can be referenced using the CPR mode.

[0163] This restriction allows hardware implementation of IBC mode using local on-chip memory.

[0164] Chroma DM mode

[0165] For the chroma DM mode, the intra prediction mode of the corresponding (corresponding) luma block covering the center position of the current chroma block is directly inherited.

[0166] Decoder-side Intra Mode Derivation (DIMD)

[0167] When DIMD is applied, two or more intra modes are derived from the reconstructed neighbor samples and these two or more predictors are combined with the planar mode predictor with weights derived from the gradients. DIMD mode is used as an alternative prediction mode and is always checked in the high-complexity rate-distortion optimization (RDO) mode.

[0168] To implicitly derive the intra prediction mode for a block, texture gradient analysis is performed at the encoder, decoder, or both. In one example, this process starts with an empty Histogram of Gradients (HoG) with 65 entries, corresponding to 65 angular modes. In another example, this process can start with an empty HoG with a predefined number of entries, corresponding to a predefined number of angular modes, where the predefined number can be any positive integer and / or 131 and / or any number greater than 65. The magnitudes of these entries are determined during the texture gradient analysis.

[0169] In the first step, DIMD selects T=3 columns and rows of templates from the left and above the current block, respectively. This region is used as a reference for gradient-based intra prediction mode derivation.

[0170] In the second step, horizontal and vertical Sobel filters are applied to all 3×3 window positions, centered on the pixels in the middle line of the template. At each window position, the Sobel filter calculates the pure horizontal and vertical intensity G x and G y The texture angle of the window is then calculated as:

[0171] angle=arctan(G x / G y ),

[0172] In some examples, it can be converted to one of 65 angular intra prediction modes. Once the intra prediction mode index of the current window is derived as idx, the magnitude of its entry in HoG[idx] is updated as follows:

[0173] ampl=|G x |+|G y |

[0174] Figure 9An example of the HoG computed after applying the above operations to all pixel locations in the template is shown.

[0175] Once the HoG is calculated, the indices with the two or more highest histogram entries are selected as two or more implicitly derived intra prediction modes for the block and further combined with the planar mode as the prediction for DIMD mode. Prediction fusion is applied as a weighted average of the three or more predictors mentioned above. To this end, the weight of the plane is fixed to 21 / 64 (approximately 1 / 3). The remaining weight 43 / 64 (approximately 2 / 3) is then shared between the two or more HoGIPMs, proportional to the magnitude of their HoG entries. Figure 10 This process is visualized, in this example only 2 IPMs are selected using the HoG bars.

[0176] In addition, two or more implicitly derived intra modes are included in the MPM list, so the DIMD process is performed before building the MPM list. The primary derived intra mode of a DIMD block is stored with the block and used for MPM list construction of neighboring blocks.

[0177] Template-based intra mode derivation (TIMD)

[0178] Template-based intra mode derivation (TIMD) mode implicitly derives the intra prediction mode of a CU from neighboring templates at the encoder, decoder, or both, instead of signaling the exact intra prediction mode bits to the decoder. Figure 11 As shown, the prediction samples of the template are generated for each candidate mode using the reference samples of the template. The SATD between the prediction and the reconstructed samples of the template is calculated as the cost. The intra prediction mode with the minimum cost is selected as the TIMD mode and used for intra prediction of the CU. The candidate mode can be 67 intra prediction modes in VVC or extended to 131 intra prediction modes. Generally, MPM can provide clues to indicate the directional information of the CU. Therefore, in order to reduce the intra mode search space and utilize the characteristics of the CU, the intra prediction mode is implicitly derived from the MPM list.

[0179] For each intra prediction mode in the MPM, the SATD between the prediction and the reconstructed samples of the template is calculated. The first two intra prediction modes with the smallest SATD are selected as TIMD modes. These two TIMD modes are fused with weights after applying the PDPC process, and the current CU is encoded using this weighted intra prediction. Position-dependent intra prediction combining (PDPC) is included in the derivation of TIMD modes.

[0180] Comparing the cost of the two selected patterns with the threshold, a cost factor of 2 is applied in the test as follows:

[0181] costMode2<2*costMode1.

[0182] If this condition is true, fusion is applied, otherwise only mode1 is used. The weight of a mode is calculated from its SATD cost as follows:

[0183] weight1=costMode2 / (costMode1+costMode2)

[0184] weight2=1-weight1

[0185] Overview of inter-frame prediction (More details can be found in JVET-T2002 Section 3.4.)

[0186] For each inter-predicted CU, the motion parameters include motion vector, reference picture index and / or reference picture list usage index, as well as additional information required for the new codec features of VVC for inter-prediction sample generation. The motion parameters can be signaled explicitly or implicitly. When a CU is coded in skip mode, the CU is associated with one PU and has no significant residual coefficients, no coded motion vector increments or reference picture indices. A merge mode is specified, in which the motion parameters of the current CU are obtained from neighboring CUs, including spatial and temporal candidates, as well as additional schemes introduced in VVC. Merge mode can be applied to any inter-predicted CU, not just skip mode. An alternative to merge mode is explicit transmission of motion parameters, where the motion vector, the corresponding reference picture index for each reference picture list and / or reference picture list usage flags and / or other required information are explicitly signaled on each CU.

[0187] In addition to the inter-frame coding and decoding features in HEVC, VVC also includes some new and improved inter-frame prediction coding and decoding tools, which are listed below:

[0188] – Extended Merger Forecast

[0189] –Merge mode with MVD (MMVD)

[0190] – Symmetrical MVD (SMVD) signal transmission

[0191] – Affine motion compensated prediction

[0192] – Sub-block based temporal motion vector prediction (SbTMVP)

[0193] – Adaptive Motion Vector Resolution (AMVR)

[0194] – Sports Field Storage: 1 / 16 th Luminance sample MV storage and 8x8 motion field compression

[0195] – Bidirectional prediction with CU-level weights (BCW)

[0196] – Bidirectional Optical Flow (BDOF)

[0197] –Decoder-side motion vector optimization (DMVR)

[0198] – Geometric Partitioning Mode (GPM)

[0199] – Combined inter-frame and intra-frame prediction (CIIP)

[0200] The following text provides details of those inter prediction methods specified in VVC.

[0201] Extended Consolidated Forecast

[0202] In VVC, the merge candidate list is constructed by including the following five types of candidates in order:

[0203] 1) Space MVP from the space neighbor CU

[0204] 2) Time MVP from the corresponding CU

[0205] 3) History-based MVP from FIFO table

[0206] 4) Paired Average MVP

[0207] 5) Zero MV.

[0208] The size of the merge list is signaled in the sequence parameter set header, and the maximum allowed size of the merge list is 6. For each CU coded in merge mode, the index of the best merge candidate is encoded using truncated unary binarization (TU). The first bin of the merge index is coded using context, and the other bins are coded using bypass.

[0209] The derivation process of each type of merge candidate is provided in this section. Similar to the practice in HEVC, VVC also supports parallel derivation of the merge candidate list (or merge candidate list) of all CUs in a certain size area.

[0210] Spatial candidate derivation

[0211] The derivation of spatial merge candidates in VVC is the same as in HEVC, except that the positions of the first two merge candidates are swapped. Figure 12 Select up to four merge candidates from the positions shown. The derivation order is B 0, A 0, B 1,A1 and B2. Position B2 is considered only when one or more CUs at positions B0, A0, B1, A1 are not available (for example, because they belong to another slice or tile) or are intra-coded. After the candidate at position A1 is added, the remaining candidates are added with a redundancy check to ensure that candidates with the same motion information are excluded from the list, thereby improving encoding and decoding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the above redundancy check. Instead, only the Figure 13 The pairs are connected by arrows in , and a candidate is added to the list only if the corresponding candidate used for redundancy check does not have the same motion information.

[0212] In addition to the above spatial candidates, non-adjacent spatial merging candidates in JVET-L0399 are inserted after TMVP in the regular merging candidate list. The pattern of spatial merging candidates is as follows Figure 14 The distance between non-adjacent spatial candidates and the current codec block is based on the width and / or height of the current codec block. Line buffer restrictions do not apply.

[0213] Time candidate derivation

[0214] In this step, only one candidate is added to the list. In particular, in the derivation of this temporal merge candidate, a scaled motion vector is derived based on the corresponding CU belonging to the corresponding reference picture. The reference picture list and reference index used to derive the corresponding CU are explicitly signaled in the slice header. The scaled motion vector of the temporal merge candidate is as follows Figure 15 The dotted line in is obtained by scaling the motion vector of the corresponding CU using the POC distances tb and td, where tb is defined as the POC difference between the reference image of the current picture and the current picture, and td is defined as the POC difference between the reference image of the corresponding picture and the corresponding picture. The reference image index of the temporal merging candidate is set to zero.

[0215] The position of the time candidate is Figure 16 The CU at position C0 is selected between candidates C0 and C1 as shown in . If the CU at position C0 is not available, is intra-coded, or is outside the current CTU row, position C1 is used. Otherwise, position C0 is used in the derivation of the temporal merge candidate.

[0216] History-based merge candidate derivation

[0217] History-based MVP (HMVP) merge candidates are added to the merge list after spatial MVP and TMVP. In this method, the motion information of the previously coded blocks is stored in a table and used as the MVP of the current CU. A table containing multiple HMVP candidates is maintained during the encoding / decoding process. When a new CTU row is encountered, the table is reset (cleared). Whenever there is a non-subblock inter-frame coded CU, the relevant motion information is added to the last entry of the table as a new HMVP candidate.

[0218] The size S of the HMVP table is set to 6, which means that up to 5 history-based MVP (HMVP) candidates can be added to the table. When inserting a new motion candidate into the table, a restricted first-in-first-out (FIFO) rule is used, where a redundancy check is first applied to find whether the same HMVP exists in the table. If found, the same HMVP is removed from the table, all subsequent HMVP candidates are moved forward, and the same HMVP is inserted as the last entry in the table.

[0219] HMVP candidates can be used in the merge candidate list construction process. The latest HMVP candidates in the table are checked in order and inserted into the candidate list after the TMVP candidate. HMVP candidates are checked for redundancy to compare with spatial or temporal merge candidates.

[0220] To reduce the number of redundant checking operations, the following simplifications are introduced:

[0221] 1. The last two entries in the table are checked for redundancy with the A1 and B1 space candidates respectively.

[0222] 2. Once the total number of available merge candidates reaches the maximum allowed number of merge candidates minus 1, the merge candidate list construction process of HMVP is terminated.

[0223] Pairwise average merge candidate derivation

[0224] Pairwise average candidates are generated by averaging predefined candidate pairs in the existing merge candidate lists, using the first two merge candidates. The first merge candidate is defined as p0Cand, and the second merge candidate can be defined as p1Cand respectively. The average motion vector is calculated for each reference list separately based on the availability of motion vectors for p0Cand and p1Cand. If two motion vectors are available in one list, even if they point to different reference pictures, the two motion vectors are averaged and their reference pictures are set to the reference picture of p0Cand; if only one motion vector is available, the motion vector is used directly; if no motion vector is available, the list is kept invalid. In addition, if the half-pixel interpolation filter index of p0Cand and p1Cand is different, it is set to 0.

[0225] When the merge list is not full after adding pairwise average merge candidates, zero MVPs are inserted at the end until the maximum number of merge candidates is reached.

[0226] Merge estimated areas

[0227] The merge estimation region (MER) allows CUs in the same merge estimation region (MER) to independently derive merge candidate lists. Candidate blocks in the same MER as the current CU are not included in the generation of the merge candidate list of the current CU. In addition, the update process of the history-based motion vector prediction candidate list is only updated when (xCb+cbWidth)>>Log2ParMrgLevel is greater than xCb>>Log2ParMrgLevel and (yCb+cbHeight)>>Log2ParMrgLevel is greater than (yCb>>Log2ParMrgLevel), where (xCb, yCb) is the top left corner luma sample position of the current CU in the picture, and (cbWidth, cbHeight) is the size of the CU. The size of the MER is selected at the encoder side and signaled in the sequence parameter set as log2_parallel_merge_level_minus2.

[0228] Bi-prediction with CU-level weights (BCW)

[0229] In High Efficiency Video Coding (HEVC), the bi-prediction signal is generated by averaging two prediction signals obtained from two different reference pictures. In VVC, the bi-prediction mode is extended beyond simple averaging to allow for a weighted average of the two prediction signals.

[0230] P bi-pred =((8-w)*P0+w*P1+4)>>3

[0231] Five weights, w∈{-2,3,4,5,10}, are allowed in weighted average bi-prediction. For each bi-predicted CU, the weight w is determined in one of two ways: 1) for non-merged CUs, the weight index is signaled after the motion vector difference; 2) for merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applied to CUs with 256 or more luma samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency pictures, all five weights are used. For non-low-latency pictures, only three weights (w∈{3,4,5}) are used.

[0232] – In the encoder, fast search algorithms are used to find weight indices without significantly increasing encoder complexity. These algorithms are summarized below. For more information, see the VTM software and document JVET-L0646. When used in conjunction with AMVR, conditional checks for unequal weights are used for 1-pel and 4-pel motion vector accuracy only when the current picture is a low-latency picture.

[0233] – When used in conjunction with affine, affine ME will be performed for unequal weights only if the affine mode is selected as the current best mode.

[0234] – Unequal weights are only checked conditionally when the two reference pictures in bi-prediction are the same.

[0235] – Unequal weights are not searched when certain conditions are met, which depend on the POC distance between the current picture and its reference pictures, the codec QP, and the temporal layer.

[0236] The BCW weight index is encoded using a context-coded bin followed by bypass-coded bins. The first context-coded bin indicates whether equal weights are used; if unequal weights are used, additional bins are signaled using bypass codec to indicate which unequal weights are used.

[0237] Weighted Prediction (WP) is a codec tool supported by the H.264 / AVC and HEVC standards for efficiently encoding video content with gradients. Support for WP has also been added to the VVC standard. WP allows weighting parameters (weights and offsets) to be signaled for each reference picture in each reference picture list L0 and L1. The weights and offsets of the corresponding reference picture are then applied during motion compensation. WP and BCW are designed for different types of video content. To avoid interaction between WP and BCW, which would complicate VVC decoder design, if a CU uses WP, the BCW weight index is not signaled and w is inferred to be 4 (i.e., equal weights are applied). For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. This applies to both normal merge mode and inherited affine merge mode. For constructed affine merge mode, affine motion information is constructed based on the motion information of up to three blocks. The BCW index of a CU using constructed affine merge mode is simply set equal to the BCW index of the first control point MV.

[0238] In VVC, CIIP and BCW cannot be applied jointly to a CU. When a CU is encoded in CIIP mode, the BCW index of the current CU is set to 2, i.e. equal weight.

[0239] Combined Inter and Intra Prediction (CIIP)

[0240] In VVC, when a CU is encoded in merge mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and if both the CU width and the CU height are less than 128 luma samples, an additional flag is issued to indicate whether the combined inter / intra prediction (CIIP) mode is applied to the current CU. As its name suggests, CIIP prediction combines the inter prediction signal with the intra prediction signal. The inter prediction signal P in CIIP mode inter is obtained by the same inter-frame prediction process applied to the conventional merge mode; the intra-frame prediction signal P intra It is obtained by the conventional intra prediction process with planar mode or TIMD. The intra and inter prediction signals are then combined by weighted averaging, where the weights are calculated based on the codec mode of the top and left neighboring blocks (e.g. Figure 17 The calculation is as follows:

[0241] – If the top neighbor is available and is intra-coded, set isIntraTop to 1, otherwise set isIntraTop to 0;

[0242] – If the left neighbor is available and is intra-coded, set isIntraLeft to 1, otherwise set isIntraLeft to 0;

[0243] – If (isIntraLeft + isIntraTop) is equal to 2, set wt to 3;

[0244] – Otherwise, if (isIntraLeft + isIntraTop) is equal to 1, set wt to 2;

[0245] – Otherwise, set wt to 1.

[0246] The CIIP forecast is formed as follows:

[0247] P CIIP =((4-wt)*P inter +wt*P intra +2)>>2

[0248] Geometric Partitioning Model (GPM)

[0249] In VVC, the geometric partition mode supports inter-frame prediction. The geometric partition mode uses a CU-level flag as a merge mode for signaling. Other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. The geometric partition mode supports a total of 64 partitions for each possible CU size w×h=2 m ×2 n , where m,n∈{3…6} excludes 8x64 and 64x8.

[0250] When using this mode, the CU is divided into two parts by a geometrically positioned straight line ( Figure 18 ). The position of the partition lines is mathematically derived from the angle and offset parameters for the specific partition. Each part of the geometric partition in the CU uses its own motion for inter-frame prediction; only a single prediction is allowed for each partition, that is, one motion vector and one reference index per part. The single prediction motion constraint is used to ensure that only two motion compensated predictions are required per CU, as in traditional bi-prediction. The single prediction motion for each partition is derived using the process described in the following sections.

[0251] If the current CU uses geometric partition mode, a geometric partition index is further signaled to indicate the partition mode (partition line) of the geometric partition (angle and offset, or called angle and distance), as well as two merge indexes (one for each partition). The number of maximum GPM candidate sizes is explicitly signaled in the SPS, and the syntax binarization of the GPM merge index is specified. After predicting each part of the geometric partition, the sample values ​​along the edge of the geometric partition are adjusted using a hybrid process of adaptive weights, as described in subsequent sections. This is the prediction signal for the entire CU, and the transform and quantization process will be applied to the entire CU as with other prediction modes. Finally, the motion field of the CU predicted using the geometric partition mode is stored as described in subsequent sections.

[0252] Single prediction candidate list construction

[0253] The single prediction candidate list is directly derived from the merge candidate list constructed according to the extended merge prediction process. Let n be the index of the single prediction motion in the geometric single prediction candidate list. The LX motion vector of the nth extended merge candidate, where X is equal to the parity of n, is used as the nth single prediction motion vector of the geometric partition mode. These motion vectors are in Figure 19 If the corresponding LX motion vector of the n-th extended merge candidate does not exist, the L(1-X) motion vector of the same candidate is used as the single prediction motion vector of the geometric partition mode.

[0254] Blending along the edges of geometric partitions

[0255] After predicting each part of the geometric partition using its own motion, blending is applied to the two prediction signals to derive samples around the geometric partition edge. The blending weight for each position of the CU is derived based on the distance between each position and the partition edge.

[0256] Specifically, as in JVET-Z0059, weighted prediction, i.e., blending, is applied to the two prediction signals for each part of the GPM derived from its own motion vector or its own intra prediction mode to generate the final prediction samples around the GPM partition boundaries.

[0257] Two integer mixing matrices (W0 and W1) are used for the GPM mixing process. In one example, the weights in the GPM mixing matrix contain a value range of [0,8] and are derived based on the displacement from the sample location to the GPM partition boundary, such as Figure 20 shown.

[0258] Specifically, the weights are given by a discrete ramp function with a displacement and two thresholds, as Figure 21 shown.

[0259] The ramp function of the Geometric Partitioning Pattern (GPM) weight is derived based on the displacement of the sample position to the GPM partition boundary and a fixed threshold τ.

[0260] Here, the threshold τ defines the width of the GPM blend region and is chosen to be a fixed value in VVC. In other words, as described in JVET-Z0137, the blend strength or blend region width θ is fixed for all different contents, as Figure 22 shown.

[0261] The weighted values ​​in the blend mask can be given by a ramp function

[0262]

[0263] In the current ECM (VVC) design, with a fixed θ = 2pel, this ramp function can be quantized as

[0264] ω m，n =Clip3(0,8,(d(m,n)+32+4)>>3)

[0265] Therefore, as described in JVET-T2002, the distance from a location (x,y) to the edge of a partition is derived as:

[0266]

[0267]

[0268] where i,j are the angle and offset indices of the geometry partition, which depend on the signaled geometry partition index. x,j and ρ y,j The sign of depends on the angle index i.

[0269] The weights for each part of the geometric partition are derived as follows:

[0270]

[0271] w1(x,y)=1-w0(x,y)

[0272] partIdx depends on the angle index i. An example of weight w0 is Figure 23 Instructions.

[0273] Motion field storage in geometric partitioning mode

[0274] Mv1 from the first part of the geometric partition, Mv2 from the second part of the geometric partition, and the combination Mv of Mv1 and Mv2 are stored in the motion field of the geometric partition mode coded CU.

[0275] The type of motion vector stored for each individual position in the motion field is determined as:

[0276] sType=abs(motionIdx)<32?2:(motionIdx≤0?(1-partIdx):partIdx)

[0277] where motionIdx is equal to d(4x+2,4y+2), which is recalculated. partIdx depends on the angle index i.

[0278] If sType is equal to 0 or 1, then Mv0 or Mv1 is stored in the corresponding motion field, otherwise if sType is equal to 2, then the combined Mv of Mv0 and Mv2 is stored. The combined Mv is generated using the following process:

[0279] 1) If Mv1 and Mv2 come from different reference picture lists (one from L0 and the other from L1), Mv1 and Mv2 are simply combined to form a dual-prediction motion vector.

[0280] 2) Otherwise, if Mv1 and Mv2 are from the same list, only the single-predicted motion Mv2 is stored.

[0281] GPM Extensions (More details can be found in JVET-W0097 and JVET-Y0065.)

[0282] At the JVET-V conference, several variants were proposed to improve the encoding and decoding efficiency of the GPM mode in VVC. These methods were included in the Exploration Experiment (EE2) for further evaluation. The main technical aspects are described as follows:

[0283] EE2-3.3 on Geometric Partition Mode with Multi-Mode Motion Vector Difference (MMVD) (GPM-MMVD): 1) Additional motion vector differences (MVDs) are added to the existing GPM merge candidates; 2) The signaling method of these MVDs is the same as MMVD in VVC, that is, a distance index plus a direction index; 3) Two flags are used to control whether MMVD is applied to each GPM partition.

[0284] EE2-3.4-3.5 on Geometric Partitioning Mode with Template Matching (TM) (GPM-TM): 1) Template matching is extended to GPM mode by optimizing the GPM motion vector (MV) based on the left and above neighboring samples of the current coding unit (CU); 2) The selection of template samples depends on the partition direction of the GPM; 3) A single flag is used to jointly control whether template matching is applied to the MVs of two GPM partitions.

[0285] JVET-W0097 proposes a method to combine EE2-3.3, EE2-3.4, and EE2-3.5 to further improve the encoding and decoding efficiency of GPM mode. Specifically, in the proposed combination, the existing designs in EE2-3.3, EE2-3.4, and EE2-3.5 remain unchanged, while the following modifications are further applied to coordinate the two codec tools:

[0286] 1) GPM-MMVD and GPM-TM are exclusively enabled for a GPM CU. This is achieved by first signaling the GPM-MMVD syntax. When both GPM-MMVD control flags are false (i.e., GPM-MMVD is disabled for both GPM partitions), the GPM-TM flag is signaled to indicate whether template matching is applied to both GPM partitions. Otherwise (at least one GPM-MMVD flag is true), the GPM-TM flag is inferred to be false.

[0287] 2) The GPM merge candidate list generation methods in EE2-3.3 and EE2-3.4-3.5 are directly combined together, where the MV pruning scheme in EE2-3.4-3.5 (where the MV pruning threshold is adjusted according to the current CU size) is used to replace the default MV pruning scheme applied in EE2-3.3; in addition, as described in EE2-3.4-3.5, multiple zero MVs are added until the GPM candidate list is completely filled.

[0288] In JVET-Y0065, in GPM with inter and intra prediction (or called GPM intra), the final prediction samples are generated by weighting the inter prediction samples and intra prediction samples for each GPM partition area. The inter prediction samples are derived through the same scheme of GPM in the current ECM, while the intra prediction samples are derived through the intra prediction mode (IPM) candidate list and the index signaled from the encoder. The size of the IPM candidate list is predefined as 3. The available IPM candidates are angular modes parallel to the GPM block boundary (parallel mode), angular modes perpendicular to the GPM block boundary (vertical mode), and Figure 24 The plane pattern shown in sub-figures (A)-(C) of . In addition, as Figure 24The GPM with intra and intra prediction shown in sub-figure (D) is restricted in the proposed method to reduce the signaling overhead of IPM and avoid increasing the size of the intra prediction circuit on the hardware decoder. In addition, direct motion vector and IPM storage are introduced on the GPM mixed region to further improve the encoding and decoding performance.

[0289] Spatial Geometric Partitioning Model (SGPM) (more details can be found in JVET-Z0124)

[0290] Similar to the inter-frame GPM, the spatial geometric partitioning mode (SGPM) consists of a partitioning mode and two related intra-frame prediction modes, such as Figure 25 If these modes are signaled directly in the bitstream, as in Figure 26 As shown in the sub-figure (A) of , a significant overhead bit will be generated. In order to express the necessary partition and prediction information more efficiently in the bitstream, a candidate list is used, and only the candidate index is signaled in the bitstream. Each candidate in the list can derive a combination of a partition mode and two intra prediction modes, such as Figure 26 As shown in sub-figure (B).

[0291] A template is used to generate this candidate list. The template has the shape of Figure 27 For each possible combination of a partition mode and two intra prediction modes, a prediction for the template is generated and the partition weights are extended to the template, as Figure 27 As shown in Figure 2. These combinations are sorted in ascending order by the SATD between their predictions and the template reconstruction. The length of the candidate list is set to 16, and these candidates are considered to be the most likely SGPM combinations for the current block. Both the encoder and decoder build the same candidate list based on the template.

[0292] To reduce the complexity of building the candidate list, the number of possible partition modes and the number of possible intra prediction modes are pruned. In the following tests, 26 of the 64 partition modes are used, and only MPMs of the 67 intra prediction modes are used.

[0293] Overlapped Block Motion Compensation (OBMC) (more details can be found in JVET-W2025)

[0294] When OBMC is applied, the top and left boundary pixels of the CU are optimized in a weighted prediction manner using motion information of neighboring blocks, as described in JVET-L0101.

[0295] The conditions under which OBMC should not be applied are as follows:

[0296] – When OBMC is disabled at the Sequence Parameter Set (SPS) level

[0297] – When the current block has intra mode or IBC mode

[0298] – When the current block applies LIC

[0299] – When the current luminance block area is less than or equal to 32

[0300] Sub-block boundary OBMC is performed by applying the same blending to the top, left, bottom and right sub-block boundary pixels. It enables for sub-block based codecs:

[0301] – Affine AMVP mode;

[0302] – Affine merge mode and sub-block based temporal motion vector prediction (SbTMVP);

[0303] – Sub-block based bilateral matching.

[0304] Proposed method

[0305] In addition to the introduced cross-component modes, such as the cross-component linear model (CCLM) and / or the multi-model linear model (MMLM) and / or the cross-component compensation model (CCCM) and / or the slope adjustment of the cross-component linear model and / or the global linear model (GLM), several new cross-component modes are proposed in some embodiments. For example, an inverse linear model (LM) and / or a prediction-based linear model (LM). In the present invention, one or more cross-component modes are used to generate a cross-component prediction, and the cross-component prediction is used to form a prediction of a current block encoded in a non-intra mode. For example, the non-intra mode refers to any inter mode or any intra block copy (IBC) mode. In one embodiment, the prediction of the current block encoded in the non-intra mode is generated using one or more inter prediction modes, one or more cross-component modes, or both, if the non-intra mode refers to an inter mode. The prediction of the current block encoded in non-intra mode is generated using one or more intra block copy prediction modes, one or more cross-component modes, or both, if the non-intra mode refers to the intra block copy mode. The cross-component mode in the present invention may refer to any one or more of the mentioned cross-component modes and / or the cross-component mode in the present invention is not limited to the mentioned cross-component mode. In one embodiment, one or more cross-component modes are selected from a merge candidate list. In another embodiment, the cross-component prediction is generated using multiple cross-component modes. For example, the cross-component prediction is generated using a multi-hypothesis cross-component linear model (MH CCLM for short). In some embodiments, in addition to forming a prediction of the current block encoded in non-intra mode, in some cases, the cross-component mode can be used to improve intra prediction. For example, adaptive intra mode selection. Another example is a linear model assisted angle / plane mode.

[0306] Inverse linear model

[0307] For the cross-component linear model mode, the luma reconstructed samples are used to derive the predictor in the chroma block. In this disclosure, an inverse linear model is proposed to derive the predictor in the luma block using the chroma information. When the inverse linear model is supported, chroma is encoded / decoded (signaled / parsed) before luma.

[0308] In one embodiment, the chroma information refers to reconstructed chroma samples. When deriving model parameters for the inverse linear model, the reconstructed adjacent chroma samples are used as the X of the adjacent luma samples for model derivation in the traditional cross-component linear model, and the reconstructed adjacent luma samples are used as the Y of the adjacent chroma samples for model derivation in the traditional cross-component linear model. In addition, the reconstructed samples in the chroma block (corresponding to the current luma block) and the derived parameters are used to generate a predictor in the current luma block. Alternatively, in this embodiment, the "information" may refer to prediction samples.

[0309] In one embodiment, chrominance refers to cb and / or cr components.

[0310] In one sub-embodiment, only information from one of the Cb and Cr blocks is used. In another sub-embodiment, chrominance information is derived from both Cb and Cr blocks. For example, the reconstructed Cb and Cr samples of the adjacent blocks are weighted and then used as input to derive the model parameters. In another example, the reconstructed Cb and Cr samples in the chrominance block (corresponding to the current luma block) are weighted and then used to derive the predictor for the current luma block.

[0311] In another embodiment, for the current luma block, the prediction generated using the proposed inverse linear model can be combined with prediction hypotheses generated by one or more other intra prediction modes.

[0312] In a sub-embodiment, “other intra prediction modes” may refer to angular intra prediction mode, DC, planar, mode intra prediction (MIP), intra sub-block prediction (ISP), multi-reference line (MRL), any other existing intra mode (supported in high efficiency video coding (HEVC) / versatile video coding (VVC)) and / or any other intra prediction mode.

[0313] In another sub-embodiment, when combining multiple prediction hypotheses, the weighting of each hypothesis can be fixed or adaptively varied. For example, equal weighting can be applied to each hypothesis. In another example, the weighting can vary with neighboring codec information, sample position, block width, height, prediction mode, or region. The following shows some examples of using neighboring codec information:

[0314] –A possible rule related to sample location is described below.

[0315] When the sample position is farther from the reference sample, the weight of predictions from other intra prediction modes is reduced.

[0316] – Another possible rule related to neighboring codec information is described below.

[0317] As more neighboring blocks (left, above, above-left, above-right, and / or below-left) are coded in a particular mode (eg, mode A), the weight of the prediction from mode A increases.

[0318] – Another possible rule related to sample location is described below.

[0319] The current block is divided into several regions. Sample positions within the same region share the same weight. If the current region is close to the reference L neighbors, predictions from other intra prediction modes are weighted higher than predictions from the cross-component linear model. The following shows some possible ways to divide the current block.

[0320] – (width and height ratio is close to or exactly 1:1): consider the distance between the current region and the left and top reference L neighbors.

[0321] – (width > n*height, where n can be any positive integer): Consider the distance between the current region and one of the top and left reference L neighbors. For example, consider the distance between the current region and the left reference L neighbor.

[0322] – (height > n*width, where n can be any positive integer): Consider the distance between the current region and one of the top and left reference L neighbors. For example, consider the distance between the current region and the top reference L neighbor.

[0323] Prediction-based linear models

[0324] This section proposes a new linear model approach. Unlike cross-component linear models, the derived model parameters are applied to the prediction samples of the first component to obtain the prediction samples of the second or third component.

[0325] P(i,j)=a·pred′ L (i,j)+b

[0326] In one embodiment, the first component is luminance.

[0327] In one sub-embodiment, the prediction samples of the first component are downsampled by a downsampling filter. For example, the downsampling filter follows the original LM design. Another example is that the downsampling filter does not access neighboring prediction / reconstructed samples. At the boundary of the current block, if neighboring samples are needed as input samples to the downsampling filter, padding prediction values ​​from the current block boundary are used.

[0328] In another embodiment, the second component is Cb.

[0329] In another embodiment, the third component is Cr.

[0330] The following shows the flow of inter-frame cross-component linear model (CCLM) based prediction, which means using cross-component prediction to form inter-frame chroma prediction.

[0331] - Improve inter-frame chroma prediction by linearly predicting chroma samples from luma samples, e.g. Figure 28 shown

[0332] -The linear prediction method can be one of the following

[0333] -CCLM_LT,CCLM_L,CCLM_T

[0334] -MMLM_LT, MMLM_L, MMLM_T, and / or any cross-component mode.

[0335] -step:

[0336] -Step 1: Infer a linear model by reconstructing neighboring luminance and chrominance samples

[0337] -Step 2: Apply the derived linear model to the current luma prediction sample to obtain the current chroma prediction sample

[0338] -pred CCLM (i,j)=α·pred L ′(i,j)+β

[0339] -pred L ′(i,j): downsampled current brightness prediction sample

[0340] At boundaries within the current block, padding is used. For example, padding is used when the downsampling process refers to any sample of the luma block other than the current luma prediction sample.

[0341] Multi-hypothesis cross-component linear model prediction (MH CCLM)

[0342] In addition to CCLM (where model parameters are derived from luminance and cb for cb and from luminance and cr for cr), more CCLM variants have emerged. Some examples are shown below.

[0343] -One variant is for cr:

[0344] Use the neighboring reconstructed samples of cb and cr as input X to derive model parameters, as the neighboring luminance samples derived by the model in traditional CCLM, and Y as the neighboring chrominance samples derived by the model in traditional CCLM

[0345] The CR predictions are then generated using the derived model parameters and the CB reconstruction samples.

[0346] -Another variant is MMLM.

[0347] Another variant is to derive model parameters from multiple corresponding luminance blocks for cb (or cr).

[0348] Each cross-component mode, for example, each CCLM method, is suitable for different scenarios. For some complex features, combined predictions may lead to better performance. Therefore, a multi-hypothesis CCLM is proposed to mix predictions from multiple CCLM methods. The CCLM methods to be mixed can come from (but are not limited to) the CCLM methods mentioned above and / or (*), as shown below. (*) One or more LM modes (or cross-component modes) used to generate one or more hypothesis predictions of LM-assisted angle / plane mode and / or inter-frame CCLM and / or MH CCLM are selected from a predefined merge candidate list (called modelList). A modelIdx is signaled to select a candidate from the candidate list (modelList), and the selected candidate is used for the current block. modelList contains one or more candidates, each candidate referring to a model (or cross-component mode) information. If there is only one candidate in the list (the size of the list is only 1), modelIdx is not signaled and / or can be inferred to be 0 or a default value.

[0349] Blending is done using a weighting scheme.

[0350] In one embodiment, the weights of the predictions from different CCLM methods to be mixed are predefined in the encoder and / or decoder.

[0351] In another embodiment, the weights of the predictions from different CCLM methods to be mixed vary according to the distance between the sample (or region) location and the reference sample location.

[0352] In another embodiment, the weights of predictions from different CCLM methods to be mixed depend on neighboring codec information.

[0353] In another embodiment, the weights of predictions from different CCLM methods to be mixed are dependent on signaling. For example, a weight index is signaled / parsed. The codeword can be fixed or adaptively varied. For example, the codeword varies with a template-based approach. The template refers to the spatial neighborhood of the current block.

[0354] The above embodiments can be combined. For example, the weights of predictions from different CCLM methods to be mixed are predefined in the encoder and / or decoder. The predefined weights vary with the template-based method. The template refers to the spatial neighborhood of the current block.

[0355] Adaptive intra mode selection

[0356] As video codecs improve, more codecs are created. The syntax overhead of choosing a codec becomes an issue. There are several straightforward approaches to reducing syntax overhead.

[0357] - Example 1: A large block uses the same codec mode.

[0358] - Example 2: Multiple components (e.g. cb and cr) share the same codec mode.

[0359] However, using these direct methods, the accuracy / performance of intra prediction decreases. The possible reasons are as follows:

[0360] - Intra prediction is highly correlated with neighboring reference samples. When a single intra prediction mode is used for the entire block, the intra prediction mode may be appropriate for samples close to the reference sample, but may not be appropriate for samples far from the reference sample.

[0361] -When doing CR, the reconstruction of CB and luminance has been generated and can be used to select the CR codec mode.

[0362] In this section, we propose to adaptively change the intra prediction mode of one or more samples or sub-blocks within the current block based on the previous coded / decoded components.

[0363] In one embodiment, the performance of different codec modes is determined by reconstructing the previously encoded / decoded component. The better mode is then used for the remaining components (those subsequently encoded and decoded). For example, for CB, a conventional intra prediction mode (e.g., angular intra prediction mode, DC, planar) produces a better prediction than the LM mode. (For example, "better" means similar to the reconstruction of CB.) Then, the conventional intra prediction mode is preferred for CR.

[0364] In a sub-embodiment, the proposed method can be based on sub-blocks. For example, a chroma block is divided into several sub-blocks. For each sub-block, if for cb, the sub-block prediction from the LM mode is better than the sub-block prediction from the traditional intra prediction mode (e.g., angular intra prediction mode, DC, planar). (e.g., "better" means similar to the reconstruction of cb and reduces the residual of cb), then the LM mode is preferred for the corresponding sub-block of cr. (An example is Figure 29 shown)

[0365] In another embodiment, the adaptation rules may be performed at the encoder and / or decoder and no additional syntax is required.

[0366] CCLM for inter blocks (CCLM for inter blocks may also be referred to as inter CCLM, and “CCLM” may be extended to or replaced by any LM mode (or any cross-component mode, e.g., (*)))

[0367] CCLM is used for intra blocks to improve chroma intra prediction. For inter blocks, chroma prediction may not be as accurate as luma prediction. Possible reasons are as follows:

[0368] – The motion vectors for the chroma components are inherited from luma (chroma does not have its own motion vectors.)

[0369] – Fewer codec tools are designed to improve inter-frame chroma prediction.

[0370] Therefore, an alternative method for applying CCLM to inter-frame blocks is proposed. With this proposed method, the chrominance prediction of inter-frame blocks can be improved based on the luminance. (*) is shown below. (*) One or more LM modes (or cross-component modes) used to generate one or more prediction hypotheses for LM-auxiliary angular / planar modes and / or inter-frame CCLM and / or MH CCLM are selected from a predefined merge candidate list (called modelList). A modelIdx is signaled to select a candidate from the candidate list (modelList), and the selected candidate is used for the current block. modelList contains one or more candidates, where each candidate refers to a model (or cross-component mode) information. If there is only one candidate in the list (the size of the list is only 1), modelIdx is not signaled and / or can be inferred to be 0 or a default value.

[0371] In one embodiment, for the chroma component, in addition to the original inter-frame prediction (generated by motion compensation, which can be uni-prediction and / or bi-prediction, or can be only uni-prediction), one or more prediction hypotheses (generated by CCLM and / or any other LM mode and / or any cross-component mode and / or (*)) are used to form the current prediction.

[0372] In one sub-embodiment, the current prediction is a weighted sum of inter-frame prediction and cross-component prediction, such as CCLM prediction. The weights are designed based on neighboring codec information, sample position, block width, height, mode, or region.

[0373] – For example, for a small block (area < threshold), the weight of CCLM prediction is higher than that of inter prediction.

[0374] – Another example, when most of the neighboring coding blocks are intra blocks, the weight of CCLM prediction is higher than that of inter prediction.

[0375] – Another example where the weight is a fixed value for the entire block.

[0376] In another embodiment, inter-frame prediction can be generated by any inter-frame mode mentioned in the above introduction / documentation. For example, the inter-frame mode can be regular merge mode. In another example, the inter-frame mode can be CIIP mode. In another example, the inter-frame mode can be CIIP PDPC. In another example, the inter-frame mode can be GPM or any GPM variant (such as GPM Intra).

[0377] In one sub-embodiment, the normal merge mode is a merge candidate selected from a merge candidate list with a signaled merge index.

[0378] In another sub-embodiment, the normal merge mode may be MMVD.

[0379] In another sub-embodiment, the LM mode used in inter-CCLM is prediction-based LM.

[0380] In another embodiment, inter-CCLM is supported only when any one (or more) predefined inter-modes are used for the current block, or when any one (or more) enable flags of the predefined inter-modes are indicated as enabled. Supporting inter-CCLM means that the prediction of the current block can choose to apply inter-CCLM or not.

[0381] When inter CCLM is applied, the prediction of the current block is generated by

[0382] In one sub-embodiment: blending one or more prediction hypotheses (generated by CCLM and / or any other LM mode and / or any cross-component mode) with the original inter prediction

[0383] – Mixing chroma prediction from existing inter-mode and prediction from LM

[0384] – Mixing: Predfinal=(wInter*PredInter+wLM*PredLM+2)>>2

[0385] – Weighting rules: wInter and wLM, e.g.

[0386] – If both top and left are intra, (wInter,wLM)=(1,3)

[0387] – Otherwise, if one of the top and left is intra, (wInter,wLM)=(2,2)

[0388] – Otherwise, (wInter, wLM) = (3, 1)

[0389] – Another example, weighting follows CIIP weighting.

[0390] – For example, PredInter = Inter prediction after OBMC (if OBMC is used)

[0391] – Another example, PredInter = inter prediction before OBMC (OBMC can be applied after mixing)

[0392] In another sub-embodiment: replacing the original inter prediction with one or more prediction hypotheses (generated by a Cross Component Linear Model (CCLM) and / or any other Linear Model (LM) mode and / or any cross-component mode).

[0393] When inter CCLM is not applied, the prediction of the current block comes from the original inter prediction.

[0394] In another embodiment, the selection of whether to apply inter-frame CCLM depends on signaling.

[0395] In one sub-embodiment, a flag is signaled in the bitstream to indicate whether inter-CCLM is applied. For example, the flag is context-coded. Another example is that only one context is used to encode and / or decode the flag. Yet another example is that multiple contexts are used to encode and / or decode the flag, and the context selection depends on block width, block height, block area, or neighboring pattern information.

[0396] In another sub-embodiment, when signaling indicates that inter-CCLM is to be applied, additional signaling is used to select one or more LM modes (e.g., CCLM_LT, CCLM_L, CCLM_T, MMLM_L, MMLM_T, MMLM_L, or any subset / extension of the above modes) from the total candidate LM modes. For example, if one LM mode is selected, the LM prediction is generated by the selected one LM. Another example is that if multiple LM modes are selected, the LM prediction is generated by a mixture of prediction hypotheses from multiple LM modes. In another example, the additional signaling refers to an index in the bitstream, which can be a truncated unary codec with and / or without context.

[0397] In another sub-embodiment, when signaling indicates that inter-frame CCLM is to be applied, one or more LMs (e.g., CCLM_LT, CCLM_L, CCLM_T, MMLM_L, MMLM_T, MMLM_L, or any subset / extension of the above modes) are implicitly selected (or predefined) from the total candidate LM modes for inter-frame CCLM.

[0398] – For example, when (*) is used to generate cross-component predictions, a candidate is implicitly selected from modellIdx.

[0399] – For example, CCLM_LT is used to generate LM predictions for inter-frame CCLM. For example, MMLM_LT is used to generate LM predictions for inter-frame CCLM.

[0400] – Another example is that the predefined rule depends on the block width, block height or block area.

[0401] – Boundary matching settings (used as predefined rules) are only applied when the block width, block height or block area is larger than the threshold.

[0402] – Boundary matching settings (used as predefined rules) are only applied when the block width, block height or block area is smaller than the threshold.

[0403] When the block width, block height, or block area is smaller than a threshold, the selected LM mode is inferred to be any one(s) of the total candidate LM modes.

[0404] – The selected LM mode is fixed to CCLM_LT.

[0405] – The selected LM mode is fixed to MMLM_LT.

[0406] Another example is that the predefined rules depend on the boundary matching settings. (Details of the boundary matching settings can be found in the Boundary Matching Settings section. The candidate mode used in the Boundary Matching Settings section refers to the candidate LM mode for each inter CCLM. The prediction of the candidate mode used in the Boundary Matching Settings section refers to the prediction generated by each candidate LM mode, or refers to the hybrid prediction generated by each candidate LM mode and the original inter.)

[0407] In another embodiment, inter-CCLM is supported only when the size condition of the current block is satisfied.

[0408] In one sub-embodiment, the size condition is that the block width, block height, and / or block area are greater than a predefined threshold. The predefined threshold can be a positive integer such as 8, 16, 32, 64, 128, 256, etc. For example, the size condition is satisfied when both the block width and block height of the current chroma block are greater than the predefined threshold. For example, the size condition is satisfied when either the block width or block height of the current chroma block is greater than the predefined threshold.

[0409] In another sub-embodiment, the size condition is that the block width, block height, or block area is less than a predefined threshold. The predefined threshold can be a positive integer such as 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, etc. For example, the size condition is satisfied when both the block width and block height of the current chroma block are less than the predefined threshold.

[0410] In another embodiment, the inter mode used in the inter block depends on an enable flag. For example, if the inter mode is normal merge, the enable flag is called the normal merge flag. Another example is that if the inter mode is CIIP, the enable flag is called the CIIP flag. Another example is that if the inter mode is CIIP PDPC, the enable flag is called the CIIPPDPC flag. Another example is that the enable flag indicates enabled (1) means that the corresponding inter mode is applied to the current block. Another example is that the enable flag indicates disabled (0) means that the corresponding inter mode is not applied to the current block. Another example is that the enable flag is signaled in the bitstream and / or inferred in some cases. Another example is that the signaling of the enable flag depends on the block width, block height or block area.

[0411] In another embodiment, inter-frame prediction can be adjusted using neighboring reconstructed samples and a predefined weighting scheme. For example, when the current block is merged, the merged prediction is blended with the neighboring reconstructed samples. In another example, the proposed scheme is enabled based on the CIIP PDPC flag. (When the CIIP flag indicates enabled, the CIIP PDPC flag may be signaled.) In another example, the predefined weighting scheme follows PDPC weighting.

[0412] (Details can be found below)

[0413] – The inter-frame predictor of the normal merge mode uses the above R x,-1 and left R -1,y Reconstruct the sample for optimization, such as Figure 30 shown

[0414] – The derivation of nScale and wT&wL is the same as for intra planar mode

[0415] –wT=32>>((y'<<1)>>nScale)

[0416] –wL=32>>((x'<<1)>>nScale)

[0417] –nScale=(floorLog2(width)+floorLog2(height)-2)>>2;

[0418] –CIIP PDPC:

[0419] – If LMCS is enabled, the inter predictor is computed in the mapped domain

[0420] –Pred(x,y)=((((wT×R x,-1 +wL×R -1,y +32)>>6)<<6)+(64-wT-wL)×Fwd(predInter(x,y))+32)>>6

[0421] – Otherwise, the inter predictor is computed in the original domain

[0422] –Pred(x,y)=((((wT×R x,-1 +wL×R -1,y +32)>>6)<<6)+(64-wT-wL)×predInter(x,y)+32)>>6

[0423] – When the CIIP flag is true, the CIIP PDPC flag is further issued to indicate whether CIIP PDPC is used

[0424] In another embodiment, the original inter prediction (generated by motion compensation) is used for the luma component and the predictions for the chroma components are generated by a Cross Component Linear Model (CCLM) and / or any other LM mode.

[0425] In one sub-embodiment, the current coding unit (CU) is considered to be an inter CU, an intra CU, or a new type of prediction mode (neither intra nor inter).

[0426] The above proposed method can also be applied to IBC blocks. (“Inter” in this section can be changed to IBC.) That is, for chroma components, block vector prediction can be combined with or replaced by CCLM (or any cross-component mode) prediction.

[0427] Boundary Matching Settings

[0428] When using the boundary matching setting, the boundary matching cost of the candidate mode refers to the discontinuity measure (including top boundary matching and / or left boundary matching) between the current prediction (the predicted samples in the current block generated by the candidate mode) and the neighboring reconstruction (the reconstructed samples in one or more neighboring blocks), such as Figure 31 The top boundary matching refers to the comparison between the current top prediction sample and the adjacent top reconstructed sample, and the left boundary matching refers to the comparison between the current left prediction sample and the adjacent left reconstructed sample.

[0429] In one embodiment, the candidate pattern with the smallest boundary matching cost is applied to the current block.

[0430] In another embodiment, the boundary matching costs of Cb and Cr can be added to the boundary matching cost of chrominance, so the selected candidate modes of Cb and Cr will be shared. (That is, the selected candidate modes of Cb and Cr will be the same.)

[0431] In another embodiment, the selected candidate modes of Cb and Cr depend on the boundary matching costs of Cb and Cr, respectively, and thus the selected candidate modes of Cb and Cr may be the same or different.

[0432] In another embodiment, a predefined subset of the current prediction is used to calculate the boundary matching cost. The n-line top boundary and / or the m-line left boundary within the current block are used. (Alternatively, the n2-line top and / or m2-line left boundary of the adjacent reconstruction are used.)

[0433] –Here is an example of calculating the cost of boundary matching. (n=2,m=2,n2=2,m2=2)

[0434]

[0435] The weights (a, b, c, d, e, f, g, h, i, j, k, l) can be any positive integers, for example, a = 2, b = 1, c = 1, d = 2, e = 1, f = 1, g = 2, h = 1, i = 1, j = 2, k = 1, l = 1.

[0436] –Here is another example of calculating the cost of boundary matching. (n=2,m=2,n2=1,m2=1)

[0437]

[0438] The weights (a, b, c, g, h, i) can be any positive integers, for example, a=2, b=1, c=1, g=2, h=1, i=1.

[0439] –Here is another example of calculating the cost of boundary matching. (n=1,m=1,n2=2,m2=2)

[0440]

[0441] The weights (d, e, f, j, k, l) can be any positive integers, for example, d = 2, e = 1, f = 1, j = 2, k = 1, l = 1.

[0442] –Here is another example of calculating the cost of boundary matching. (n=1,m=1,n2=1,m2=1)

[0443]

[0444] The weights (a, c, g, i) can be any positive integers, for example, a=1, c=1, g=1, i=1.

[0445] –Here is another example of calculating the cost of boundary matching. (n=2,m=1,n2=2,m2=1)

[0446]

[0447] The weights (a, b, c, d, e, f, g, i) can be any positive integers, for example, a = 2, b = 1, c = 1, d = 2, e = 1, f = 1, g = 1, i = 1.

[0448] –Here is another example of calculating the cost of boundary matching. (n=1,m=2,n2=1,m2=2)

[0449]

[0450] The weights (a, c, g, h, i, j, k, l) can be any positive integers, for example, a=1, c=1, g=2, h=1, i=1, j=2, k=1, l=1.

[0451] (The following examples about n and m also apply to n2 and m2.)

[0452] For example, n can be any positive integer, such as 1, 2, 3, 4, etc.

[0453] For example, m can be any positive integer, such as 1, 2, 3, 4, etc.

[0454] For example, n and / or m vary with the width, height, or area of ​​the block.

[0455] – One possible approach is that for larger blocks (area > threshold2), m becomes larger.

[0456] –Threshold2=64,128,or256.

[0457] – When area > threshold2, m increases to 2. (Originally, m is 1.)

[0458] – When area > threshold2, m increases to 4. (Originally, m is 1 or 2.)

[0459] – Another possible approach is that for taller blocks (height > thrershold2*width) m becomes larger and / or n becomes smaller.

[0460] –Threshold2=1, 2, or 4.

[0461] – When height > thrershold2 * width, m increases to 2. (Originally, m is 1.)

[0462] – When height > thrershold2 * width, m increases to 4. (Originally, m is 1 or 2.)

[0463] – Another possible approach is that for larger blocks (area > threshold2), n becomes larger.

[0464] –Threshold2=64,128,or256.

[0465] – When area > threshold2, n increases to 2. (Originally, n is 1.)

[0466] – When area > threshold2, n increases to 4. (Originally, n is 1 or 2.)

[0467] – Another possible approach is that for wider blocks (width > thrershold2*height), n becomes larger and / or m becomes smaller.

[0468] –Threshold2=1, 2, or 4.

[0469] – When width > thrershold2 * height, n increases to 2. (Originally, n is 1.)

[0470] – When width > thrershold2 * height, n increases to 4. (Originally, n is 1 or 2.)

[0471] LM auxiliary angle / plane mode

[0472] For traditional intra prediction modes (e.g., angular intra prediction mode, DC, and planar), reference samples are taken from the top and left neighboring reconstructed samples. Therefore, the accuracy of intra prediction decreases for the bottom right sample within the current block. In this section, LM (or cross-component mode) is used to improve the prediction of traditional intra prediction modes.

[0473] In one embodiment, the prediction of the current block is formed by a weighted sum of one or more prediction hypotheses from the traditional intra prediction mode and one or more prediction hypotheses from the LM mode (or inter-component mode).

[0474] - In one sub-embodiment, equal weight is applied to both.

[0475] - In another sub-embodiment, the weight varies with neighboring codec information, sample position, block width, height, mode or area.

[0476] For example, when the sample position is far away from the upper left region, the prediction weight from the traditional intra prediction mode decays.

[0477] –For more weighting schemes, please refer to the “Inverse LM” section.

[0478] Some examples are shown below.

[0479] – A possible rule related to sample location is described below.

[0480] As the sample position moves away from the reference sample, the prediction weights from other intra prediction modes (non-inter-component modes) decrease.

[0481] – Another possible rule related to neighboring codec information is described below.

[0482] When more neighboring blocks (left, above, upper left, upper right, and / or lower left) are coded in a specific mode (e.g., mode A), the prediction weight from mode A becomes higher. For example, mode A refers to a specific cross-component mode, such as CCCM_LT, or a specific cross-component family, such as the CCCM family (including CCCM_LT, CCCM_L, and / or CCCM_T), and / or the CCLM family (including CCLM_LT, CCLM_L, and / or CCLM_T), and / or the MMLM family (including MMLM_LT, MMLM_L, and / or MMLM_T). The predefined weighting set includes multiple weighted candidates, such as {1, 3}, {3, 1}, and / or equally weighted {2, 2}, for one prediction from a traditional intra prediction mode and another prediction from a cross-component mode. When all or most neighboring blocks are coded by mode A, weighted candidates with higher weights are used for prediction from the cross-component mode. When only part or one neighboring block is coded by mode A, weighted candidates with equal weights are used. When few or no neighboring blocks are coded by mode A, weighted candidates with smaller weights are used for prediction from the cross-component mode. For example, the neighboring blocks include any subset of coded blocks that are spatially adjacent to the top boundary or left boundary of the current block. In this case, the neighboring blocks may refer to the top neighboring blocks (located at the top of the upper right corner of the current block) and the left neighboring blocks (located to the left of the lower left corner of the current block). For another example, the neighboring blocks include any subset of coded blocks that are within a predefined range and spatially adjacent to the top boundary or left boundary of the current block. In this case, the neighboring blocks may be adjacent to or non-adjacent to the current block.

[0483] – Another possible rule related to sample location is described below.

[0484] The current block is divided into several regions. Sample positions within the same region share the same weight. If the current region is close to the reference L neighbor, the prediction weight from other intra prediction modes (non-cross-component mode) is higher than the prediction weight from CCLM (or cross-component mode). The following shows some possible ways to divide the current block.

[0485] – (width and height ratio is close to or exactly 1:1): consider the distance between the current region and the left and top reference L neighbors.

[0486] – (width > n*height, where n can be any positive integer): considers the distance between the current region and the top reference L neighbors.

[0487] – (height > n*width, where n can be any positive integer): consider the distance between the current region and the reference L neighbors on the left.

[0488] (*) One or more LM modes (or cross-component modes) used to generate one or more prediction hypotheses for LM-assisted angular / planar mode and / or inter CCLM and / or MH CCLM are selected from a predefined merge candidate list (called modelList). A modelIdx is transmitted to select a candidate from the candidate list (modelList), and the selected candidate is used for the current block. modelList contains one or more candidates, where each candidate refers to one model (or cross-component mode) information. If there is only one candidate in the list (the size of the list is only 1), modelIdx is not transmitted and / or can be inferred to be 0 or a default value.

[0489] In one embodiment, when constructing modelList, one or more predefined candidates are added. The predefined candidates may include any subset / extension of the following candidates.

[0490] –CCLM series: CCLM_LT, CCLM_L, CCLM_T

[0491] –MMLM series: MMLM_LT, MMLM_L, MMLM_T

[0492] –CCCM series: CCCM_LT, CCCM_L, CCCM_T

[0493] In another embodiment, when constructing the modelList (similar to the merged candidate list), one or more of the following candidate types are included.

[0494] - Spatial model information from spatially neighboring blocks (corresponds to “spatial motion vector prediction from spatially neighboring coding units” for inter frames)

[0495] - Temporal model information from the corresponding block (corresponds to “temporal motion vector prediction from the corresponding coding unit” for inter frames)

[0496] – History-based model information from FIFO table (corresponds to “History-based motion vector prediction from FIFO table” for inter frames)

[0497] – Pairwise average model information (corresponding to “pairwise average motion vector prediction” for inter frames)

[0498] – Default model information (corresponding to “zero motion vector” for inter-frames)

[0499] In a sub-embodiment where the candidate type is "spatial model information from spatially neighboring blocks", valid spatially neighboring blocks may be from spatially adjacent and / or non-adjacent neighbors (or a subset of any blocks in the neighboring search area of ​​the current block) that meet a predefined condition. For example, the predefined condition is that the neighbor is encoded and / or decoded by or using a cross-component mode (e.g., CCLM, MMLM, CCCM, GLM, a mode that inherits mode information from a similar merged candidate list, MH CCLM, and / or any cross-component mode whose syntax does not belong to a traditional intra-frame prediction mode) or a cross-component dependent mode (e.g., chroma fusion (or LM-assisted angular / planar mode), inter CCLM, and / or any traditional mode whose syntax does not belong to a cross-component mode but uses cross-component information to generate predictions). When scanning spatially neighboring blocks, if a candidate is valid, the candidate is added to the list. The following shows some scanning orders for adding spatial model information from spatially neighboring blocks to the list.

[0500] – For example, Figure 32 As shown, the scanning order follows B1 (top)(A1 (left)(B0 (top right)(A0 (bottom left)(B2 (top left)) or any predefined order.

[0501] - For example, the scanning order follows that neighboring candidates take precedence over (or after) non-neighboring candidates.

[0502] In another sub-embodiment, the temporal model information comes from a corresponding block from a reference picture or a corresponding picture as an inter-mode.

[0503] In another sub-embodiment, based on historical model information, a history-based table (FIFO table) is constructed and stores model information from previously coded blocks. The table can be reset at the beginning and / or end of a coding tree unit (CTU), slice, picture, tile, and / or sequence. One or more candidates can be added to the list based on the table.

[0504] In another sub-embodiment, model information is averaged pairwise, where the model information for a candidate is derived based on model information for a plurality of previous candidates in the list.

[0505] In another sub-embodiment, if the list is not full after inserting all predefined candidates, default model information is added. Some examples of default model information are shown below.

[0506] – For example, the default alpha (or α, a, or scaling parameter) is {0, 1 / 8, -1 / 8, 2 / 8, -2 / 8, 3 / 8}, and beta (or β, b, or offset parameter) is based on the selected default alpha, the average neighboring reconstructed luma sample value, and the average neighboring reconstructed chroma (Cb / Cr) sample value.

[0507] In another sub-embodiment, when a candidate is selected from the list and the model information of the candidate is used for the current block or the model information is inherited from the previously encoded block (when a candidate is placed in the list), only a subset of the model information is inherited. For example, only alpha is inherited. The beta of the current block is obtained by inheriting the alpha, the average neighboring reconstructed luminance sample value and / or the average neighboring reconstructed chrominance (Cb / Cr) sample value. For example, the MMLM model information is inherited, the scaling parameters and / or the classification threshold are inherited. The offset parameters in each class are derived based on the inherited classification threshold and / or the average neighboring reconstructed luminance sample value and / or the average neighboring reconstructed chrominance (Cb / Cr) sample value in each class. If no neighboring reconstructed samples are available in a class, the offset parameters are inherited directly from the candidate. For example, the CCCM model information is inherited, all convolution parameters, offsets and / or classification thresholds are inherited. For example, if the GLM model information is inherited, if the GLM candidate is a 3-parameter GLM mode, all gradient mode indices and model parameters are inherited; otherwise, if the GLM candidate is a 2-parameter GLM mode, the offset parameters are derived by using the inherited scaling parameters, average neighboring reconstructed luma sample values, and / or average neighboring reconstructed chroma (Cb / Cr) sample values. For example, if the chroma fusion model information is inherited, the derived MMLM parameters are inherited and used, just as if the MMLM candidate for the current block was inherited.

[0508] In another sub-embodiment, when a candidate is selected from the list and the model information of the candidate is used for the current block, all model information is inherited, for example, alpha and beta.

[0509] In another sub-embodiment, the mode to be propagated (using chroma fusion or referred to as LM-assisted angular / planar mode, inter-frame CCLM or MH CCLM) is set / stored as an inherited mode, referring to any cross-component mode and / or any cross-component model and / or a subset of any cross-component model information used for the current block, such as CCLM, MMLM, CCCM and / or GLM.

[0510] In another sub-embodiment, the candidate types are aligned with the candidate types of the merge mode.

[0511] In another sub-embodiment, when building the list, a pruning operation is applied to avoid duplicate candidates in the list.

[0512] In another sub-embodiment, the signaling of modelIdx of the current block depends on the context codec, block width, block height, block area and / or explicit syntax, such as signaling at SPS, PPS, slice, CTU, picture, sequence and / or tile level.

[0513] In another sub-embodiment, the candidate selection (from the list) for the current block depends on a predefined process. For example, the predefined process is a TIMD or DIMD-like method. When the predefined process refers to a TIMD-like method, (1) the cost of each candidate in the list is calculated based on the distortion of the reconstructed sample on the neighboring template and the predicted sample of the candidate on the neighboring template, (2) one or more candidates with smaller cost (or smaller distortion) are selected, and / or (3) a cross-component prediction of the current block is generated using the one or more selected candidates. For example, the predefined process depends on the neighboring templates of the current block. For example, the predefined process depends on the mode information on the neighboring blocks. If most of the predefined neighboring blocks use a specific mode, the first candidate (in the list) pointing to the specific mode is selected. The signaling of modelIdx is bypassed (no signaling is required).

[0514] In another embodiment, it is proposed to use the LM mode to generate the lower right area within or near the current block. When performing intra prediction, the reference samples can be based not only on the original left and top neighboring reconstructed samples, but also on the proposed right and bottom LM predicted samples. An example is shown below.

[0515] – Before performing intra prediction on chroma blocks, reconstruct the corresponding luma blocks.

[0516] – “Neighboring luma reconstructed samples of the corresponding luma block” and “neighboring chroma reconstructed samples of the current chroma block” are used to derive LM parameters.

[0517] – The “reconstructed samples of the corresponding luminance block” using the derived parameters are used to obtain the lower right LM prediction samples of the current chrominance block. (The lower right area of ​​the current chrominance block can be Figure 33 Any subset of regions in

[0518] – The prediction of the current block is generated bidirectionally by referring to the original L neighboring regions (the original top and left regions obtained using conventional intra prediction mode) and the proposed inverse L region (obtained using LM).

[0519] In one sub-embodiment, the predictors from the original top and left regions are weighted in combination with the predictors from the bottom and left regions.

[0520] – In one example, equal weight is applied to both.

[0521] – In one example, the weights vary with neighboring codec information, sample position, block width, height, mode, or region.

[0522] For example, the prediction weight of the conventional intra prediction mode decays when the sample position is far away from the top and left regions.

[0523] In another embodiment, the proposed method can be applied to the inverse LM. Then, when performing luma intra prediction, the final prediction is bidirectional. (Similar to the example of chroma blocks above)

[0524] In another embodiment, after partitioning is performed to understand the curve pattern of luminance, the proposed LM auxiliary angle / plane pattern helps chrominance to obtain the correct curve angle.

[0525] The method proposed in this invention can be enabled and / or disabled according to implicit rules (such as block width, height or area) or according to explicit rules (such as syntax at block, slice, picture, sps or pps level).

[0526] The term "block" in the present invention may refer to TU / TB, CU / CB, PU / PB or CTU / CTB.

[0527] The term "LM" in the present invention may be considered as a CCLM / MMLM mode or any other extension / variant of CCLM (e.g., the CCLM extension / variant suggested in the present invention). The CCLM variant here means that when a block indicates the use of one of the cross-component modes (e.g., CCLM_LT, MMLM_LT, CCLM_L, CCLM_T, MMLM_L, MMLM_T, and / or an intra-prediction mode, other than one of the traditional DC, planar, and angular modes), some optional modes may be selected for the current block. An example is shown below with the convolution cross-component mode (CCCM) as an optional mode. When this optional mode is applied to the current block, chrominance predictions are generated using cross-component information with a model (possibly including nonlinear terms). The optional modes may follow the template selection of CCLM, so the CCCM family includes CCCM_LT, CCCM_L, and / or CCCM_T.

[0528] The method proposed in this invention (for CCLM) can be used for any other LM mode.

[0529] Any combination of the methods suggested in the present invention may be applied.

[0530] Any of the above-mentioned proposed methods can be implemented in an encoder and / or decoder. For example, any of the proposed methods can be implemented in an intra / inter / IBC codec module of an encoder and / or decoder, and / or in a motion compensation module and / or a merge candidate derivation module of an encoder and / or decoder. Alternatively, any of the proposed methods can be implemented as a circuit connected to the intra / inter / IBC codec module and / or the motion compensation module and / or the merge candidate derivation module of an encoder and / or decoder.

[0531] Figure 34 is a block diagram illustrating a video encoder supporting the proposed inter-frame chroma prediction mode according to an embodiment of the present invention. By way of example, but not limitation, the video encoder 100 may be a VVC encoder. The video encoder 100 may perform intra-frame and inter-frame prediction encoding and decoding on video blocks within a video frame. Intra-frame prediction encoding and decoding relies on spatial prediction to reduce or eliminate spatial redundancy in video data within a given video frame or picture. Inter-frame prediction encoding and decoding relies on temporal prediction to reduce or eliminate temporal redundancy in video data within adjacent video frames or pictures in a video sequence. The proposed inter-frame chroma prediction mode is a coding tool for chroma samples in a chroma block and uses at least one cross-component prediction (which is an intra-frame prediction) to improve the accuracy of motion compensated prediction (which is an inter-frame prediction).

[0532] like Figure 34 As shown, the video encoder 100 includes an encoding circuit 101 and a video data memory 102. The video data memory 102 is configured to receive data of a current pixel block of a current picture to be encoded as a video, wherein the current block includes at least one chroma block. The encoding circuit 101 is configured to encode the current block through a target inter-frame prediction mode. The encoding circuit 101 may include a prediction processing circuit 104, a residual generation circuit 106, a transform circuit (marked as "T") 108, a quantization circuit (marked as "Q") 110, an entropy encoding circuit (e.g., a variable length coding (VLC) encoder) 112, an inverse transform circuit (marked as "IQ") 114, an inverse transform circuit (marked as "IT") 116, a reconstruction circuit 118, one or more loop filters 120, and a decoded picture buffer (DPB) 122. It should be noted that Figure 34 The encoder architecture shown in FIG is for illustration purposes only and is not meant to limit the present invention. In fact, any video encoder that uses / supports the proposed inter-frame chroma prediction mode is within the scope of the present invention.

[0533] The prediction processing circuit 104 may include a partition circuit 124, a motion estimation circuit (labeled "ME") 126, a motion compensation circuit (labeled "MC") 128, an intra-frame prediction circuit (labeled "IP") 130, and an inter-frame chroma prediction circuit (labeled "ICP") 132. The proposed inter-frame chroma prediction mode is supported by the prediction processing circuit 104 (particularly the inter-frame chroma prediction circuit 132 of the prediction processing circuit 104). Since the present invention focuses on the proposed inter-frame chroma prediction mode, and those skilled in the art should be able to easily understand the details of other circuit components included in the video encoder 100, further description of the principles of the other circuit components included in the video encoder 100 is omitted here for brevity.

[0534] The motion compensation circuit 128 of the prediction processing circuit 104 is configured to obtain a motion compensated prediction MCP of the chroma samples contained in the chroma block (e.g., a Cr block or a Cb block). For example, the motion estimation circuit 126 can determine the motion vector of the luminance block in the same codec unit according to the selected inter-frame prediction mode, and the motion vector of the luminance block can be directly inherited to the chroma block in the same codec unit. Therefore, the motion compensation circuit 128 can refer to the motion vector of the luminance block to determine the motion compensated prediction MCP (this is an inter-frame prediction) of the chroma samples contained in the chroma block. However, this is for illustrative purposes only and is not intended to limit the present invention. In fact, any inter-frame prediction method that can obtain a motion compensated prediction MCP of the chroma samples contained in the chroma block (e.g., a Cr block or a Cb block) can be adopted by the prediction processing circuit 104.

[0535] The inter-frame chroma prediction circuit 132 is configured to obtain at least one (i.e., one or more) cross-component prediction CCPs of the chroma samples and determine the inter-frame prediction of the chroma samples (e.g., final prediction P_CB / CR) by jointly considering the motion compensated prediction MCP and the cross-component prediction CCP. For example, the inter-frame chroma prediction circuit 132 mixes the motion compensated prediction MCP and the cross-component prediction CCP to generate a weighted prediction as the inter-frame prediction of the chroma samples. In some embodiments of the present invention, the cross-component prediction CCP mixed with the motion compensated prediction MCP may include a CCLM-based prediction and / or a prediction generated using any cross-component mode. In some embodiments of the present invention, the motion compensated prediction MCP and the cross-component prediction CCP may be mixed using a predefined (fixed) weighting, such as {3:1} or {1:3}.

[0536] Regarding the generation of the cross-component prediction CCP, the inter-frame chroma prediction circuit 132 may construct a merge candidate list MCL for cross-component prediction and determine the cross-component prediction CCP based on the cross-component model candidate selected from the merge candidate list MCL. In some embodiments of the present invention, the candidate type may be aligned with the type used in the inter-frame motion merge candidate list, or may be a subset of the type used in the inter-frame motion merge candidate list. The difference between the merge candidate list MCL and the inter-frame motion merge candidate list is that the merge candidate list MCL includes inherited cross-component models, while the inter-frame motion merge candidate list includes inherited motion information.

[0537] For example, the merge candidate list MCL may include spatial candidates. Specifically, the inter-frame chroma prediction circuit 132 may add a cross-component model located at a spatially adjacent position to the merge candidate list MCL. The spatially adjacent position and the sample position of the chroma sample are located in the same frame. The spatially adjacent position can be an adjacent position or a non-adjacent position relative to the current chroma block boundary.

[0538] For example, the merge candidate list MCL may include temporal candidates. Specifically, the inter-frame chroma prediction circuit 132 may add a cross-component model located at a temporally corresponding position to the merge candidate list MCL. The temporally corresponding position and the sample position of the chroma sample are located at the same position in different frames.

[0539] For another example, the merge candidate list MCL may include history-based candidates. Specifically, the inter-frame chroma prediction circuit 132 may add cross-component models to the merge candidate list MCL from a history table, where the history table may be implemented by a first-in-first-out (FIFO) buffer that contains cross-component models of previously encoded chroma blocks (i.e., chroma blocks encoded before the current chroma block).

[0540] For another example, the merge candidate list MCL may include a default candidate. Specifically, the inter-frame chroma prediction circuit 132 may add a default cross-component model (ie, a predefined cross-component model) to the merge candidate list MCL.

[0541] In some embodiments of the present invention, after the inter-frame chroma prediction circuit 132 selects a cross-component model candidate from the merge candidate list MCL for subsequent calculation of the cross-component predictor CCP, the inter-frame chroma prediction circuit 132 can output a mode index IDX to the entropy coding circuit 112 so that the mode index IDX is encoded into the encoded video bitstream output from the video encoder 100. Specifically, the pixel data of the current block of the current picture to be encoded as a video includes a syntax for indicating the mode index for selecting the cross-component model candidate. For example, the mode index IDX is signaled to indicate the selection of the cross-component model candidate at the video encoder 100. Alternatively, the selection of the cross-component model candidate can be implicitly derived without any mode index signaling. For example, the inter-frame chroma prediction circuit 132 can adopt an implicit derivation method similar to that used by TIMD. Specifically, the inter-frame chroma prediction circuit 132 calculates the template matching (TM) cost of each cross-component model candidate included in the merge candidate list MCL using reference samples of neighboring templates, and selects the cross-component model candidate with the minimum TM cost included in the merge candidate list MCL as the cross-component model candidate for obtaining the cross-component predictor CCP. Since both the video encoder and the video decoder select cross-component model candidates from the same merge candidate list MCL constructed in the video encoder and the video decoder in the same template-based manner, there is no need to signal the pattern index IDX from the video encoder to the video decoder. In this way, signaling overhead can be reduced.

[0542] As described above, when the proposed inter-frame chroma prediction is enabled, the inter-frame chroma prediction circuit 132 uses the cross-component predictor CCP to improve the accuracy of the inter-frame predictor (e.g., the motion compensated predictor MCP). In some embodiments of the present invention, the prediction processing circuit 104 does not enable the proposed inter-frame chroma prediction mode unless all enabling conditions are met. For example, the enabling condition may include a size condition of the chroma block, where the size condition may be defined by block height, block width, and / or block area. Therefore, the size condition is met when the block height and / or width and / or area are less than (or greater than) a predefined threshold.

[0543] For another example, the enabling condition may include supported prediction modes for the chroma block, where the supported prediction modes may include only inter modes or any sub-modes of inter modes. Thus, the prediction mode condition is satisfied when the selected prediction mode used to determine the motion compensated predictor MCP is one of the supported inter modes (or any sub-modes of the supported inter modes).

[0544] In addition, when all enabling conditions are met, the pixel data of the current block of the current picture to be encoded as a video includes a flag to indicate the application of the target inter-frame prediction mode. For example, when all enabling conditions are met, the inter-frame chroma prediction circuit 132 sets a flag EN (e.g., EN=1) to indicate the application of the proposed inter-frame chroma prediction mode, and outputs the flag EN (e.g., EN=1) to the entropy coding circuit 112. Therefore, the entropy coding circuit 112 encodes the flag EN (e.g., EN=1) into the encoded video bitstream. In this way, the flag EN (e.g., EN=1) is signaled from the video encoder 100 to the video decoder via the encoded video bitstream. When the proposed inter-frame chroma prediction is not enabled, the flag EN is not signaled by the encoder and / or inferred at the encoder.

[0545] Figure 35 is a block diagram illustrating a video decoder supporting the proposed inter-frame chroma prediction mode according to an embodiment of the present invention. As an example, but not limited to, the video decoder 200 may be a VVC decoder. The video decoder 200 includes a decoding circuit 201 and a video data memory 202. The video data memory 202 is configured to receive pixel data of a current block of a current picture to be decoded as a video, wherein the current block includes at least one chroma block. The decoding circuit 201 is configured to decode the current block through a target inter-frame prediction mode. The decoding circuit 201 may include an entropy decoding circuit (e.g., a VLC decoder) 204, an inverse quantization circuit (labeled as "IQ") 206, an inverse transform circuit (labeled as "IT") 208, a reconstruction circuit 210, a prediction processing circuit 212, one or more loop filters 214, and a decoded picture buffer (DPB) 216. It should be noted that Figure 35 The decoder architecture shown is for illustration purposes only and is not meant to limit the present invention. In fact, any video decoder that uses / supports the proposed inter-frame chroma prediction mode falls within the scope of the present invention.

[0546] The prediction processing circuit 212 may include a motion compensation circuit (labeled "MC") 218, an intra-frame prediction circuit (labeled "IP") 220, and an inter-frame chroma prediction circuit (labeled "ICP") 222. The proposed inter-frame chroma prediction mode is supported by the prediction processing circuit 212 (particularly, the inter-frame chroma prediction circuit 222 of the prediction processing circuit 212). Since the present invention focuses on the proposed inter-frame chroma prediction mode, and those skilled in the art should be able to easily understand the details of other circuit components included in the video decoder 200, further description of the principles of the other circuit components included in the video decoder 200 is omitted here for brevity.

[0547] The motion compensation circuit 218 of the prediction processing circuit 212 is configured to obtain a motion compensated prediction (i.e., an inter-frame prediction) MCP of chroma samples contained in a chroma block (e.g., a Cr block or a Cb block). The motion vector used by the decoder-side motion compensation circuit 218 to obtain the motion compensated prediction MCP is the same as the motion vector used by the encoder-side motion compensation circuit 128 to obtain the motion compensated prediction MCP.

[0548] The functionality of the decoder-side inter-frame chroma prediction circuit 222 is similar / identical to that of the encoder-side inter-frame chroma prediction circuit 132. The inter-frame chroma prediction circuit 222 is configured to obtain at least one (i.e., one or more) cross-component prediction (i.e., intra-frame prediction) CCP of the chroma sample and determine the inter-frame prediction (e.g., final prediction P_CB / CR) of the chroma sample by jointly considering the motion-compensated prediction MCP and the cross-component prediction CCP. For example, the inter-frame chroma prediction circuit 222 blends the motion-compensated prediction MCP and the cross-component prediction CCP to generate a weighted prediction as the inter-frame prediction of the chroma sample. In some embodiments of the present invention, the cross-component prediction CCP may include a CCLM-based prediction and / or a prediction generated using any cross-component mode. In some embodiments of the present invention, the motion-compensated prediction MCP and the cross-component prediction CCP may be blended using a predefined (fixed) weighting, such as {3:1} or {1:3}.

[0549] Regarding the generation of the cross-component prediction CCP, the inter-frame chroma prediction circuit 222 can construct a merge candidate list MCL for cross-component prediction and determine the cross-component prediction CCP based on the cross-component model candidate selected from the merge candidate list MCL. In some embodiments of the present invention, the candidate type can be aligned with the type used in the inter-frame motion merge candidate list, or can be a subset of the type used in the inter-frame motion merge candidate list. The merge candidate list MCL constructed by the decoder-side inter-frame chroma prediction circuit 222 is the same as the one constructed by the encoder-side inter-frame chroma prediction circuit 132.

[0550] For example, the merge candidate list MCL may include spatial candidates. Specifically, the inter-frame chroma prediction circuit 222 may add a cross-component model located at a spatially adjacent position to the merge candidate list MCL. The spatially adjacent position and the sample position of the chroma sample are located in the same frame. The spatially adjacent position can be an adjacent position or a non-adjacent position relative to the current chroma block boundary.

[0551] For another example, the merge candidate list MCL may include temporal candidates. Specifically, the inter-frame chroma prediction circuit 222 may add a cross-component model located at a temporally corresponding position to the merge candidate list MCL. The temporally corresponding position and the sample position of the chroma sample are located at the same position in different frames.

[0552] For another example, the merge candidate list MCL may include history-based candidates. Specifically, the inter-frame chroma prediction circuit 222 may add cross-component models to the merge candidate list MCL from a history table, where the history table may be implemented by a first-in-first-out (FIFO) buffer containing cross-component models of previously encoded chroma blocks (i.e., previous chroma blocks decoded before the current chroma block).

[0553] For another example, the merge candidate list MCL may include a default candidate. Specifically, the inter-frame chroma prediction circuit 222 may add a default cross-component model (ie, a predefined cross-component model) to the merge candidate list MCL.

[0554] When a mode index IDX (which indicates the selection of a cross-component model candidate) is transmitted from a video encoder (e.g., video encoder 100) to a video decoder 200, the entropy decoding circuit 204 can parse the mode index IDX from the encoded video bitstream and notify the prediction processing circuit 212 (particularly, the inter-frame chroma prediction circuit 222 of the prediction processing circuit 212) of the mode index IDX. Specifically, the data of the current pixel block of the current picture to be decoded as a video includes syntax for indicating the mode index for selecting the cross-component model candidate. For example, the inter-frame chroma prediction circuit 222 refers to the signaled mode index IDX and selects the same cross-component model candidate used by the video encoder (e.g., video encoder 100) from the merge candidate list MCL.

[0555] Alternatively, the selection of cross-component model candidates can be implicitly derived without parsing any pattern index signaled from the encoded video bitstream. For example, the inter-frame chroma prediction circuit 222 can adopt an implicit derivation method similar to that used by TIMD. Specifically, the inter-frame chroma prediction circuit 222 uses reference samples of neighboring templates to calculate the cost of each cross-component model candidate contained in the merge candidate list MCL, and selects the cross-component model candidate with the smallest cost contained in the merge candidate list MCL as the cross-component model candidate for obtaining the cross-component prediction CCP. Since both the video encoder and the video decoder follow the same template-based approach to select cross-component model candidates from the same merge candidate list MCL constructed in the video encoder and video decoder, there is no need to signal the pattern index IDX from the video encoder to the video decoder, nor is there a need to parse the signaled pattern index IDX in the video decoder. In this way, signaling overhead can be reduced.

[0556] As described above, when the proposed inter-frame chroma prediction is enabled, the inter-frame chroma prediction circuit 222 uses the cross-component predictor CCP to improve the accuracy of the inter-frame predictor (e.g., the motion compensated predictor MCP). In some embodiments of the present invention, the prediction processing circuit 212 does not enable the proposed inter-frame chroma prediction mode unless all enabling conditions are met. For example, the enabling condition may include a size condition for the chroma block, where the size condition may be defined by block height, block width, and / or block area. Therefore, the size condition is met when the block height and / or width and / or area are less than (or greater than) a predefined threshold.

[0557] As another example, the enabling condition may include supported prediction modes for the chroma block, where supported prediction modes may include only inter modes or any sub-modes of inter modes. Thus, the prediction mode condition is satisfied when the selected prediction mode used to determine the motion compensated predictor MCP is one of the supported inter modes (or any sub-modes of the supported inter modes).

[0558] In addition, when the proposed inter-frame chroma prediction is enabled, for example, because all enabling conditions are met, the data of the current pixel block of the current picture to be decoded as a video includes a flag to indicate the application of the target inter-frame prediction mode. For example, when the proposed inter-frame chroma prediction is enabled, the inter-frame chroma prediction circuit 222 uses the cross-component predictor CCP to improve the accuracy of the inter-frame predictor (for example, the motion compensated predictor MCP). In the case where the flag EN (for example, EN=1) is sent from the video encoder (for example, the video encoder 100) to the video decoder 200, the entropy decoding circuit 204 parses the flag EN (for example, EN=1) from the encoded video bitstream and notifies the prediction processing circuit 212 (particularly the inter-frame chroma prediction circuit 222 of the prediction processing circuit 212) of the flag EN (for example, EN=1). Therefore, the prediction processing circuit 212 enables the proposed inter-frame chroma prediction mode in response to the flag EN (for example, EN=1) indicating the application of the proposed inter-frame chroma prediction mode. When the proposed inter chroma prediction is not enabled, the flag EN is not sent to the decoder and / or is not parsed from the decoder and / or inferred in the decoder.

[0559] Figure 36 FIG. 1 is a flow chart illustrating a video encoding and decoding method according to an embodiment of the present invention. The video encoding and decoding method may be Figure 34 The video encoder 100 shown in FIG. 1 is used for encoding video data or by Figure 35 The video decoder 200 shown in FIG is used to decode an encoded video bitstream. The steps do not need to be repeated as long as the results are substantially the same. Figure 36. In step 3602, data of a current pixel block of a current picture to be encoded or decoded as a video is received, wherein the current block includes at least one chroma block. In step 3604, the current block is encoded or decoded using a target inter-frame prediction mode. Step 3604 includes sub-steps 3606, 3608, and 3610. In sub-step 3606, a motion compensated prediction of the chroma samples contained in the at least one chroma block is obtained. In sub-step 3608, at least one cross-component prediction of the chroma samples is obtained. In sub-step 3610, an inter-frame prediction of the chroma samples is determined by jointly considering (e.g., mixing) the motion compensated prediction and the at least one cross-component prediction. Since those skilled in the art can easily understand the details of the video encoding and decoding method after reading the above paragraphs and referring to the accompanying drawings, further description is omitted here for brevity.

[0560] Those skilled in the art will readily appreciate that numerous modifications and variations can be made to the apparatus and method while retaining the teachings of the present invention. Accordingly, the above disclosure should be interpreted only in accordance with the limits of the appended claims.

Claims

1. A method for video encoding and decoding, comprising: Receiving data of a current pixel block to be encoded or decoded as a current picture of a video, wherein the current block includes at least one chroma block; as well as Encoding or decoding the current block using a target inter-frame prediction mode includes: obtaining a motion compensated prediction of chroma samples contained in the at least one chroma block; obtaining at least one cross-component prediction of the chroma sample; and An inter prediction of the chroma sample is determined by jointly considering the motion compensated prediction and the at least one cross-component prediction.

2. The method according to claim 1, wherein The at least one cross-component prediction includes a cross-component linear model (CCLM) based prediction.

3. The method according to claim 1, wherein Obtaining at least one cross-component prediction of the chroma sample comprises: Constructing a merge candidate list for cross-component prediction; and The at least one cross-component prediction is determined according to a cross-component model candidate selected from the merge candidate list.

4. The method according to claim 3, wherein Constructing the merge candidate list for cross-component prediction includes: Cross-component models that are located at spatially adjacent positions are added to the merge candidate list.

5. The method according to claim 3, wherein Constructing the merge candidate list for cross-component prediction includes: The cross-component models at the corresponding time positions are added to the merge candidate list.

6. The method according to claim 3, wherein Constructing the merge candidate list for cross-component prediction includes: A cross-component model is added to the merge candidate list from a history table, where the history table contains cross-component models of previously encoded chroma blocks.

7. The method according to claim 3, wherein Constructing the merge candidate list for cross-component prediction includes: Add the default cross-component model to the merge candidate list.

8. The method according to claim 3, wherein The data to be encoded as the current pixel block of the current picture includes syntax for indicating a mode index for selecting the cross-component model candidate.

9. The method according to claim 3, wherein The data to be decoded as the current pixel block of the current picture includes syntax for indicating a mode index for selecting the cross-component model candidate.

10. The method according to claim 3, wherein Determining the at least one cross-component prediction according to the cross-component model candidate selected from the merge candidate list includes: Calculating a cost for each cross-component model candidate included in the merge candidate list using reference samples of neighboring templates; and A cross-component model candidate with a minimum cost included in the merge candidate list is selected as the cross-component model candidate for obtaining the at least one cross-component prediction.

11. The method according to claim 1, wherein Determining the inter prediction of the chroma sample by jointly considering the motion compensated prediction and the at least one cross-component prediction includes: The motion compensated prediction and the at least one inter-component prediction are blended to generate a weighted prediction as an inter prediction of the chroma samples.

12. The method according to claim 11, wherein The at least one cross-component prediction includes a cross-component linear model (CCLM) based prediction.

13. The method according to claim 11, wherein The motion compensated prediction and the at least one cross-component prediction are mixed using a predefined weight.

14. The method according to claim 1, wherein The target inter prediction mode is enabled when all enabling conditions are met.

15. The method according to claim 14, wherein The enabling condition includes a size condition of the at least one chroma block.

16. The method according to claim 14, wherein The enabling condition includes a supported prediction mode of the at least one chroma block.

17. The method according to claim 14, wherein When all enabling conditions are met, the data to be encoded as the current pixel block of the current picture includes a flag to indicate application of the target inter-frame prediction mode.

18. The method according to claim 1, wherein When all enabling conditions are met, the data to be decoded as the current pixel block of the current picture includes a flag to indicate application of the target inter-frame prediction mode.

19. A video encoder comprising: a video data memory configured to receive data of a current pixel block to be encoded as a current picture of a video, wherein the current block includes at least one chroma block; as well as An encoding circuit is configured to encode the current block using a target inter-frame prediction mode, wherein the encoding circuit comprises: motion compensation circuitry configured to obtain motion compensated predictions of chroma samples contained in the at least one chroma block; as well as The inter-frame chroma prediction circuit is configured to obtain at least one cross-component prediction of the chroma sample and determine the inter-frame prediction of the chroma sample by jointly considering the motion compensated prediction and the at least one cross-component prediction.

20. A video decoder comprising: a video data memory configured to receive data to be decoded as a current pixel block of a current picture of a video, wherein the current block includes at least one chroma block; as well as A decoding circuit is configured to decode the current block using a target inter-frame prediction mode, wherein the decoding circuit comprises: motion compensation circuitry configured to obtain motion compensated predictions of chroma samples contained in the at least one chroma block; as well as The inter-frame chroma prediction circuit is configured to obtain at least one cross-component prediction of the chroma sample and determine the inter-frame prediction of the chroma sample by jointly considering the motion compensated prediction and the at least one cross-component prediction.

Images