Method and apparatus for cross-component prediction for video coding

Abstract

The present disclosure provides a method for decoding video data, comprising: obtaining a video block from a bitstream; obtaining an internal brightness sample point value of the video block, an external brightness sample point value of an external region of the video block and an external chrominance sample point value of the external region; determining a filter pattern based on the external luma sample value and the external chroma sample value, where the filter pattern includes one or more linear weighting coefficients for a corresponding one or more pre-computed luma sample values, and where the filter pattern includes one or more linear weighting coefficients for a corresponding one or more pre-computed luma sample values. The one or more pre-computed luma sample values comprise at least one pre-computed luma sample value obtained based on a sample difference between external luma sample values, predicting an internal chroma sample value of the video block based on the internal luma sample value using the filter mode; and obtaining a decoded video block using the predicted internal chroma sample values.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 356,841, filed on June 29, 2022, and U.S. Provisional Application No. 63 / 391,096, filed on July 21, 2022. The entire contents of the above applications are incorporated herein by reference. Technical Field

[0003] Generally speaking, aspects of the present disclosure relate to image / video coding and compression, and more specifically, aspects of the present disclosure relate to methods and apparatus for cross-component prediction techniques. Background Art

[0004] Various video codec technologies can be used to compress video data. Video coding is performed according to one or more video coding standards. For example, video coding standards include Versatile Video Codec (VVC), High Efficiency Video Codec (H.265 / HEVC), Advanced Video Codec (H.264 / AVC), Moving Picture Experts Group (MPEG) codec, etc. Video codecs typically use prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that exploit redundancy present in video images or sequences. An important goal of video coding technology is to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality. Summary of the invention

[0005] The following is a simplified summary of one or more aspects of the present disclosure to provide a basic understanding of these aspects. The summary is not an exhaustive overview of all contemplated aspects, and is neither intended to identify the key or important elements of all aspects nor to describe the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to a more detailed description given later.

[0006] According to one aspect of the present disclosure, a method for decoding video data is provided, comprising: obtaining a video block from a bitstream; obtaining an internal luma sample value of the video block, an external luma sample value of an external area of ​​the video block, and an external chroma sample value of the external area; determining a filter mode based on the external luma sample value and the external chroma sample value, wherein the filter mode includes one or more linear weighting coefficients for corresponding one or more pre-calculated luma sample values, and wherein the one or more pre-calculated luma sample values ​​include at least one pre-calculated luma sample value obtained based on a sample difference between external luma sample values, predicting an internal chroma sample value of the video block based on the internal luma sample value using the filter mode; and obtaining a decoded video block using the predicted internal chroma sample value.

[0007] According to another aspect of the present disclosure, a method for encoding video data is provided, comprising: obtaining a video block; obtaining internal luma sample values ​​of the video block, external luma sample values ​​of an external area of ​​the video block, and external chroma sample values ​​of the external area; determining a filter mode based on the external luma sample values ​​and the external chroma sample values, wherein the filter mode comprises one or more linear weighting coefficients for corresponding one or more pre-calculated luma sample values, and wherein the one or more pre-calculated luma sample values ​​comprise at least one pre-calculated luma sample value obtained based on a sample difference between external luma sample values, predicting internal chroma sample values ​​of the video block based on the internal luma sample values ​​using the filter mode; and generating a bitstream including an encoded video block by using the predicted internal chroma sample values.

[0008] According to another aspect of the present disclosure, a computer system is provided, including: one or more processors; and one or more storage devices storing computer executable instructions, which, when executed, cause the one or more processors to perform the operations of the method of the present disclosure.

[0009] According to another aspect of the present disclosure, a computer program product is provided, which stores computer executable instructions, which, when executed, cause the one or more processors to perform the operations of the method of the present disclosure.

[0010] According to another aspect of the present disclosure, a computer-readable medium is provided, which stores computer-executable instructions. When the instructions are executed, one or more processors receive a bit stream and perform the operations of the method of the present disclosure based on the bit stream.

[0011] According to another aspect of the present disclosure, a computer-readable medium is provided that stores computer-executable instructions that, when executed, cause one or more processors to perform the operations of the method of the present disclosure and send a bitstream including encoded video information associated with the predicted chroma samples.

[0012] According to another aspect of the present disclosure, a computer-readable medium storing a bit stream is provided, wherein the bit stream is to be decoded by performing the operation of the method of the present disclosure.

[0013] According to another aspect of the present disclosure, a computer-readable medium storing a bit stream is provided, wherein the bit stream is obtained by executing the operation of the method of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The disclosed aspects will hereinafter be described in conjunction with the accompanying drawings, which are provided for the purpose of illustrating rather than limiting the disclosed aspects.

[0015] Figure 1 A block diagram of a general block-based hybrid video coding system is shown.

[0016] FIG. 2A to FIG. 2E Five types of partitioning are shown, including quaternary partitioning, horizontal binary partitioning, vertical binary partitioning, horizontal ternary partitioning, and vertical ternary partitioning.

[0017] Figure 3 An overall block diagram of a block-based video decoder is shown.

[0018] Figure 4 An example of the positions of the left and upper samples and the samples of the current block involved in CCLM mode is shown.

[0019] FIG. 5A to FIG. 5C An example of deriving CCLM parameters is shown.

[0020] Figure 6 An example of classifying neighboring samples into two groups based on the value Threshold is shown.

[0021] Figure 7 An example of classifying neighboring samples into two groups based on inflection points is shown.

[0022] Fig. 8A and Figure 8B The effect of the scaling adjustment parameter "u" is shown.

[0023] Figure 8C Collocated reconstructed luminance samples are shown.

[0024] Fig.8D Neighboring reconstructed sample points are shown.

[0025] FIG. 8E to FIG. 8H The steps of intra mode derivation at the decoder side are shown.

[0026] Fig. 9 An example of four reference rows of neighboring prediction blocks is shown.

[0027] Fig. 10A An exemplary mode of a convolutional cross-component model (CCCM) is shown.

[0028] Fig. 10B An exemplary reference region is shown, consisting of 6 rows of chroma samples above and to the left of the PU.

[0029] Fig. 10C and Fig. 10D A schematic diagram showing the correlation between chrominance samples and one or more luma samples.

[0030] Fig.11 An example of using 6 taps in a multiple linear regression (MLR) model according to one or more aspects of the present disclosure is shown.

[0031] Fig.12 Exemplary different filter shapes and / or numbers of taps according to one or more aspects of the present disclosure are shown.

[0032] Fig.13 An example is shown where the FLM may use only the top or left luma and / or chroma samples (extended) for parameter derivation.

[0033] Fig.14 An example is shown where FLM can use different rows for parameter derivation.

[0034] FIG. 15A to FIG. 15D Some examples of 1-tap / 2-tap pre-operation are shown.

[0035] Fig.16 Examples of different shapes / numbers of filter taps are shown.

[0036] Fig.17 Examples of different shapes / numbers of filter taps are shown.

[0037] Fig.18A and Fig.18B Examples of different shapes / numbers of filter taps are shown.

[0038] FIG. 19A to FIG. 19G Examples of different sets of filter taps are shown.

[0039] Fig. 20A workflow of a method for decoding video data according to one or more aspects of the present disclosure is shown.

[0040] Fig.21 A workflow of a method for encoding video data according to one or more aspects of the present disclosure is shown.

[0041] Fig. 22 An exemplary computing system in accordance with one or more aspects of the present disclosure is shown. DETAILED DESCRIPTION

[0042] Reference will now be made in detail to specific embodiments, examples of which are shown in the accompanying drawings. In the following detailed description, a number of non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to one of ordinary skill in the art that various alternatives may be used without departing from the scope of the claims, and that the subject matter may be practiced without these specific details. For example, it will be apparent to one of ordinary skill in the art that the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.

[0043] It should be noted that the terms "first", "second", etc. used in the specification and claims of the present disclosure and the drawings are used to distinguish objects, but not to describe any specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those shown in the drawings or described in the present disclosure.

[0044] The first version of the VVC standard was completed in July 2020, which provides approximately 50% bitrate savings or equivalent perceptual quality compared to the previous generation video codec standard HEVC. Although the VVC standard provides significant codec improvements over its predecessor, there is evidence that superior codec efficiency can be achieved by adopting additional codec tools. Recently, the Joint Video Exploration Team (JVET), in collaboration with ITU-TVCEG and ISO / IEC MPEG, began exploring advanced technologies that can significantly improve codec efficiency over VVC. In April 2021, a software code base named Enhanced Compression Model (ECM) was established for future video codec exploration work. The ECM reference software is based on the VVC Test Model (VTM) developed by JVET for VVC, and further extends and / or improves several existing modules (e.g., intra / inter prediction, transform, loop filter, etc.). In the future, any new codec tools beyond the VVC standard will need to be integrated into the ECM platform and tested using the JVET Common Test Conditions (CTC).

[0045] Similar to all previous video codec standards, ECM is built on a block-based hybrid video codec framework. Figure 1 A block diagram of a general block-based hybrid video coding system is shown. The input video signal is processed block by block, which is called a coding unit (CU). In ECM-1.0, a CU can be up to 128×128 pixels. However, like VVC, a coding tree unit (CTU) is divided into CUs to accommodate different local characteristics based on quadtree / binary / ternary trees. In the multi-type tree structure, a CTU is first divided by a quadtree structure. Then, each quadtree leaf node can be further divided by a binary tree structure and a ternary tree structure. As shown in FIG. Figure 2A , 2B As shown in 2C, 2D and 2E, there are five types of segmentation: quadruple segmentation, vertical binary segmentation, horizontal binary segmentation, vertical extended quadruple segmentation and horizontal extended quadruple segmentation.

[0046] exist Figure 1In the video processing, spatial prediction and / or temporal prediction can be performed. Spatial prediction (or "intra-frame prediction") uses pixels from samples of coded neighboring blocks (which are called reference samples) in the same video picture / slice to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also known as "inter-frame prediction" or "motion compensated prediction") uses reconstructed pixels from coded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal for a given CU is usually signaled by one or more motion vectors (MVs), which indicate the amount and direction of motion between the current CU and its temporal reference. Moreover, if multiple reference pictures are supported, a reference picture index is additionally sent, which is used to identify which reference picture in the reference picture storage the temporal prediction signal comes from. After spatial and / or temporal prediction, the mode decision block in the encoder selects the best prediction mode, for example, based on a rate-distortion optimization method. Then, the prediction block is subtracted from the current video block; and the prediction residual is decorrelated and quantized using a transform. The quantized residual coefficients are inverse quantized and inverse transformed to form a reconstructed residual, which is then added back to the prediction block to form a reconstructed signal for the CU. In addition, loop filtering (such as a deblocking filter, sample adaptive offset (SAO), and adaptive loop filter (ALF)) can be applied to the reconstructed CU before the reconstructed CU is placed in the reference picture storage and used for encoding and decoding future video blocks. In order to form an output video bitstream, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy coding unit to be further compressed and packaged to form a bitstream. It should be noted that the term "block" or "video block" used herein may be a portion of a frame or picture, specifically a rectangular (square or non-square) portion. With reference to, for example, HEVC and VVC, a block or a video block may be or correspond to a coding tree unit (CTU), a CU, a prediction unit (PU), or a transform unit (TU) and / or may be or correspond to a corresponding block, such as a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB)) and / or a sub-block.

[0047] Figure 3The overall block diagram of a block-based video decoder is shown. The video bitstream is first entropy decoded at the entropy decoding unit. The coding mode and prediction information are sent to the spatial prediction unit (if intra-frame coding) or the temporal prediction unit (if inter-frame coding) to form a prediction block. The residual transform coefficients are sent to the inverse quantization unit and the inverse transform unit to reconstruct the residual block. The prediction block and the residual block are then added together. The reconstructed block can also be loop filtered before being stored in the reference picture store. The reconstructed video in the reference picture store is then sent out to drive the display device and is used to predict future video blocks.

[0048] The main focus of the present disclosure is to further enhance the codec efficiency of codec tools applied to cross-component prediction, cross-component linear model (CCLM) in ECM. In the following, some relevant codec tools in ECM will be briefly reviewed. Thereafter, some defects in the existing design of CCLM are discussed. Finally, a solution for improving the existing CCLM prediction design is provided.

[0049] Cross-component linear model prediction

[0050] To reduce cross-component redundancy, the Cross-Component Linear Model (CCLM) prediction mode is used in VVC, for which the chrominance samples are predicted based on the reconstructed luma samples of the same CU using the following linear model:

[0051] pred C (i,j)=α·rec L ′(i,j)+ β (1)

[0052] Among them, pred C (i, j) represents the predicted chroma sample in the CU, and rec L '(i,j) represents the reconstructed brightness sample point rec L The downsampled reconstructed luma samples of the same CU obtained by performing downsampling on (i, j). The above α and β are linear model parameters derived from up to four neighboring chroma samples and their corresponding downsampled luma samples, which can be called neighboring luma-chroma sample pairs. Assuming that the current chroma block has a size of W×H, W' and H' are obtained as follows:

[0053] - When LM mode is applied, W'=W, H'=H;

[0054] - When LM-A mode is applied, W'=W+H;

[0055] - When LM-L mode is applied, H'=H+W;

[0056] Among them, in LM mode, the upper and left samples of the CU are used together to calculate the linear model coefficients; in LM_A mode, only the upper samples of the CU are used to calculate the linear model coefficients; and in LM_L mode, only the left samples of the CU are used to calculate the linear model coefficients.

[0057] If the positions of the upper neighboring samples of the chroma block are represented as S[0,-1]...S[W'-1,-1], and the positions of the left neighboring samples of the chroma block are represented as S[-1,0]...S[-1,H'-1], then the positions of the four neighboring chroma samples are selected as follows:

[0058] - When LM mode is applied and the upper and left neighboring samples are available, select S[W' / 4,-1], S[3*W' / 4,-1], S[-1,H' / 4], S[-1,3*H' / 4] as the positions of the four neighboring chroma samples;

[0059] -When LM-A mode is applied or only upper neighbor samples are available, select S[W' / 8,-1], S[3*W' / 8,-1], S[5*W' / 8,

[0060] -1], S[7*W' / 8,-1] as the positions of the four adjacent chrominance samples;

[0061] - When LM-L mode is applied or only left neighboring samples are available, select S[-1,H' / 8], S[-1,3*H' / 8], S[-1,5*H' / 8], S[-1,7*H' / 8] as the positions of the four neighboring chroma samples.

[0062] Four adjacent brightness samples corresponding to the selected position are obtained by downsampling operation, and the obtained four adjacent brightness samples are compared four times to find two larger values ​​x 0 A and x 1 A and two smaller values ​​x 0 B and x 1 B The chrominance sample values ​​corresponding to the two larger values ​​and the two smaller values ​​are represented as y and 0 A ,y 1 A ,y 0 B and 1 B Then, export X as follows a , X b , Y a and Y b :

[0063] X a =(x 0 A +x 1 A +1)>>1;

[0064] X b =(x 0 B +x 1 B +1)>>1;

[0065] Y a =(y 0 A +y 1 A +1)>>1;

[0066] Y b =(y 0 B +y 1 B +1)>>1 (2)

[0068] Finally, the linear model parameters α and β are obtained according to the following equations.

[0069]

[0070] β=Y b -α·X b (4)

[0071] Figure 4 Examples of the positions of left and top samples and samples of the current block involved in the CCLM mode are shown, including the positions of left and top samples of an N×N chroma block in a CU and the positions of left and top samples of a 2N×2N luminance block in a CU.

[0072] The division operation used to calculate the parameter α is implemented using a lookup table. In order to reduce the memory required to store the table, the diff value (the difference between the maximum and minimum values) and the parameter α are represented by exponential notation. For example, diff is approximated using a 4-bit significant part and an exponent. Therefore, the table for 1 / diff is reduced to 16 elements of 16 significant values, as shown below:

[0073] DivTable [ ] = { 0, 7, 6, 5, 5, 4, 4, 3, 3, 2, 2, 1, 1, 1, 1, 0} (5)

[0074] This will have the benefit of reducing both the complexity of the calculations and the size of the memory required to store the required tables.

[0075] In addition to the upper template and the left template being used together to calculate the linear model coefficients, they can also be used alternatively in the other 2 LM modes (referred to as LM_A and LM_L modes).

[0076] In LM_T mode, only the upper template is used to calculate the linear model coefficients. To get more samples, the upper template is expanded to (W+H) samples. In LM_L mode, only the left template is used to calculate the linear model coefficients. To get more samples, the left template is expanded to (H+W) samples.

[0077] In LM_LT mode, the left template and the upper template are used to calculate the linear model coefficients.

[0078] To match the chroma sample positions of a 4:2:0 video sequence, two types of downsampling filters are applied to the luma samples to achieve a 2:1 downsampling ratio in both horizontal and vertical directions. The choice of downsampling filter is specified by the SPS level flag. The two downsampling filters are as follows, corresponding to "type 0" and "type 2" content, respectively.

[0079]

[0080] Note that when the upper reference line is located at a CTU boundary, only one luma line (common line buffer in intra prediction) is used to make the downsampled luma samples.

[0081] This parameter calculation is performed as part of the decoding process, not just as an encoder retrieval operation. As a result, no syntax is used to convey the α and β values ​​to the decoder.

[0082] For chroma intra mode coding, a total of 8 intra modes are allowed for chroma intra mode coding. These modes include five traditional intra modes and three cross-component linear model modes (CCLM, LM_A and LM_L). The chroma mode signaling and derivation process are shown in Table 1. Chroma mode coding depends directly on the intra prediction mode of the corresponding luminance block. Since separate block partitioning structures for luminance components and chrominance components are enabled in the I slice, one chroma block can correspond to multiple luminance blocks. Therefore, for the chroma DM mode, the intra prediction mode of the corresponding luminance block covering the center position of the current chroma block is directly inherited.

[0083] Table 1 - Chroma prediction modes derived from luma mode when CCLM is enabled

[0084]

[0085] Regardless of the value of sps_cclm_enabled_flag, a single binarization table is used, as shown in Table 2.

[0086] Table 2 - Unified binarization table for chroma prediction mode

[0087]

[0088]

[0089] In Table 2, the first binary bit indicates whether it is normal mode (0) or LM mode (1). If it is LM mode, the next binary bit indicates whether it is LM_CHROMA (0). If it is not LM_CHROMA, the next 1 binary bit indicates whether it is LM_L (0) or LM_A (1). For this case, when sps_cclm_enabled_flag is 0, the first binary bit of the corresponding intra_chroma_pred_mode binarization table can be discarded before entropy coding. Or, in other words, the first binary bit is inferred to be 0 and is therefore not coded. This single binary table is used for both cases where sps_cclm_enabled_flag is equal to 0 and 1. The first two binary bits in Table 2 perform context coding and decoding using their own context model, and the remaining binary bits perform bypass coding and decoding.

[0090] In addition, to reduce the luma-chroma latency in dual trees, when the 64×64 luma coding tree node is split using NotSplit (and ISP is not used for 64×64 CU) or QT, the chroma CU in the 32×32 / 32×16 chroma coding tree node is allowed to use CCLM as follows:

[0091] - If a 32×32 chroma node is not split or is split by QT, all chroma CUs in the 32×32 node can use CCLM.

[0092] - If a 32x32 chroma node utilizes horizontal BT splitting, and the 32x16 child nodes are not split or use vertical BT splitting, then all chroma CUs in the 32x16 chroma node can use CCLM.

[0093] Under all other luma and chroma coding tree partition conditions, CCLM is not allowed for chroma CUs.

[0094] During the ECM development, the simplified derivation (min-max approximation) of α and β was removed. Instead, a linear least squares solution was performed between the causal reconstructed data of the downsampled luma samples and the causal chroma samples to derive the model parameters α and β.

[0095]

[0096] Among them, Rec C (i) and Rec' L (i) indicates the reconstructed chroma samples and the down-sampled reconstructed luma samples around the target block, and I indicates the total number of samples of neighboring data.

[0097] The LM_A and LM_L models are also called multidirectional linear models (MDLM). Figure 5A An example of the operation of the MDLM is shown when the block content cannot be predicted from the L-shaped reconstructed region. Figure 5B MDLM_L is shown using only the left reconstructed samples to derive CCLM parameters. Figure 5C MDLM_T is shown using only the top reconstructed samples to derive CCLM parameters.

[0098] An integerization of the least mean square (LMS) discussed above (see equations (8)-(9)) has been proposed as an improvement to the CCLM. The initial integerization design of the LMS CCLM was first proposed in JCTVC-C206. The method was then improved through a series of simplifications to form the ECM LMS version, including: JCTVC-F0233 / I0178, which converts the α precision n α from 13 to 7; JCTVC-I0151, which reduced the maximum multiplier bit width; and JCTVC-H0490 / I0166, which reduced the divide LUT entries from 64 to 32.

[0099] As discussed in equation (1), the integerized design models the correlation of the luma and chroma signals using a linear relationship. The chroma values ​​are predicted from the reconstructed luma values ​​of the co-located blocks.

[0100] In YUV420 sampling, the luma component and the chroma component have different sampling ratios. The sampling ratio of the chroma component is half of the sampling ratio of the luma component, and has a phase difference of 0.5 pixels in the vertical direction. The reconstructed luma needs downsampling in the vertical direction and subsampling in the horizontal direction to match the size of the chroma signal. For example, downsampling can be achieved by:

[0101] Rec L ′(i,j)=(rec L (2i,2j)+rec L (2i,2j+1))>>1 (10)

[0102] In equation (8), in order to maintain high data accuracy, floating-point operations are required when calculating the linear model parameter α. Also, when α is represented by a floating-point value, floating-point multiplication is involved in equation (1). In this section, an integer implementation of the algorithm is designed. Specifically, the fractional part of the parameter α is calculated using n α The parameter α value is obtained by amplifying and rounding the integer value α' and a'=a×(1<<n α ) is expressed. Then the linear model of equation (1) is changed to:

[0103] pred C [x,y]=(α'·Rec L '[x,y]>>n α )+β' (11)

[0104] Where β' is the rounded value of the floating point β, and α' can be calculated as follows.

[0105]

[0106] It is proposed to replace the division operation of equation (12) by table lookup and multiplication. First, A2 is reduced to reduce the table size. A1 is also reduced to avoid product overflow. Then, in A2, only the The most significant bit of the value is defined, and all other bits are set to zero. The approximate value A2' can be calculated as:

[0107]

[0108] where [...] indicates rounding operation, and It can be calculated as:

[0109]

[0110] Here, bdepth(A2) represents the bit depth of value A2.

[0111] Perform the same operation on A1 as follows:

[0112]

[0113] Considering the quantized representations of A1 and A2, equation (12) can be rewritten as follows.

[0114]

[0115]

[0116] in, Indicates that The length of the lookup table is used to avoid division.

[0117] In the simulation, the constant parameters are set as:

[0118] ·n α Equal to 13, this value is a compromise between data accuracy and computational cost.

[0119] · Equal to 6, so that the lookup table size is 64, when bdepth(A2)<6 (for example, A2<32), the table size can be further reduced to 32 by enlarging A2.

[0120] ·n table Equal to 15, produces a 16-bit data representation of the table element.

[0121] · Set to 15 to avoid product overflow and maintain 16-bit multiplication.

[0122] Finally, α' is clipped to [-2 -15 ,2 15 -1] to preserve the 16-bit multiplication in equation (11). With this clipping, when n α When it is equal to 13, the actual a value is limited to [-4,4), which helps prevent error amplification.

[0123] By calculating the parameter α', the parameter β' is calculated as follows:

[0124]

[0125] The division in the above equation can be simply replaced by a shift since the value I is a power of 2.

[0126] Similar to the discussion above about equation (1), in HM6.0, an intra prediction mode called LM is applied to predict the chroma PU based on a linear model using the reconstruction of the co-located luma PU. The parameters of the linear model consist of the slope (a>>k) and the y-intercept (b), which are derived from the neighboring luma and chroma pixels using the least mean square solution. The values ​​of the prediction samples predSamples[x,y], where x,y=0...nS-1, where nS specifies the block size of the current chroma PU, are derived as follows:

[0127] predSamples[x,y]=Clip1 C (((p Y '[x,y]*a)>>k)+b), where x,y=0..nS-1 (17)

[0128] Among them, P Y'[x,y] is the reconstructed pixel from the corresponding luminance component. When the coordinates x and y are equal to or greater than 0, P Y ' is the reconstructed pixel from the same luma PU. When x or y is less than 0, P Y ' is the reconstructed neighboring pixel of the same-position brightness PU.

[0129] Some intermediate variables L, C, LL, LC, k2 and k3 in the derivation process are derived as follows:

[0130]

[0131]

[0132] k2 = Log2( (2*nS) >> k3 ) (18-5)

[0133] k3 = Max(0, BitDepth C + Log2(nS) - 14) (18-6)

[0134] Therefore, the variables a, b, and k can be derived as:

[0135] a1= (LC << k2) – L*C (19-1)

[0136] a2= (LL << k2) – L*L (19-2)

[0137] k1= Max(0,Log2(abs(a2))-5)–Max(0,Log2(abs(a1))-14)+2 (19-3)

[0138] a1s = a1 >> Max(0, Log2( abs( a1 ) ) - 14 ) (19-4)

[0139] a2s = abs( a2 >> Max(0, Log2( abs( a2 ) ) - 5 ) ) (19-5)

[0140] a3 = a2s < 1? 0 :Clip3( -2 15 , 2 15 -1, a1s*lmDiv + ( 1 << ( k1 - 1 ) )>> k1 ) (19-6)

[0141] a = a3 >> Max( 0, Log2( abs( a3 ) ) - 6 ) (19-7)

[0142] k = 13 – Max( 0, Log2( abs( a ) ) - 6 ) (19-8)

[0143] b = ( L – ( ( a*C ) >> k1 ) + ( 1 << ( k2 - 1 ) ) ) >> k2, (19-9)

[0144] Here, lmDiv is specified in a 63-item lookup table (i.e., Table 3), which is generated online as follows:

[0145] lmDiv(a2s)=( (1 << 15) + a2s / 2 ) / a2s . (20)

[0146] Table 3 - lmDiv Specifications

[0147] a2s 1 2 3 4 5 6 7 8 9 10 11 12 13 lmDiv 32768 16384 10923 8192 6554 5461 4681 4096 3641 3277 2979 2731 2521 a2s 14 15 16 17 18 19 20 21 22 23 24 25 26 lmDiv 2341 2185 2048 1928 1820 1725 1638 1560 1489 1425 1365 1311 1260 a2s 27 28 29 30 31 32 33 34 35 36 37 38 39 lmDiv 1214 1170 1130 1092 1057 1024 993 964 936 910 886 862 840 a2s 40 41 42 43 44 45 46 47 48 49 50 51 52 lmDiv 819 799 780 762 745 728 712 697 683 669 655 643 630 a2s 53 54 55 56 57 58 59 60 61 62 63 64 lmDiv 618 607 596 585 575 565 555 546 537 529 520 512

[0148] In equation (19-6), a1s is a 16-bit signed integer and lmDiv is a 16-bit unsigned integer. Therefore, a 16-bit multiplier and 16-bit storage are required. It is proposed to reduce the bit depth of the multiplier to the internal bit depth and reduce the size of the lookup table, as detailed below.

[0149] The bit depth of a1s is reduced to the internal bit depth by changing equation (19-4) to the following equation:

[0150] a1s = a1 >> Max(0, Log2( abs( a1 ) ) – (BitDepth C – 2)) . (21)

[0151] The value of lmDiv with internal bit depth is implemented using the following equation (22) and stored in a lookup table:

[0152] lmDiv(a2s)=( (1 << (BitDepth C -1)) + a2s / 2 ) / a2s. (22)

[0153] Table 4 shows an example of an internal bit depth of 10.

[0154] Table 4 - Specification of lmDiv with internal bit depth equal to 10

[0155]

[0156]

[0157] Equation (19-3) and equation (19-8) are also modified as follows:

[0158] k1=Max(0,Log2(abs(a2))-5)–Max(0,Log2(abs(a1))–(BitDepth C –2)), and (23-1)

[0159] k = BitDepth C –1–Max(0,Log2(abs(a))-6) (23-2)

[0160] It is also proposed to reduce the number of entries from 63 to 32, and the bits per entry from 16 to 10, as shown in Table 5. By doing so, a memory saving of nearly 70% can be achieved. The corresponding changes to equation (19-6), equation (20), and equation (19-8) are as follows:

[0161] a3=a2s<32?0:Clip3(-2 15 ,2 15 -1,a1s*lmDiv+(1<<(k1-1))>>k1) (24-1)

[0162] lmDiv(a2s)=((1<<(BitDepth C +4))+a2s / 2) / a2s (24-2)

[0163] k = BitDepth C +4–Max(0,Log2(abs(a))-6). (24-3)

[0164] Table 5 - Specification of lmDiv with internal bit depth equal to 10

[0165] a2s 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 lmDiv 512 496 482 468 455 443 431 420 410 400 390 381 372 364 356 349 a2s 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 lmDiv 341 334 328 321 315 309 303 298 293 287 282 278 273 269 264 260

[0166] Multi-model linear model prediction

[0167] In ECM-1.0, a multi-model LM (MMLM) prediction mode is proposed, for which chrominance samples are predicted based on the reconstructed luma samples of the same CU using two linear models, as shown below:

[0168]

[0169] Among them, pred C (i, j) represents the predicted chroma sample in the CU, and rec L′(i,j) represents the downsampled reconstructed luma sample of the same CU. Threshold is calculated as the average of the neighboring reconstructed luma samples. Figure 6 An example of classifying neighboring samples into two groups based on the value Threshold is shown. For each group, the minimum brightness sample A(X) from two samples (which is the minimum brightness sample A(X) within the group) is selected. A , Y A ) and the maximum brightness sample B(X B , Y B )) to derive the parameter α from the linear relationship between the brightness and chromaticity values i and β i (where i is equal to 1 and 2 respectively). Here, X A , Y A is the x-coordinate (i.e., luminance value) and y-coordinate (i.e., chrominance value) value of sample point A, and X B , Y B are the x-coordinate and y-coordinate values ​​of sample point B. The linear model parameters α and β are obtained according to the following equations.

[0170]

[0171] Such a method is also called the min-max method. The division in the above equation can be avoided and replaced by multiplication and shift.

[0172] For coding blocks with a square shape, the above two equations are directly applied. For non-square coding blocks, the neighboring samples of the longer boundary are first subsampled to have the same number of samples as the shorter boundary.

[0173] In addition to the scenario in which the upper template and the left template are used together to calculate the linear model coefficients, these two templates can also be used alternatively in the other two MMLM modes (referred to as MMLM_A mode and MMLM_L mode).

[0174] In MMLM_A mode, only the pixel samples in the upper template are used to calculate the linear model coefficients. In order to get more samples, the upper template is expanded to a size of (W+W). In MMLM_L mode, only the pixel samples in the left template are used to calculate the linear model coefficients. In order to get more samples, the left template is expanded to a size of (H+H).

[0175] Note that when the upper reference line is located at a CTU boundary, only one luma line (stored in the line buffer for intra prediction) is used to make the downsampled luma samples.

[0176] For chroma intra mode coding, a total of 11 intra modes are allowed for chroma intra mode coding. These modes include five traditional intra modes and six cross-component linear model modes (CCLM, LM_A, LM_L, MMLM, MMLM_A and MMLM_L). The chroma mode signaling and derivation process are shown in Table 6. Chroma mode coding depends directly on the intra prediction mode of the corresponding luminance block. Since separate block partitioning structures for luminance components and chrominance components are enabled in the I slice, one chroma block can correspond to multiple luminance blocks. Therefore, for the chroma DM mode, the intra prediction mode of the corresponding luminance block covering the center position of the current chroma block is directly inherited.

[0177] Table 6 - Chroma prediction modes derived from luma mode when MMLM_ is enabled

[0178]

[0179] The MMLM mode and the LM mode can also be used together in an adaptive manner. For MMLM, the two linear models are as follows:

[0180]

[0181] Among them, pred C (i, j) represents the predicted chroma sample in the CU, and rec L ′(i,j) represents the downsampled reconstructed luma sample of the same CU. Threshold can be simply determined based on the average value of luma and chroma together with their minimum and maximum values. Figure 7 An example of classifying neighboring samples into two groups based on the inflection point T indicated by the arrow is shown. The linear model parameters α1 and β1 are obtained based on the minimum brightness sample A(X A , Y A ) and Threshold(X T , Y T )) is derived from the linear relationship between the brightness and chromaticity values. The linear model parameters α2 and β2 are derived from the two samples (which are the maximum brightness sample B (X B , Y B ) and Threshold(X T , Y T )) is derived from the linear relationship between the brightness and chromaticity values. Here, X A , Y A is the x-coordinate (i.e., luminance value) and y-coordinate (i.e., chrominance value) value of sample point A, and X B , Y B is the x-coordinate and y-coordinate value of sample point B. The linear model parameter α of each group is obtained according to the following equation iand β i , where i is equal to 1 and 2 respectively.

[0182]

[0183] For coding blocks with a square shape, the above equation is directly applied. For non-square coding blocks, the neighboring samples of the longer boundary are first sub-sampled to have the same number of samples as the shorter boundary.

[0184] In addition to the scenario in which the upper template and the left template are used together to determine the linear model coefficients, these two templates can also be used alternatively in the other two MMLM modes (referred to as MMLM_A mode and MMLM_L mode, respectively).

[0185] In MMLM_A mode, only the pixel samples in the upper template are used to calculate the linear model coefficients. In order to get more samples, the upper template is expanded to a size of (W+W). In MMLM_L mode, only the pixel samples in the left template are used to calculate the linear model coefficients. In order to get more samples, the left template is expanded to a size of (H+H).

[0186] Note that when the upper reference line is located at a CTU boundary, only one luma line (stored in the line buffer for intra prediction) is used to make the downsampled luma samples.

[0187] For chroma intra mode coding, there is a conditional check to select LM mode (CCLM, LM_A and LM_L) or multi-model LM mode (MMLM, MMLM_A and MMLM_L). The conditional check is as follows:

[0188]

[0189] Among them, BlkSizeThres LM Indicates the minimum block size for LM mode, and BlkSizeThres MM Indicates the minimum block size of the MMLM mode. The symbol d represents a predetermined threshold. In one example, d can take a value of 0. In another example, d can take a value of 8.

[0190] For chroma intra mode coding, a total of 8 intra modes are allowed for chroma intra mode coding. These modes include five traditional intra modes and three cross-component linear model modes. The chroma mode signaling and derivation process are shown in Table 1. It is worth noting that for a given CU, if it is encoded in the linear model mode, it is determined whether it is a conventional single model LM mode or an MMLM mode based on the above conditional check. Unlike the case shown in Table 6, there is no separate MMLM mode that needs to be signaled. Chroma mode coding depends directly on the intra prediction mode of the corresponding luminance block. Since separate block partitioning structures for luminance components and chrominance components are enabled in the I slice, one chroma block can correspond to multiple luminance blocks. Therefore, for the chroma DM mode, the intra prediction mode of the corresponding luminance block covering the center position of the current chroma block is directly inherited.

[0191] During the ECM development, scaling (slope) adjustments to the CCLM were proposed as a further improvement, for example as described in JVET-Y0055 / Z0049.

[0192] As discussed above, CCLM uses a 2-parameter model to map luma values ​​to chroma values. The scaling parameter "a" and the bias parameter "b" define the mapping as follows:

[0193] chromaVal = a * lumaVal + b (30)

[0194] It is proposed to signal the adjustment "u" to the scaling parameters to update the model to the following form:

[0195] chromaVal = a' * lumaVal + b' (31)

[0196] Where, a'=a+u, and b'=bu*y r .

[0197] With this choice, the mapping function will be centered around the value y with brightness r The point is tilted or rotated. It is proposed to use the average value of the reference brightness samples used in model creation as y r , in order to provide meaningful modifications to the model. Figures 8A to 8B shows the effect of scaling adjustment parameter "u", where Fig. 8A shows a model created without the scaling tuning parameter "u", and Figure 8B A model created with a scaling adjustment parameter "u" is shown.

[0198] In one example, the scaling adjustment parameter is provided as an integer between -4 and 4 (inclusive) and signaled in the bitstream. The unit of the scaling adjustment parameter is 1 / 8 of the chroma sample value per luma sample value (for 10-bit content).

[0199] In one example, adjustments are made to CCLM models ("LM_CHROMA_IDX" and "MMLM_CHROMA_IDX") that use reference samples above and to the left of the block, but not to "one-sided" mode. This choice is based on a tradeoff between codec efficiency and complexity.

[0200] When applying scaling adjustments to a multi-mode CCLM model, both models may be adjusted and thus for a single chroma block at most two scaling updates are signaled.

[0201] To enable scaling at the encoder, the encoder can perform a SATD-based search for the best value of the scaling update for Cr and a similar SATD-based search for Cb. If either result is a non-zero scaling adjustment parameter, the combined scaling adjustment pair (SATD-based update for Cr, SATD-based update for Cb) will be included in the list of RD checks for the TU.

[0202] Fusion of Chroma Intra Prediction Modes

[0203] During the development of ECM, JVET-Y0092 / Z0051 proposed the fusion of chroma intra mode.

[0204] The intra prediction modes enabled for chroma components in ECM-4.0 are six cross-component linear model (LM) modes (including CCLM_LT, CCLM_L, CCLM_T, MMLM_LT, MMLM_L, and MMLM_T modes), direct mode (DM), and four default chroma intra prediction modes. The four default modes are given by the list {0, 50, 18, 1}, and if the DM mode already belongs to the list, the mode in the list will be replaced by mode 66.

[0205] The decoder-side intra mode derivation (DIMD) method for luma intra prediction is included in ECM-4.0. First, the horizontal and vertical gradients are calculated for each reconstructed luma sample of the L-shaped template of the second adjacent row and column of the current block to construct a gradient histogram (HoG). Then, the two intra prediction modes with the largest and second largest histogram magnitude values ​​are mixed with the planar mode to generate the final predictor for the current luma block.

[0206] In order to improve the coding efficiency of chroma intra prediction, two methods are proposed, including the chroma intra prediction mode derived at the decoder side (DIMD chroma) and the fusion of non-LM mode and MMLM_LT mode.

[0207] In the first embodiment, a DIMD chroma mode is proposed. The proposed DIMD chroma mode uses a DIMD derivation method to derive the chroma intra prediction mode of the current block based on the co-located reconstructed luma samples. Specifically, the horizontal gradient and the vertical gradient are calculated for each co-located reconstructed luma sample of the current chroma block to construct HoG, such as Figure 8C Then, the intra prediction mode with the largest histogram magnitude value is used to perform chroma intra prediction of the current chroma block.

[0208] When the intra prediction mode derived from the DIMD chroma mode is the same as the intra prediction mode derived from the DM mode, the intra prediction mode having the second largest histogram magnitude value is used as the DIMD chroma model.

[0209] As shown in Table 7, a CU level flag is signaled to indicate whether the proposed DIMD chroma mode is applied.

[0210] Table 7. Binarization process of intra_chroma_pred_mode in the proposed method

[0211] intra_chroma_pred_mode Binary bit string Chroma Intra Mode 0 1100 List[0] 1 1101 List[1] 2 1110 List [2] 3 1111 List [3] 4 10 DIMD Chroma 5 0 DM

[0212] In the second embodiment, the fusion of chroma intra prediction modes is proposed, wherein the DM mode and four default modes can be fused with the MMLM_LT mode as shown below:

[0213] pred=(w0*pred0+w1*pred1+(1<<(shift-1)))>>shift

[0214] Where pred0 is the predictor obtained by applying the non-LM mode, pred1 is the predictor obtained by applying the MMLM_LT mode, and pred is the final predictor of the current chroma block. The two weights (w0 and w1) are determined by the intra prediction mode of the adjacent chroma blocks, and shift is set to be equal to 2. Specifically, when both the upper and left adjacent blocks are encoded and decoded using the LM mode, {w0, w1} = {1, 3}; when both the upper and left adjacent blocks are encoded and decoded using the non-LM mode, {w0, w1} = {3, 1}; otherwise, {w0, w1} = {2, 2}.

[0215] For syntax design, if non-LM mode is selected, a flag is signaled to indicate whether fusion is applied, and the proposed fusion is only applied to I slices.

[0216] In the third embodiment, the DIMD chroma mode is combined with the fusion of the chroma intra prediction mode. Specifically, the DIMD chroma mode described in the first embodiment is applied, and for I slices, the DM mode, the four default modes, and the DIMD chroma mode can be fused with the MMLM_LT mode using the weights described in the second embodiment, while for non-I slices, only the DIMD chroma mode can be fused with the MMLM_LT mode using equal weights.

[0217] In a fourth embodiment, the DIMD chroma mode with reduced processing is combined with a fusion of chroma intra prediction modes. Specifically, the DIMD chroma mode with reduced processing derives an intra mode based on the adjacent reconstructed Y, Cb and Cr samples in the second adjacent row and column, such as Fig.8D The other parts are the same as those of the third embodiment.

[0218] In one embodiment, when DIMD is applied, two intra modes are derived from the reconstructed neighboring samples, and these two predictors are combined with a planar mode predictor with weights derived from gradients, as described in JVET-00449. The division operations in the weight derivation are performed using the same lookup table (LUT) based integration scheme as used by CCLM. For example, the division operations in the orientation calculation

[0219] Orient=G y / G x

[0220] is calculated using the following LUT-based scheme:

[0221] x = Floor(Log2(Gx))

[0222] normDiff=((Gx<<4)>>x)&15

[0223] x+=(3+(normDiff!=0)?1:0)

[0224] Orient=(Gy*(DivSigTable[normDiff]|8)+(1<<(x-1)))>>x

[0225] in,

[0226] DivSigTable

[16] ={0,7,6,5,5,4,4,3,3,2,2,1,1,1,1,0}.

[0227] The derived intra modes are included into the main list of intra most probable modes (MPMs), so the DIMD process is performed before constructing the MPM list. The main derived intra modes of a DIMD block are stored with the block and used for MPM list construction of neighboring blocks.

[0228] Figures 8E to 8H The steps of intra mode derivation at the decoder side are shown, where the intra prediction direction is estimated without intra mode signaling. Fig. 8E The first step shown involves estimating the gradient at each sample point (for example Fig. 8E as shown in light grey). Figure 8F The second step shown in consists of mapping the gradient value to the closest prediction direction within [2,66]. Figure 8G The third step shown in FIG. 1 includes selecting two prediction directions, where for each prediction direction, all absolute gradients Gx and Gy of neighboring pixels in that direction are summed, and the top two directions are selected. Figure 8H The fourth step shown includes enabling weighted intra prediction using the selected direction.

[0229] Multiple reference line (MRL) intra prediction uses more reference lines for intra prediction. Fig. 9 In FIG. 4 , an example of 4 reference lines is depicted, where the samples of segments A and F are not taken from the reconstructed neighboring samples, but are filled with the closest samples from segments B and E, respectively. HEVC intra-picture prediction uses the nearest reference line (i.e., reference line 0). In MRL, 2 additional lines are used (reference line 1 and reference line 3).

[0230] The index of the selected reference row (mrl_idx) is signaled and used to generate the intra predictor. For reference row idx greater than 0, only include additional reference row modes in the MPM list and only signal the mpm index for which there are no remaining modes. The reference row index is signaled before the intra prediction mode and in case a non-zero reference row index is signaled, the planar mode is excluded from the intra prediction mode.

[0231] For blocks in the first row inside a CTU, MRL is disabled to prevent the use of extended reference samples outside the current CTU row. In addition, PDPC is also disabled when additional rows are used. For MRL mode, the derivation of the DC value in the DC intra prediction mode for non-zero reference row index is consistent with the derivation of reference row index 0. MRL requires 3 adjacent luminance reference rows stored in the CTU to generate predictions. The Cross Component Linear Model (CCLM) tool also requires 3 adjacent luminance reference rows for its downsampling filter. The definition of MRL using the same 3 rows is consistent with CCLM to reduce the storage requirements of the decoder.

[0232] During the development of ECM, the Convolutional Cross-Component Model (CCCM) for Chroma Intra Mode was proposed.

[0233] It is proposed to apply a convolutional cross-component model (CCCM) to predict chroma samples from reconstructed luma samples, similar in spirit to the current CCLM mode. As with CCLM, when chroma subsampling is used, the reconstructed luma samples are downsampled to match the lower resolution chroma grid.

[0234] Also, similar to CCLM, there is an option to use a single model or a multi-model variant of CCCM. The multi-model variant uses two models, one derived for samples above the average luminance reference value, and another derived for the remaining samples (following the spirit of the CCLM design). For PUs with at least 128 available reference samples, the multi-model CCCM mode can be selected.

[0235] The proposed convolutional 7-tap filter consists of a 5-tap plus sign-shaped spatial component, a nonlinear term, and a bias term. The input to the spatial 5-tap component of the filter consists of the center (C) luma sample (which is co-located with the chroma sample to be predicted) and its upper / north (N), lower / south (S), left / west (W), and right / east (E) neighboring samples, as Fig. 10A shown.

[0236] The nonlinear term P is expressed as a power of two of the center luma sample C and scaled to the sample value range of the content:

[0237] P=(C*C+midVal)>>bitDepth

[0238] That is, for 10 bits of content, it is calculated as:

[0239] P=(C*C+512)>>10

[0240] The bias term B represents a scalar offset between the input and output (similar to the offset term in CCLM) and is set to the intermediate chrominance value (512 for 10-bit content).

[0241] The output of the filter is calculated as the filter coefficient c i Convolution with the input value and clipped to the range of valid chroma samples:

[0242] predChromaVal=c0C+c1N+c2S+c3E+c4W+c5P+c6B

[0243] The filter coefficients c are calculated by minimizing the MSE between the predicted chrominance samples and the reconstructed chrominance samples in the reference region i . Fig. 10B A reference region is shown consisting of 6 rows of chroma samples above and to the left of the PU. The reference region extends one PU width to the right and one PU height below the PU boundary. The region is adjusted to include only available samples. The extension of the region shown in blue is needed to support the "side sampling" of the plus-shaped spatial filter and to fill in when in unavailable areas.

[0244] MSE minimization is performed by computing the autocorrelation matrix for the luma input and the cross-correlation vector between the luma input and the chroma output. The autocorrelation matrix is ​​LDL-decomposed and back-substitution is used to compute the final filter coefficients. The process roughly follows the calculation of the ALF filter coefficients in ECM, however the LDL decomposition is chosen instead of the Cholesky decomposition to avoid the use of square root operations. The proposed method uses only integer arithmetic.

[0245] The use of this mode is signaled using the PU level flag for the CABAC codec. A new CABAC context is included to support this. In terms of signaling, CCCM is treated as a sub-mode of CCLM. That is, the CCCM flag is signaled only when the intra prediction mode is LM_CHROMA_IDX (to enable single-mode CCCM) or MMLM_CHROMA_IDX (to enable multi-model CCCM).

[0246] The encoder performs two new RD checks in the chroma prediction mode loop, one for checking single-model CCCM mode and one for checking multi-model CCCM mode.

[0247] In the existing CCLM or MMLM design, adjacent reconstructed luma-chroma sample pairs are classified into one or more sample groups based on the value Threshold, which only considers the luma DC value. That is, the luma-chroma sample pairs are classified by considering only the intensity of the luma sample. However, the luma component usually retains rich textures, and the current luma sample may be highly correlated with the adjacent luma samples. Such inter-sample correlation (AC correlation) may be beneficial to the classification of luma-chroma sample pairs and may bring additional coding efficiency.

[0248] like Fig. 10C As shown in , CCLM assumes that a given chroma sample is only related to the corresponding luma sample (L0.5, which can be taken as the fractional luma sample position), and uses a simple linear regression (SLR) estimate using ordinary least squares (OLS) to predict the given chroma sample. Fig. 10DAs shown in , in some video contents, one chrominance sample can be simultaneously correlated with multiple luminance prediction samples (AC or DC correlation), so the multiple linear regression (MLR) model can further improve the accuracy.

[0249] Although the CCCM mode can enhance the efficiency of intra-frame prediction, its performance still has room for further improvement. At the same time, some parts of the existing CCCM mode also need to be simplified to achieve efficient codec hardware implementation, or improved to have better codec efficiency. In addition, the trade-off between its implementation complexity and its codec efficiency benefits needs to be further improved.

[0250] Marginal Linear Model (ELM)

[0251] In order to improve the coding efficiency of the luma component and the chroma component, a classifier that considers luma edge or AC information is introduced, which is opposite to the above implementation in which only the luma DC value is considered. In addition to the existing band classification MMLM, the present disclosure also provides an exemplary classifier. The process of generating a linear prediction model for different sample groups may be similar to CCLM or MMLM (e.g., via least squares or a simplified minimum-maximum method, etc.), but different metrics are used for classification. Different classifiers can be used to classify adjacent luma samples (e.g., adjacent luma-chroma sample pairs) and / or luma samples corresponding to the chroma samples to be predicted. Luma samples corresponding to chroma samples can be obtained by downsampling operations to match the positions of corresponding chroma samples of a 4:2:0 video sequence. For example, luma samples corresponding to chroma samples can be obtained by performing downsampling operations on more than one (e.g., 4) reconstructed luma samples corresponding to chroma samples (e.g., located around chroma samples). Alternatively, for example, in the case of a 4:4:4 video sequence, the luma samples may be obtained directly from the reconstructed luma samples. Alternatively, the luma samples may be obtained from corresponding luma samples of the reconstructed luma samples located at corresponding co-located positions of the corresponding chroma samples. For example, the luma sample to be classified may be obtained from one of the four reconstructed luma samples corresponding to the chroma sample, the chroma sample being located at the upper left position of the four reconstructed luma samples, which position may be regarded as the co-located position of the chroma samples.

[0252] The first classifier can classify the luma samples according to the luma sample edge strength. For example, a direction (e.g., 0 degrees, 45 degrees, or 90 degrees, etc.) can be selected to calculate the edge strength. The direction can be formed by the current sample and the neighboring samples along the direction (e.g., the neighboring samples located at 45 degrees to the upper right of the current sample). The edge strength can be calculated by subtracting the neighboring samples from the current sample. The edge strength can be quantized into one of the M segments by M-1 thresholds, and the first classifier can use M classes to classify the current sample. Alternatively or additionally, N directions can be formed by the current sample and N neighboring samples along N directions. N edge strengths can be calculated by subtracting N neighboring samples from the current sample, respectively. Similarly, if each of the N edge strengths can be quantized into one of the M segments by M-1 thresholds, the first classifier can use MN classes to classify the current sample.

[0253] The second classifier can be used to classify based on local patterns. For example, the current luma sample Y0 can be compared with its N neighboring luma samples Yi. If the value of Y0 is greater than the value of Yi, the score may be increased by one, otherwise the score may be reduced by one. The score can be quantized to form K classes. The second classifier can classify the current sample into one of the K classes. For example, the neighboring luma samples can be obtained from four neighboring samples located above, to the left, to the right, and below the current luma sample, i.e., there are no diagonal neighboring samples.

[0254] It is contemplated that multiple first classifiers, second classifiers, or different instances of the first classifiers or second classifiers or other classifiers described herein may be combined. For example, the first classifier may be combined with an existing classifier based on an MMLM threshold. For another example, an instance A of the first classifier may be combined with another instance B of the first classifier, wherein instances A and B adopt different directions (e.g., vertical and horizontal directions, respectively).

[0255] Those skilled in the art will recognize that, although the existing CCLM design in the VVC standard is used as the basic CCLM method in the specification, the proposed cross-component method described in the present disclosure can also be applied to other predictive codecs with similar design spirits. For example, for chroma from luma (CfL) in the AV1 standard, the proposed method can also be applied by dividing the luma-chroma sample pairs into multiple sample groups.

[0256] Those skilled in the art will recognize that Y / Cb / Cr can also be expressed as Y / U / V in the field of video coding and decoding. For example, if the video data is in RGB format, the proposed method can also be applied by simply mapping the YUV symbol to GBR.

[0257] Filter-based Linear Model (FLM)

[0258] Considering the possibility that one chrominance sample may be correlated with multiple luma samples simultaneously, a filter-based linear model (FLM) using the MLR model is introduced as follows.

[0259] For the chroma samples to be predicted, the reconstructed co-located and adjacent luma samples can be used to predict the chroma samples to capture the inter-sample correlation between co-located luma samples, adjacent luma samples, and chroma samples. The reconstructed luma samples are linearly weighted and combined with an "offset" to generate predicted chroma samples (C: predicted chroma samples, L i : the i-th reconstructed co-located or adjacent brightness sample, α i : filter coefficient, β: offset, N: filter tap), as shown in the following equation (32-1). Note that the linear weighting plus the offset value directly forms the predicted chroma sample (which can be adaptive to low pass, high pass according to the video content), and it is then added to the residual to form the reconstructed chroma sample.

[0260]

[0261] In some implementations like CCCM, the offset term may also be implemented as the intermediate chrominance value B (512 for 10-bit content) multiplied by another coefficient, as shown in equation (32-2) below.

[0262]

[0263] For a given CU, the top and left reconstructed luma and chroma samples can be used to derive or train the FLM parameters (α i, , β).

[0264] As with CCLM, α i and β can be derived via OLS. The top and left training samples are collected and a pseudo-inverse matrix is ​​computed at both the encoder and decoder sides to derive the parameters, which are then used to predict the chroma samples in a given CU. Let N denote the number of filter taps applied to the luma samples, M denote the total top and left reconstructed luma-chroma sample pairs used for training the parameters, represents the luma sample with the i-th sample pair and the j-th filter tap, C i Denotes the chrominance sample with the i-th sample pair, the following equation shows the pseudo-inverse matrix A + The derivation and parameters of . Fig.11 An example is shown where N is 6 (6 taps), M is 8, and parameters are derived or trained using luma samples from the top 2 rows and 3 columns from the left, and chroma samples from the top 1 row and 1 column from the left.

[0265]

[0266] Note that without the bias β, one can only i Predict chroma samples, which may be a subset of the proposed method.

[0267] The proposed ELM / FLM / GLM (discussed below) can be directly extended to the CfL design in the AV1 standard, which explicitly sends the model parameters (α, β). For example, (1-tap case) α and / or β are derived at the encoder at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level and signaled to the decoder for use in CfL mode.

[0268] To further improve the encoding and decoding performance, additional designs can be used in FLM prediction. Fig.11 As shown and discussed above, a 6-tap luma filter is used for FLM prediction. However, while a multi-tap filter can fit the training data (e.g., the top and left neighboring reconstructed luma and chroma samples) well, in some cases, the training data does not capture the complete characteristics of the test data, which may lead to overfitting and may not predict the test data (i.e., the chroma block samples to be predicted) well. In addition, different filter shapes can adapt well to different video block contents, resulting in more accurate predictions.

[0269] To address this issue, the filter shape and the number of filter taps may be predefined or signaled or switched at a sequence parameter set (SPS), an adaptive parameter set (APS), a picture parameter set (PPS), a picture header (PH), a slice header (SH), a region, a CTU, a CU, a sub-block, or a sample level. A set of filter shape candidates may be predefined, and the selection of the set of filter shape candidates may be signaled or switched at the SPS, APS, PPS, PH, SH, region, CTU, CU, sub-block, or sample level. Different components (e.g., U and V) may have different filter switching controls. For example, a set of filter shape candidates may be predefined (e.g., indicated by indices 0 to 5), and a filter shape (1,2) may represent a 2-tap luminance filter, a filter shape (1,2,4) may represent a 3-tap luminance filter, and so on. Fig.11 As shown. The filter shape selection for U and V components can be switched in PH or CU or CTU level. Note that N taps can represent N taps with or without offset β as described in this document. Table 8 below gives an example.

[0270] Table 8 - Example signaling and switching for different filter shapes

[0271]

[0272]

[0273] Different chroma types and / or color formats may have different predefined filter shapes and / or taps. Fig.12 As shown, predefined filter shapes (1,2,4,5) can be used for 4:2:0 type-0, predefined filter shapes (0,1,2,4,7) can be used for 4:2:0 type-2, and predefined filter shapes (1,4) can be used for 4:2:2, and predefined filter shapes (0,1,2,3,4,5) can be used for 4:4:4.

[0274] In another aspect of the present disclosure, unavailable luma and chroma samples used to derive the MLR model can be filled in from available reconstructed samples. Fig.12 The 6-tap (0,1,2,3,4,5) filter shown, then for the CU located at the left picture boundary, the left column including sample (0,3) is unavailable (outside the picture boundary), so sample (0,3) is a repeated padding from sample (1,4) to apply the 6-tap filter. Note that the padding process can be applied to both training data (reconstructed luma and chroma samples adjacent to the top and left) and test data (luma and chroma samples in the CU).

[0275] One or more shapes / numbers of filter taps can be used for FLM prediction, examples are given in Fig.16 , Fig.17 and FIG. 18A to FIG. 18B One or more sets of filter taps may be used for FLM prediction, an example being Figures 19A to 19G Shown in.

[0276] As mentioned above, the MLR model (linear equation) must be derived at both the encoder and the decoder. According to one or more aspects of the present disclosure, several methods are proposed to derive the pseudo-inverse matrix A + , or directly solve the linear equation. Other known methods can also be applied, such as Newton's method, Cayley–Hamilton method, and eigendecomposition as mentioned in https: / / en.wikipedia.org / wiki / Invertible_matrix.

[0277] In the present disclosure, A + Indicated as A -1, to simplify. The linear equation can be solved as follows: 1. Solve A by adjoint matrix (adjA), closed form, analytical solution -1 :

[0278] An n×n general form, a 2×2, and a 3×3 case are shown below. If FLM uses 3×3, then 2 scalers need to be solved plus an offset.

[0279] b=Ax,x=(A T A) -1 A T b=A + b, represented by A -1 b

[0280]

[0281] By removing the (n-1)×(n-1) submatrix with the j-th row and the i-th column.

[0282]

[0283] 2. Gauss-Jordan elimination method

[0284] The linear equation can be solved using Gauss-Jordan elimination by augmenting the matrix [AI n ] and a series of elementary row operations to obtain the simplified row echelon form [I|X]. 2×2 and 3×3 examples are shown below.

[0285]

[0286] 3. Cholesky decomposition

[0287] To solve Ax=b, we can first decompose A using the Cholesky-Crout algorithm to obtain an upper triangular matrix and a lower triangular matrix, and then apply a forward substitution and a backward substitution in sequence to obtain the solution. A 3x3 example is shown below.

[0288]

[0289] In addition to the above examples, some cases require special handling. For example, if certain conditions cause a linear equation to be unsolvable, default values can be used to fill the chrominance prediction values. The default values can be predefined or signaled or switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. For example, when predefined as 1<<(bitDepth-1), meanC, meanL, or meanC-meanL (average current chrominance or other chrominance, available luminance values, or a subset of the reconstructed neighboring regions of the FLM).

[0290] The following example shows a case where matrix A is unsolvable, in which the default predictor can be assigned to the entire current block:

[0291] 1. Solved by closed form (analytical, adjoint matrix), but A is singular (i.e., detA = 0);

[0292] 2. Solved by Cholesky decomposition, but A cannot be Cholesky decomposed, g jj <REG_SQR, where REG_SQR is a small value that can be predefined or signaled or switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level.

[0293] Fig.11 Illustrates a typical case of deriving FLM parameters using the top 2 and / or left 3 luminance rows and the top 1 and / or left 1 chrominance row. However, as mentioned above, due to different block contents and the reconstruction quality of different neighboring samples, using different regions for parameter derivation may bring coding and decoding benefits. Several methods for selecting the application region for parameter derivation are proposed below:

[0294] 1. Similar to MDLM, FLM derivation can only use the top or left luminance and / or chrominance samples to derive parameters. Whether to use FLM, FLM_L, or FLM_T can be predefined or signaled or switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. Assuming the size of the current chrominance block is W×H, W' and H' are obtained as follows:

[0295] - When applying the FLM mode, W' = W, H' = H;

[0296] - When applying the FLM_T mode, W' = W + We; where We represents the extended top luminance / chrominance samples;

[0297] - When FLM_L mode is applied, H'=H+He; where He represents the extended left luma / chroma samples.

[0298] The number of extended luma / chroma samples (We, He) may be predefined or signaled or switched in SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level.

[0299] For example, (We, He) = (H, W) is predefined as VVC CCLM, or (W, H) is predefined as ECM CCLM. Unavailable (We, He) luma / chroma samples can be repeatedly filled from the nearest (horizontal, vertical) luma / chroma samples.

[0300] Fig.13 A diagram of FLM_L and FLM_T is shown (eg, under 4 taps). When FLM_L or FLM_T is applied, only H' or W' luma / chroma samples, respectively, are used for parameter derivation.

[0301] 2. Similar to MRL, different row indices can be predefined or signaled or switched in SPS / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level to indicate the selected luminance-chrominance sample pair rows. This may benefit from different reconstruction qualities of different row samples.

[0302] Fig.14 It is shown that, similar to MRL, FLM can use different rows for parameter derivation (e.g., under 4 taps). For example, FLM can use light blue / yellow luma and / or chroma samples in index 1.

[0303] 3. Expand the CCLM region and obtain the complete top N rows and / or left M rows for parameter derivation. Fig.14 It is shown that all dark blue and light blue and yellow regions can be used simultaneously. Using larger regions (data) for training may produce a more robust MLR model.

[0304] It should be understood that throughout the disclosure, luma sample values ​​of an outer region of a video block to be decoded may be referred to as “outer luma sample values”, and chroma sample values ​​of an outer region may be referred to as “outer luma sample values”.

[0305] The corresponding syntax for FLM prediction may be defined in Table 9 as follows. Wherein, FLC represents fixed length code, TU represents truncated unary code, EGk represents exponential Golomb code with k order, where k may be fixed or signaled / switched in SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level, SVLC represents signed EG0, and UVLC represents unsigned EG0.

[0306] Table 9 - Example of FLM syntax

[0307]

[0308]

[0309] Note that the binarization of each syntax element can be changed.

[0310] Based on the existing linear model design, a new method for cross-component prediction is proposed to further improve the encoding and decoding accuracy and efficiency. The main aspects of the proposed method are detailed as follows.

[0311] While the FLM discussed above provides the best flexibility (giving the best performance), it requires solving for many unknown parameters if the number of filter taps increases. When the inverse matrix is ​​larger than 3×3, the closed form derivation is not suitable (too many multipliers) and an iterative method like Cholesky is required, which increases the burden on the decoder processing loop. In this section, pre-operation before applying the linear model is proposed, including the use of sample gradients to exploit the correlation between the luminance AC information and the chrominance intensity. With the help of the gradient, the number of filter taps can be effectively reduced.

[0312] Note that the methods / examples in this section can be combined / reused from any of the designs discussed above, including but not limited to classification, filter shapes, matrix derivation (with special treatments), application areas, syntax. In addition, the methods / examples listed in this section can also be applied to any of the designs discussed above to achieve better performance at a certain complexity trade-off.

[0313] Note that reference samples / training templates / reconstructed neighborhood as used herein typically refer to luma samples used to derive MLR model parameters, which are then applied to internal luma samples in a CU to predict chroma samples in the CU.

[0314] According to the proposed method, pre-operation (e.g., pre-linear weighting, sign, scale / absolute value, thresholding, ReLU) can be applied to reduce the dimensionality of unknown parameters instead of directly using luma sample intensity values ​​as input to the linear model. In one example, the pre-operation can include: calculating sample differences based on luma sample values. As understood by those skilled in the art, sample differences can be characterized as gradients, and thus this new method is also referred to as a gradient linear model (GLM) in some embodiments.

[0315] Note that the following detailed description discusses scenarios in which the proposed pre-operation can be reused for the SLR model / combined with the SLR model (also known as the 1-tap case), and reused for the MLR model / combined with the MLR model (also known as the multi-tap case, e.g., 2-tap).

[0316] For example, instead of applying a 2-tap on 2 luma samples, the 2 luma samples may be pre-computed and then a simpler 1-tap may be applied to reduce complexity. FIG. 15A to FIG. 15D Some examples of 1-tap / 2-tap (with offset) pre-operation are shown, where the 2-tap coefficients are represented as (a,b). Note that FIG. 15A to FIG. 15D Each circle shown in represents an illustrative chroma position in the YUV 4:2:0 format. As discussed above, in the YUV 4:2:0 format, a luma sample corresponding to a chroma sample can be obtained by performing a downsampling operation on more than one (e.g., 4) reconstructed luma samples corresponding to the chroma sample (e.g., located around the chroma sample). In other words, a chroma position can correspond to one or more luma samples including a co-located luma sample. Different 1-tap modes are designed for different gradient directions and use different "interpolated" luma samples (weighted to different luma positions) for gradient calculation. For example, a typical filter [1,0,-1; 1,0,-1] such as Fig.15A , Fig. 15C and Fig.15D As shown, this means the following operation:

[0317]

[0318] Among them, rec L Represents the reconstructed luma sample value, and Rec L ″(i,j) represents the pre-calculated brightness sample value. Also note that Fig.15A , Fig. 15C and Fig.15D The 1-tap filter shown can be understood as an alternative to the downsampling filter as used in CCLM (see equations (6)-(7)), with the filter coefficients changed.

[0319] Pre-operation can be based on gradient, edge direction (detection), pixel intensity, pixel change, pixel variance, Roberts / Prewitt / compass / Sobel / Laplacian operators, high-pass filters (by calculating gradients or other related operators), low-pass filters (by performing weighted average operations)... etc. The edge direction detector listed in the example can be extended to different edge directions. For example, 1 tap (1,-1) or 2 taps (a,b) are applied along different directions to detect different edge gradients. The filter shape / coefficients can be symmetric about the chrominance position, with FIG. 15A to FIG. 15D Same as example (420 type 0 case).

[0320] Pre-operation parameters (coefficient, sign, scale / absolute value, thresholding, ReLU) can be fixed or signaled / switched at SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. Note that in the example, if multiple coefficients apply to one sample (e.g., -1, 4), they can be merged (e.g., 3) to reduce operations.

[0321] In one example, the pre-operation may involve calculating the sample difference of the luma sample value. Alternatively, the pre-operation may include: performing downsampling by weighted averaging. In some cases, the pre-operation may be applied repeatedly. For example, a template filter may be applied to the template using a low-pass smoothing FIR filter [1,2,1] / 4 or [1,2,1; 1,2,1] / 8 (i.e., downsampling) to remove outliers, and then a 1-tap GLM filter may be applied to calculate the sample difference to derive a linear model. It is contemplated that the sample difference may also be calculated and then downsampling may be enabled.

[0322] In one example, pre-operation coefficients (applied last (eg, 3), or applied intermediately (eg, -1, 4) to each luma sample) may be limited to power-of-2 values ​​to save multipliers.

[0323] In one aspect of the present disclosure, the proposed new method can be reused / combined with the CCLM discussed above, which utilizes a simple linear regression (SLR) model and uses one corresponding luma sample value to predict the chroma sample value. This is also referred to as the 1-tap case. In this case, deriving the linear model further includes deriving a scaling parameter α and an offset parameter β by using pre-calculated neighboring luma sample values ​​and neighboring chroma sample values. Alternatively, the linear model can be rewritten as:

[0324] C=α·L+β (35)

[0325] Where L here represents the "pre-computed" luma sample. The parameter derivation of the 1-tap GLM can reuse the CCLM design, but consider the directional gradient (possibly with a high-pass filter). In one example, the scaling parameter α can be derived by using a division lookup table (described in detail below) for simplification.

[0326] In one example, when the GLM is combined with the SLR model, the scaling parameter α and the offset parameter β can be derived by using the minimum-maximum method discussed above. Specifically, the scaling parameter α and the offset parameter β can be derived by comparing the pre-calculated adjacent luminance sample values ​​to determine the minimum luminance sample value Y A and the maximum brightness sample value Y B ; Determine the minimum brightness sample value Y A and the maximum brightness sample value Y B The corresponding chroma sample value X A and X B ; and according to the following equation, based on the minimum brightness sample value Y A , maximum brightness sample value Y B and the corresponding chrominance sample value X A and X B Derive the scaling parameter α and the offset parameter β:

[0327]

[0328] In one example, when the GLM is combined with the SLR model, the scaling adjustment discussed above can be reused. In this case, the encoder can determine the scaling adjustment value (e.g., "u") to be signaled in the bitstream and add the scaling adjustment value to the derived scaling parameter α. The decoder can determine the scaling adjustment value (e.g., "u") from the bitstream and add the scaling adjustment value to the derived scaling parameter α. The added value is ultimately used to predict the intra chroma sample value.

[0329] In one aspect of the present disclosure, the proposed new method can be reused / combined with FLM, which utilizes a multiple linear regression (MLR) model and uses multiple luma sample values ​​to predict chroma sample values. This is also called a multi-tap case, such as 2-tap. In this case, the linear model can be rewritten as:

[0330]

[0331]

[0332] In this case, multiple scaling parameters α and offset parameters β can be derived by using the pre-operated adjacent luma sample values ​​and adjacent chroma sample values. In one example, the offset parameter β is optional. In one example, at least one of the multiple scaling parameters α can be derived by using the sample difference. In addition, another scaling parameter of the multiple scaling parameters α can be derived by using the downsampled luma sample value. In one example, at least one of the multiple scaling parameters α can be derived by using the horizontal or vertical sample difference calculated based on the downsampled adjacent luma sample value. In other words, the linear model can combine multiple scaling parameters α associated with different pre-operations.

[0333] Implicit filter shape derivation

[0334] In one example, the directional filter shape used can be derived at the decoder to save bit overhead instead of explicitly signaling the selected filter shape index. For example, at the decoder, multiple directional gradient filters can be applied to each reconstructed luminance sample of the L-shaped template of the i-th adjacent row and column of the current block. Then, the filtered values ​​(gradients) can be accumulated for each direction in the multiple directional gradient filters respectively. In the example, the accumulated value is the accumulated value of the absolute value of the corresponding filtered value. After accumulation, the direction of the directional gradient filter with the largest accumulated value can be determined as the derived (luminance) gradient direction. For example, a gradient histogram (HoG) can be constructed to determine the maximum value. The derived direction can be further used as the direction for predicting the chrominance samples in the current block.

[0335] The following example involves reusing the decoder-side intra mode derivation (DIMD) method included in ECM-4.0 for luma intra prediction:

[0336] Step 1: Apply 2 directional gradient filters (3×3 horizontal / vertical Sobel) to each reconstructed luminance sample of the L-shaped template of the second neighboring row and column of the current block;

[0337] Step 2: Accumulate the filtered value (gradient) for each directional gradient filter in the directional gradient filter through SAD (sum of absolute differences);

[0338] Step 3: constructing a Histogram of Gradients (HoG) based on the accumulated filtered values; and

[0339] Step 4: The maximum value in HoG is determined as the derived (brightness) gradient direction, based on which the GLM filter can be determined.

[0340] In one example, if the shape candidates are [-1, 0, 1; -1, 0, 1] (horizontal) and [1, 2, 1; -1, -2, -1] (vertical), when the maximum value is associated with the horizontal shape, the shape [-1, 0, 1; -1, 0, 1] is used for GLM-based chrominance prediction.

[0341] The shape of the gradient filter used to derive the gradient direction can be the same or different in shape from the GLM filter. For example, both filters can be horizontal [-1, 0, 1; -1, 0, 1], or the two filters can have different shapes, and the GLM filter can be determined based on the gradient filter.

[0342] The proposed GLM can be combined with the MMLM or ELM discussed above. When combined with classification, each group can share or have its own filter shape, where the syntax indicates the shape for each group. For example, as an exemplary classifier, the horizontal gradient grad_hor can be classified into the first group, which corresponds to the first linear model, and the vertical gradient grad_ver can be classified into the second group, which corresponds to the second linear model. In one example, the horizontal brightness pattern can be generated only once.

[0343] Another possible classifier is provided as follows. Using the classifier, the neighboring and internal luma sample pairs of the current video block can be classified into a plurality of groups based on one or more thresholds. Note that, as discussed above, each neighboring / internal chroma sample and its corresponding luma sample can be referred to as a luma-chroma sample pair. The one or more thresholds are associated with the intensity of the neighboring / internal luma samples. In this case, each of the plurality of groups corresponds to a corresponding linear model of the plurality of linear models.

[0344] When combined with the MMLM classifier, the following operations may be performed: neighboring reconstructed luminance-chrominance sample pairs of the current video block are classified into two groups based on Threshold; different linear models are derived for different groups, wherein the derivation process may be simplified by GLM, i.e., the number of taps is reduced by the above-mentioned pre-operation; similarly, the luminance-chrominance sample pairs (internal luminance-chrominance sample pairs, wherein each internal luminance-chrominance sample pair in the internal luminance-chrominance sample pair includes an internal chrominance sample value predicted by the derived linear model) within the CU are classified into two groups based on Threshold; different linear models are applied to the reconstructed luminance samples in different groups; and chrominance samples in the CU are predicted based on different classified linear models. Wherein, Threshold may be the average value of the neighboring reconstructed luminance samples. Note that the number of classes (2) can be extended to multiple classes by increasing the number of Thresholds (e.g., equally divided based on the min / max values ​​of adjacent reconstructed (downsampled) luma samples, fixed or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level).

[0345] In one example, instead of the MMLM luma DC intensity, the filtered values ​​of the FLM / GLM applied to the neighboring luma samples are used for classification. For example, if a 1-tap (1, -1) GLM is applied, the average AC value (physical meaning) is used. The processing may be: based on one or more filter shapes, one or more filtered values, and K-1Threshold Ti, the neighboring reconstructed luma-chroma sample pairs are classified into K groups; different MLR models are derived for different groups, wherein the derivation process may be GLM simplified, that is, the number of taps is reduced by the above-mentioned pre-operation; similarly, based on one or more filter shapes, one or more filtered values, and K-1Threshold Ti, the luma-chroma sample pairs (internal luma-chroma sample pairs, wherein each internal luma-chroma sample pair in the internal luma-chroma sample pair includes an internal chroma sample value predicted by the derived linear model) within the CU are classified into K groups; different linear models are applied to the reconstructed luma samples in different groups; and the chroma samples in the CU are predicted based on different classified linear models. Among them, Threshold can be predefined (e.g., 0, or can be a table) or signaled / switched at SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. For example, Threshold can be the average AC value (filtered value) of adjacent reconstructed (can be downsampled) luma samples (2 groups), or based on minimum / maximum AC equal division (K groups).

[0346] It has also been proposed to combine GLM with ELM classifiers. FIG. 15A to FIG. 15D As shown, a filter shape (e.g., 1 tap) can be selected to calculate the edge strength. The direction is determined as the direction along which the sample difference between the current sample and N neighboring samples (e.g., all 6 luma samples) is calculated. For example, Fig.15A The filter at the middle upper part of (shape [1,0,-1; 1,0,-1]) indicates the horizontal direction, because the sample difference between samples can be calculated in the horizontal direction, and the filter below it (shape [1,2,1; -1,-2,-1]) indicates the vertical direction, because the sample difference between samples can be calculated in the vertical direction. The positive coefficients and negative coefficients in each of the filters enable the calculation of sample differences. The processing may then include: calculating an edge strength through the filtered value (e.g., equivalent); quantizing the edge strength into M segments through M-1 thresholds Ti; classifying the current sample using K classes (e.g., K == M); deriving different MLR models for different groups, wherein the derivation process may be simplified by GLM, that is, reducing the number of taps through the above-mentioned pre-operation; classifying the luminance-chrominance sample pairs inside the CU into K groups; applying different MLR models to the reconstructed luminance samples in different groups; and predicting the chrominance samples in the CU based on the differently classified MLR models. Note that the filter shape used for classification can be the same or different from the filter shape used for MLR prediction. Both thresholds M-1 and Ti and the number of thresholds M-1 and Ti can be fixed or signaled / switched in SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. In addition, other classifiers / combined classifiers as discussed in ELM can also be used for GLM.

[0347] If the number of classification samples in a group is less than a number (e.g., a predefined 4), the default values ​​mentioned in the discussion of matrix derivation for the MLR model can be applied to the group parameters (α i, , β). If the corresponding neighboring reconstructed samples are not available for the selected LM mode, a default value may be applied. For example, when the MMLM_L mode is selected but the remaining samples are invalid.

[0348] Several methods related to the simplification of GLM are introduced below to further improve the encoding and decoding efficiency.

[0349] The matrix / parameter derivation in FLM requires floating-point operations (e.g., closed-form division), which is expensive for decoder hardware, thus requiring a fixed-point design. For the 1-tap GLM case, it can be viewed as a modified luminance reconstructed sample generation of CCLM (e.g., horizontal gradient direction, from CCLM[1,2,1; 1,2,1] / 8 to GLM[-1,0,1; -1,0,1]), and the original CCLM process can be reused for GLM, including fixed-point operations, MDLM downsampling, division table, applying size constraints, min-max approximation, and scaling adjustment. For all terms, the 1-tap GLM can have its own configuration or share the same design with CCLM. For example, the parameters are derived using a simplified min-max method (instead of LMS) and combined with scaling adjustment after deriving the GLM model. In this case, the center point for rotating the slope (luminance value y r ) becomes the average value of the reference brightness sample "gradient". For another example, when GLM is turned on for the CU, CCLM slope adjustment is inferred to be turned off, and there is no need to signal the syntax related to slope adjustment.

[0350] This section takes the typical case reference sample (top row and left column) as an example. Fig.14 As shown, the extended reconstructed region may also use the same essential simplification and may have a syntax (eg, MDLM, MRL) to indicate a specific region.

[0351] Please note that the following aspects can be combined and applied jointly: For example, the partitioning process is performed in combination with reference sample downsampling and the partitioning table.

[0352] When applying classification (MMLM / ELM), each group can apply the same or different simplification operations. For example, before applying the right shift, fill the samples of each group to the target sample number, and then apply the same derivation process and the same partition table.

[0353] Fixed-point implementation

[0354] The CCLM design can be reused for the 1-tap case, division by n can be implemented by right shift, and division by A2 can be implemented by LUT. Integerization parameters, including n involved in the integerization design of LMS CCLM α , n tableand the intermediate parameters used to derive the linear model (Equations (19)-(20)), which can be the same as CCLM or have different values ​​to obtain higher accuracy. The integerization parameters can be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level and can be adjusted according to the sequence bit depth. For example, n table =bit depth + 4.

[0355] MDLM downsampling

[0356] When GLM is combined with MDLM, the total existing samples used for parameter derivation may not be a power of 2 value and need to be padded to a power of 2 to use the right shift operation instead of division. For example, for an 8×4 chroma CU, MDLM requires W+H=12 samples, while MDLM_T has only 8 samples available (reconstructed), which can then be equivalently filled with 4 downsampled samples (0, 2, 4, 6). The code to implement such an operation is as follows:

[0357]

[0358] Other filling methods may also be applied, such as repeated / mirror filling relative to the last neighboring sample point (rightmost / bottommost).

[0359] The filling method used for GLM can be the same as or different from the filling method of CCLM.

[0360] Note that in the ECM version, the 8×4 chroma CU MDLM_T / MDLM_L requires 2T / 2L=16 / 8 samples respectively, in which case the same padding method can be applied to meet the target power-of-2 number of samples.

[0361] Division LUT

[0362] The partitioning LUT proposed for CCLM / LIC (local illumination compensation) in known standard developments such as AVC / HEVC / AV1 / VVC / AVS can be used for GLM division. For example, the LUT in JCTVC-I0166 is reused with bit depth = 10 (Table 4). The division LUT can be different from CCLM. For example, CCLM uses min-max with DivTable (as in Equation 5), but GLM uses a 32-entry LMS division LUT (as in Table 5).

[0363] When GLM is combined with MMLM, meanL values ​​may not always be positive (e.g., using filtered / gradient values ​​to classify groups), so sgn(meanL) needs to be extracted and abs(meanL) used to look up the division LUT. Note that the division LUTs used for MMLM classification and parameter derivation can be different. For example, use a lower precision LUT (such as the LUT in min-max) for mean classification and a higher precision LUT (such as in LMS) for parameter derivation.

[0364] Size constraints and latency constraints

[0365] Similar to CCLM design, some size constraints can be applied for ELM / FLM / GLM. For example, the same constraints for luma-chroma delay in dual trees can be applied.

[0366] Size constraints can be based on CU area / width / height / depth. Thresholds can be predefined or signaled at SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. For example, for chroma CU area, the predefined threshold can be 128.

[0367] In one example, in response to determining that a video block satisfies an enablement threshold, at least one pre-operation is performed, wherein the enablement threshold is associated with an area, width, height, or segmentation depth of the video block. Specifically, the enablement threshold may define a minimum or maximum area, width, height, or segmentation depth of the video block. As will be appreciated by those skilled in the art, a video block may include a current chroma block and its co-located luma block. It is also proposed that the above-mentioned enablement threshold is applied jointly to the current chroma block and its co-located luma block. For example, in response to determining that both the current chroma block and its co-located luma block satisfy the enablement threshold, at least one pre-operation is performed.

[0368] Line buffer reduction

[0369] Similar to the CCLM design, if the co-located luma region of the current chroma CU contains the first row inside a CTU, the top template sample generation can be limited to 1 row to reduce the row buffer storage of CTU rows. Note that when the upper reference row is at a CTU boundary, only one luma row (common row buffer in intra prediction) is used to make the downsampled luma samples.

[0370] For example, in Fig.13In , if the co-located luma region of the current chroma CU contains the first row inside a CTU, the top template can be restricted to using only 1 row (instead of 2 rows) for parameter derivation (other CUs can still use 2 rows). This can save luma sample row buffer storage when processing CTUs row by row at the decoder hardware. Several methods can be used to achieve row buffer reduction. Note that the example of a limited "1" row can be extended to N rows using similar operations. Similarly, 2-tap or multi-tap can also apply such operations. When multi-tap is applied, chroma samples may also need to apply operations.

[0371] For example, Fig.15A The 1-tap filter [1,0,-1; 1,0,-1] shown is explained as an example. The filter can be reduced to [0,0,0; 1,0,-1], i.e., only the lower row coefficients are used. Alternatively, the limited upper row luma samples can be filled from the lower row luma samples (repeat, mirror, 0, meanL, meanC, ..., etc.).

[0372] Take N=4 as an example, that is, the video block is located at the top boundary of the current CTU, and the neighboring luminance sample values ​​and corresponding chrominance sample values ​​of the top 4 rows are used to derive the linear model. Note that the corresponding chrominance sample values ​​may refer to the corresponding neighboring chrominance sample values ​​of the top 4 rows (e.g., for YUV 4:4:4 format). Alternatively, the corresponding chrominance sample values ​​may refer to the corresponding top 2 rows of neighboring chrominance sample values ​​(e.g., for YUV 4:2:0 format). In this case, the neighboring luminance sample values ​​and corresponding chrominance sample values ​​of the top 4 rows can be divided into two regions—a first region, which includes valid sample values ​​(e.g., the luminance sample values ​​and corresponding chrominance sample values ​​of the most recent row); and a second region, which includes invalid sample values ​​(e.g., the other three rows of luminance sample values ​​and corresponding chrominance sample values). The coefficients of the filter corresponding to the sample positions that do not belong to the first region can then be set to zero, so that only the sample values ​​from the first region are used to calculate the sample difference. For example, as discussed above, in this case, the filter [1,0,-1; 1,0,-1] can be simplified to [0,0,0; 1,0,-1]. Alternatively, the nearest sample values ​​in the first region can be padded to the second region so that the padded sample values ​​can be used to calculate the sample differences.

[0373] Fusion of Chroma Intra Prediction Modes

[0374] In one example, since GLM can be used as a special CCLM mode, the fusion design can be reused or have its own way. Multiple (two or more) weights can be applied to generate the final predictor. For example,

[0375] pred=(w0*pred0+w1*pred1+(1<<(shift-1)))>>shift

[0376] where pred0 is a non-LM based predictor and pred1 is a GLM based predictor, or

[0377] pred0 is a predictor based on one of the CCLMs (including all MDLM / MMLMs), and pred1 is a predictor based on a GLM, or

[0378] pred0 is a GLM based predictor and pred1 is a GLM based predictor.

[0379] Different I / P / B slices may have different weight designs (w0 and w1), depending on whether the neighboring blocks are coded with CCLM / GLM / other coding modes or block size / width / height.

[0380] For example, the design for the weight can be determined by the intra prediction mode of the adjacent chroma block, and shift is set to be equal to 2. Specifically, when the upper and left adjacent blocks are both coded and decoded using the LM mode, {w0, w1} = {1, 3}; when the upper and left adjacent blocks are both coded and decoded using the non-LM mode, {w0, w1} = {3, 1}; otherwise, {w0, w1} = {2, 2}. For non-I slices, w0 and w1 can both be set to be equal to 2.

[0381] For grammar design, if non-LM mode is selected, a flag is signaled to indicate whether fusion is applied.

[0382] As described above, GLM has a good gain-complexity trade-off because it can reuse the existing CCLM module without introducing additional derivations. Such a 1-tap design can be further extended or generalized according to one or more aspects of the present disclosure.

[0383] In one aspect of the present disclosure, for a chroma sample to be predicted, a single corresponding luma sample L may be generated by combining the co-located luma sample and the adjacent luma sample. For example, the combination may be a combination of different linear filters, such as a high-pass gradient filter (GLM) and a low-pass smoothing filter (e.g., a [1,2,1; 1,2,1] / 8 FIR downsampling filter that can be generally used in CCLM); and / or a linear filter and a nonlinear filter (e.g., with a power of n, such as L n , n can be a positive number, a negative number, or a combination of +- fractions (e.g., +1 / 2, square root, or +3, cube, which can be rounded and rescaled to the bit depth dynamic range).

[0384] In one aspect of the present disclosure, the combination may be applied repeatedly. For example, a combination of GLM and [1,2,1; 1,2,1] / 8 FIR may be applied on the reconstructed luma samples, and then a nonlinear power of 1 / 2 may be applied. For example, the nonlinear filter may be implemented as a LUT (lookup table), e.g., for bit depth = 10, power of n, n = 1 / 2, LUT[i] = (int)(sqrt(i) + 0.5) << 5, i = 0-1023, where 5 will be scaled to the bit depth = 10 dynamic range. When a linear filter cannot efficiently handle the luma-chroma relationship, a nonlinear filter may provide an option. Whether to use nonlinear terms may be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level.

[0385] In one or more aspects of the present disclosure, GLM may refer to a generalized linear model (which may be used to generate a single brightness sample linearly or nonlinearly, and the generated single brightness sample may be fed into a CCLM linear model to derive parameters of the CCLM linear model), and the linear / nonlinear generation may be referred to as a general mode. Different gradients or general modes may be combined to form another mode. For example, a gradient mode may be combined with a CCLM downsampled value; a gradient mode may be combined with a nonlinear L 2 Value combination; a gradient pattern can be combined with another gradient pattern. The two gradient patterns to be combined can have different directions or the same direction. For example, [1,1,1; -1, -1, -1] and [1,2,1; -1, -2, -1] can be combined, both of which have vertical directions. You can also combine [1,1,1; -1, -1, -1] and [1,0, -1; 1,0, -1], both of which have vertical and horizontal directions, such as FIG. 15A to FIG. 15D The combination may include additive, subtractive or linear weighting.

[0386] GLM applied to the downsampled domain

[0387] As described above, the pre-operation may be repeatedly applied, and the GLM may be applied to the pre-linearly weighted / pre-operated samples. For example, as CCLM, a template filter may be applied to the luma samples to remove outliers (i.e., CCLM downsampling smoothing filter) using a low-pass smoothing FIR filter [1,2,1; 1,2,1] / 8 and generate downsampled luma samples (one downsampled luma sample corresponding to one chroma sample). And thereafter, a 1-tap GLM may be applied to the smoothed downsampled luma samples to derive the MLR model.

[0388] Some gradient filter modes (such as 3×3 Sobel or Prewitt operator) can be applied to the downsampled luma samples. The following table shows some of the gradient filter modes.

[0389]

[0390]

[0391] The gradient filter pattern can be combined with other gradient / generic filter patterns in the downsampled luma domain. In one example, the combined filter pattern can be applied to the downsampled luma samples. For example, the combined filter pattern can be derived by performing addition or subtraction operations on the corresponding coefficients of the gradient filter pattern and a DC / lowpass based filter pattern (such as the filter pattern [0,0,0; 0,1,0; 0,0,0] or [1,2,1; 2,4,1; 1,2,1]). In another example, the combined filter pattern can be derived by performing addition or subtraction operations on the coefficients of the gradient filter pattern and a nonlinear value (such as L 2 ) to derive the combined filter pattern. In another example, the combined filter pattern is derived by performing an addition or subtraction operation on the corresponding coefficients of the gradient filter pattern and another gradient filter pattern with a different direction or the same direction. In another example, the combined filter pattern is derived by performing a linear weighting operation on the coefficients of the gradient filter pattern.

[0392] A GLM applied in the downsampled domain can fit into the CCCM framework but may sacrifice high frequency accuracy because low-pass smoothing is applied before applying the GLM.

[0393] GLM as input to CCCM

[0394] As shown above, CCCM, like CCLM, applies luma downsampling before convolution. When chroma subsampling is used, the reconstructed luma samples are downsampled to match the lower resolution chroma grid. Since a 1-tap GLM can also be viewed as changing the CCLM downsampling filter coefficients (e.g., from [1,2,1; 1,2,1] / 8 to [1,2,1,-1,-2,-1], i.e., from low pass to high pass), the GLM can be used as the input to the CCCM. Specifically, the gradient filter of the GLM replaces the luma downsampling filter ([1,2,1; 1,2,1] / 8) with gradient-based coefficients (e.g., [1,2,1,-1,-2,-1]). In this case, the CCCM operation becomes a "linear / nonlinear combination of gradients" as shown below:

[0395] predChromaVal=c0C+c1N+c2S+c3E+c4W+c5P+c6B

[0396] Where C, N, S, E, W, P are the gradients of the current or neighboring samples (compared to the original downsampled values ​​for CCCM). The related GLM methods described in this disclosure can be applied in the same way before entering the CCCM convolution, such as classification, separate Cb / Cr control, syntax, mode combination, PU size limit, etc.

[0397] Gradient-based coefficient replacement can be applied to specific CCCM taps. In addition, not only high-pass, but also low-pass / band-pass / all-pass coefficient replacement can be used. The replacement can be combined with the FLM / CCCM shape switching discussed above (resulting in different tap numbers). For example, FIG. 15A to FIG. 15D The gradient mode in can be used for replacement. In one example, the operation for applying GLM to the input of CCCM includes: predefining one or more coefficient candidates for CCCM / FLM downsampling; determining the CCCM / FLM filter shape and number of filter taps for the CU; applying different CCLM downsampling coefficients to different filter taps, where the coefficients can be a high-pass filter (GLM) or a low-pass / band-pass / all-pass filter; generating downsampled luma samples for CCCM input samples (using the applied coefficients); and inputting the generated downsampled luma samples into the CCCM process.

[0398] Some examples of changing the CCLM downsampling filter coefficients are shown below:

[0399] Example 1:

[0400] The candidate filters are [1,2,1; 1,2,1] / 8 and [1,0,-1; 1,0,-1]; predChromaVal = c0C+c1N+c2S+c3E+c4W+c5P+c6B, using the typical CCCM cross shape, 7 taps; N, S, W, E use the filter [1,2,1; 1,2,1] / 8, retaining the original CCLM downsampling filter; and C, P use the filter [1,0,-1; 1,0,-1], that is, the horizontal gradient filter, and P physically represents the gradient^2.

[0401] Example 2:

[0402] The candidate filters are [1,2,1; 1,2,1] / 8, [1,0,-1; 1,0,-1], [1,2,1; -1,-2,-1], [2,1,-1; 1,-1,-2], and [-1,1,2; -2,-1,1]; predChromaVal = c0C0+c1C1+c2C2+c3C3+c4C4+c5P+c6B; C0 uses the filter [1,2,1; 1,2,1] / 8, retaining the original CCLM downsampling filter; C1 uses the filter [1,0,-1 ; 1,0,-1]; C2 uses filter [1,2,1; -1,-2,-1]; C3 uses filter [2,1,-1; 1,-1,-2]; C4 uses filter [-1,1,2; -2,-1,1]; C5 uses filter [1,2,1; 1,2,1] / 8, retaining the original CCLM downsampling filter; C0 to C5, and P have the same downsampled brightness position (in the typical CCCM cross shape == C); and C1 to C4 are generated by Sobel-based gradient filters in different directions (such as Figures 15A to 15D shown).

[0403] Example 3:

[0404] The candidate filters are [1,2,1; 1,2,1] / 8, [1,0,-1; 1,0,-1], [1,2,1; -1,-2,-1], [0,1,1; 0,1,1], and [1,1,0; 1,1,0]; predChromaVal = c0C0+c1C1+c2C2+c3C3+c4C4+c5P+c6B; C0 uses the filter [1,2,1; 1,2,1] / 8, retaining the original CCLM downsampling filter; C1 uses the filter [1,0,- 1; 1,0,-1]; C2 uses filter [1,2,1; -1, -2, -1]; C3 uses filter [0,1,1; 0,1,1]; C4 uses filter [1,1,0; 1,1,0]; C5 uses filter [1,2,1; 1,2,1] / 8, retaining the original CCLM downsampling filter; C0 to C5, and P have the same downsampled brightness position (in the typical CCCM cross shape == C); C1 to C2 are generated by Sobel-based gradient filters in different directions (such as Figures 15A to 15D as shown); and C3 to C4 are generated by a low-pass filter.

[0405] Which CCCM / FLM taps apply coefficient replacement can be pre-defined at SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level (as shown in the example above) or signaled / switched.

[0406] For each CCCM / FLM tap, coefficient candidates for CCCM / FLM downsampling may be predefined or signaled / switched at SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level.

[0407] Example 4:

[0408] The candidate filters are [1,2,1; 1,2,1] / 8, [1,0,-1; 1,0,-1], [1,2,1; -1,-2,-1], [2,1,-1; 1,-1,-2], and [-1,1,2; -2,-1,1]; predChromaVal = c0C+c1N+c2S+c3E+c4W+c5P+c6B, using the typical CCCM crossover shape, 7 taps; C uses a switching downsampling filter between the 5 candidate filters; N, S, W, E use the filter [1, 2, 1; 1, 2, 1] / 8, retaining the original CCLM downsampling filter; and P uses a switching downsampling filter between the 5 candidate filters.

[0409] Example 5:

[0410] The candidate filters are: [1,2,1; 1,2,1] / 8, [1,0,-1; 1,0,-1], [1,2,1; -1,-2,-1], [0,1,1; 0,1,1] and [1,1,0; 1,1,0]; predChromaVal = c0C+c1W+c2E+c3P+c4B, that is, a horizontal minus shape, 5 taps; C uses a switching downsampling filter between the 5 candidate filters; W and E use a switching downsampling filter between the 3 candidate filters: [1,2,1; 1,2,1] / 8, [0,1,1; 0,1,1] and [1,1,0; 1,1,0]; and P uses the filter [1, 2, 1; 1, 2, 1] / 8, retaining the original CCLM downsampling filter.

[0411] In one or more aspects of the present disclosure, one or more syntaxes may be introduced to indicate information about GLM. Table 10 below shows an example of GLM syntax.

[0412] Table 10

[0413]

[0414] FLC: Fixed Length Code

[0415] TU: Truncated Unary Code

[0416] EGk: Exponential Golomb code of order k, where k can be fixed or signaled / switched at SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level.

[0417] SVLC: Signed EG0

[0418] UVLC: Unsigned EG0

[0419] Note that the binarization of each syntax element can be changed.

[0420] In one aspect of the present disclosure, the GLM on / off control for the Cb / Cr components can be performed jointly or separately. For example, at the CU level, 1 flag can be used to indicate whether GLM is valid for this CU. If valid, 1 flag can be used to indicate whether both Cb / Cr are valid. If neither is valid, 1 flag indicates that either Cb or Cr is valid. When Cb and / or Cr are valid, the filter index / gradient (general) mode can be signaled separately. All flags can have their own context model or be bypassed coded.

[0421] In another aspect of the present disclosure, whether to signal the GLM on / off flag can depend on the luma / chroma coding mode and / or CU size. For example, in the ECM5 chroma intra mode syntax, when applying MMLM or MMLM_L or MMLM_T, GLM can be inferred as off; when the CU area < A, where A can be predefined or signaled / switched at SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level; if combined with CCCM, when CCCM is on, GLM can be inferred as off.

[0422] Note that when GLM is combined with MMLM, different models can share the same gradient / general mode or have their own gradient / general mode.

[0423]

[0424] When GLM is combined with CCCM / FLM, if the current CU is enabled for CCCM / FLM, the CU-level GLM enable flag can be inferred as off.

[0425] hasGlmFlag &=!pu.cccmFlag;

[0426] CCCM without downsampling

[0427] CCCM requires processing of downsampled luma reference values ​​before calculating model parameters and applying the CCCM model, which increases the burden on the decoder processing cycle. In this section, CCCM without downsampling is proposed, including: utilizing un-downsampled luma reference values ​​and / or the selection of different un-downsampled luma references. One or more filter shapes can be used for the purposes described below.

[0428] Please note that the methods / examples in this section can be combined / reused with the methods mentioned above, including but not limited to methods related to classification, filter shape, matrix derivation (with special treatments), application area, and syntax. In addition, the methods / examples listed in this section can also be applied together with the above methods / examples (more taps) to have better performance under certain complexity trade-offs.

[0429] In the present disclosure, reference samples / training templates / reconstructed neighborhoods generally refer to luma samples used to derive MLR model parameters, which are then applied to internal luma samples in a CU to predict chroma samples in the CU.

[0430] Filter shape

[0431] One or more shapes / numbers of filter taps can be used for CCCM prediction, such as Fig.16 , Fig.17 and FIG. 18A to FIG. 18B One or more sets of filter taps can be used for FLM prediction, for example in Figures 19A to 19G The selected luma reference value is not downsampled. One or more predefined shapes / numbers of filter taps may be used for CCCM prediction based on previously decoded information on TB / CB / slice / picture / sequence level.

[0432] Although multi-tap filters can fit the training data well (i.e., the reconstructed luminance / chrominance samples adjacent to the top / left), this can lead to overfitting in some cases where the training data does not capture the complete characteristics of the test data, and the test data (i.e., the chrominance block samples to be predicted) cannot be predicted well. In addition, different filter shapes can adapt well to different video block contents, resulting in more accurate predictions. To address this issue, the number of filter shapes / filter taps can be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. A set of filter shape candidates can be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. Different components (U / V) can have different filter switching controls. For example, as shown in the following table, a set of filter shape candidates (index = 0 to 5) are predefined, and the filter shape (1,2) represents a 2-tap brightness filter, while the filter shape (1,2,4) represents a 3-tap brightness filter, etc. Fig.11 As shown in . The filter shape selection for U / V components can be switched in PH or CU / CTU level. Note that N-tap can represent N-tap with or without offset β as described above.

[0433]

[0434] Different chroma types / color formats can have different predefined filter shapes / taps. For example, for 420 type-0: (1,2,4,5), 420 type-2: (0,1,2,4,7), 422: (1,4), 444: (0,1,2,3,4,5) use a predefined filter shape such as Fig.12 as shown in .

[0435] Luma / chroma samples that are not available for deriving the MLR model can be filled in from available reconstructed samples. Fig.12 , then for the CU located at the left picture boundary, the left column including (0,3) is unavailable (beyond the picture boundary), so (0,3) is repeated padding from (1,4) to apply the 6-tap filter. Note that the padding process is applied to both the training data (the reconstructed luma / chroma samples in the top / left neighbor) and the test data (the luma / chroma samples in the CU).

[0436] According to one or more embodiments of the present disclosure, the luma / chroma samples that are unavailable for deriving the MLR model may be skipped and not used. Thus, no filling process is required for the unavailable luma / chroma samples.

[0437] CCLM / MMLM with LDL decomposition

[0438] CCCM needs to process LDL decomposition to calculate the model parameters of CCCM model, thus avoiding the use of square root operations and only requiring integer operations. In this section, CCLM / MMLM with LDL decomposition is proposed. As mentioned above, LDL decomposition can also be used in ELM / FLM / GLM.

[0439] Please note that the methods / examples in this section can be combined / reused with the methods mentioned above, including but not limited to methods related to classification, filter shape, matrix derivation (with special processing), application area, and syntax. In addition, the methods / examples listed in this section can also be applied together with the above methods / examples to have better performance under certain complexity trade-offs.

[0440] In the present disclosure, reference samples / training templates / reconstructed neighborhoods generally refer to luma samples used to derive MLR model parameters, which are then applied to internal luma samples in a CU to predict chroma samples in the CU.

[0441] CCLM / MMLM with extended range

[0442] One or more reference samples can be used for CCLM / MMLM prediction, i.e. Fig. 10B As shown in , the reference region can be the same as the reference region in CCCM. Different reference regions can be used for CCLM / MMLM prediction based on previously decoded information at TB / CB / slice / picture / sequence level.

[0443] Although training data with multiple reference regions can fit the calculation of model parameters well, this may lead to overfitting in some cases where the training data does not capture the complete characteristics of the test data, and the test data (i.e., the chroma block samples to be predicted) cannot be predicted well. In addition, different reference regions can adapt well to different video block contents, resulting in more accurate predictions. To address this issue, the number of reference shapes / reference regions can be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. A set of reference region candidates can be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. Different components (U / V) can have different reference region switching controls. For example, a set of candidate reference regions (index = 0 to 4) is predefined, as shown in the following table. The reference region selection for the U / V component can be switched at the PH or CU / CTU level. Different colorimetric types / color formats may have different predefined reference areas.

[0444]

[0445] Luma / chroma samples that are unavailable for deriving the MLR model can be filled from the available reconstructed samples, and the filling process is applied to both the training data (top / left neighboring reconstructed luma / chroma samples) and the test data (luma / chroma samples in the CU).

[0446] According to one or more embodiments of the present disclosure, the luma / chroma samples that are unavailable for deriving the MLR model may be skipped and not used. Thus, no filling process is required for the unavailable luma / chroma samples.

[0447] FLM / GLM / ELM / CCCM with minimum sample point restriction

[0448] FLM needs to process the downsampled luma reference value and calculate the model parameters, which increases the burden on the decoder processing cycle, especially for small blocks. In this section, FLM with minimum sample restriction is proposed, for example, FLM is only used for samples greater than a predefined number (such as 64, 128). One or more different restrictions can be used for this purpose, for example, FLM is only used for samples greater than a predefined number (such as 256) in a single model, and FLM is only used for samples greater than a predefined number (such as 128) in a multi-model.

[0449] According to one or more embodiments of the present disclosure, the number of predefined minimum sample points for a single model may be greater than or equal to the number of predefined minimum sample points for a multi-model. For example, FLM / GLM / ELM / CCCM is only used for sample points greater than or equal to a predefined number (such as 128) in a single model, and FLM / GLM / ELM / CCCM is only used for sample points greater than or equal to a predefined number (such as 256) in a multi-model.

[0450] According to one or more embodiments of the present disclosure, the number of predefined minimum samples for FLM / GLM / ELM may be greater than or equal to the number of predefined minimum samples for CCCM. For example, CCCM is used only in a single model for samples greater than or equal to a predefined number (such as 0), and CCCM is used only in a multi-model for samples greater than or equal to a predefined number (such as 128). FLM is used only in a single model for samples greater than or equal to a predefined number (such as 128), and FLM is used only in a multi-model for samples greater than or equal to a predefined number (such as 256).

[0451] Please note that the methods / examples in this section can be combined / reused with the methods mentioned above, including but not limited to methods related to classification, filter shape, matrix derivation (with special treatments), application area, and syntax. In addition, the methods / examples listed in this section can also be applied together with the above methods / examples (more taps) to have better performance under certain complexity trade-offs.

[0452] Fig. 20 A workflow of a method 2000 for decoding video data according to one or more aspects of the present invention is shown.

[0453] At step 2010 , method 2000 includes obtaining a video block from a bitstream.

[0454] At step 2020 , the method 2000 includes obtaining an internal luma sample value of the video block, an external luma sample value of an external region of the video block, and an external chroma sample value of an external region.

[0455] At step 2030, method 2000 includes determining a filter pattern based on the external luma sample values ​​and the external chroma sample values, wherein the filter pattern includes one or more linear weighting coefficients for corresponding one or more pre-computed luma sample values, and wherein the one or more pre-computed luma sample values ​​include at least one pre-computed luma sample value obtained based on a sample difference between the external luma sample values.

[0456] At step 2040, method 2000 includes predicting intra chroma sample values ​​of a video block based on intra luma sample values ​​using a filter pattern.

[0457] At step 2050, method 2000 includes obtaining a decoded video block using the predicted intra chroma sample values.

[0458] In one example, the filter pattern also includes a coefficient for a non-linearity value and a coefficient for a bias value.

[0459] In one example, the one or more pre-calculated luma sample values ​​further include at least one pre-calculated luma sample value obtained by weighted average calculation.

[0460] In one example, the one or more pre-operated luma sample values ​​further include at least one pre-operated luma sample value obtained by applying one of a low pass filter, a high pass filter, or a non-linear filter.

[0461] In one example, the non-linear value comprises a square of a corresponding pre-computed luma sample value obtained by a weighted average operation, and wherein the non-linear value is scaled to a luma sample value range of the video block.

[0462] In one example, the non-linear value includes a value obtained by scaling a sum of a square of a corresponding pre-calculated luma sample value obtained via a weighted average operation and an intermediate chroma value corresponding to a luma sample value range of the video block.

[0463] In one example, the non-linear value comprises a square of a corresponding pre-computed luma sample value obtained based on a sample difference between luma sample values, and wherein the non-linear value is scaled to a luma sample value range of the video block.

[0464] In one example, the non-linear value includes a value obtained by scaling a square of a corresponding pre-operated luma sample value obtained based on a sample difference between luma sample values ​​and a sum of intermediate chroma values ​​corresponding to a luma sample value range of the video block.

[0465] In one example, the filter pattern includes two or more linear weighting coefficients for corresponding two or more pre-operated luma sample values, and wherein: at least two of the two or more pre-operated luma sample values ​​correspond to the same luma position and are obtained using the same or different filters; or at least two of the two or more pre-operated luma sample values ​​correspond to different luma positions and are obtained using the same or different filters.

[0466] In one example, the non-linear value and at least one of the one or more pre-computed luma sample values ​​correspond to the same or different luma positions.

[0467] In one example, one or more of the linear weighting coefficients are switched for different coding units (CUs) or between different chroma components.

[0468] Fig.21 A workflow of a method for encoding video data according to one or more aspects of the present disclosure is shown.

[0469] At step 2110 , method 2100 includes obtaining a video block.

[0470] At step 2120 , the method 2100 includes obtaining an internal luma sample value of the video block, an external luma sample value of an external region of the video block, and an external chroma sample value of an external region.

[0471] At step 2130, method 2100 includes determining a filter pattern based on external luma sample values ​​and external chroma sample values, wherein the filter pattern includes one or more linear weighting coefficients for corresponding one or more pre-computed luma sample values, and wherein the one or more pre-computed luma sample values ​​include at least one pre-computed luma sample value obtained based on a sample difference between external luma sample values.

[0472] At step 2140 , method 2100 includes predicting intra chroma sample values ​​of a video block based on intra luma sample values ​​using a filter pattern.

[0473] At step 2150 , the method 2100 includes generating a bitstream including the encoded video block by using the predicted intra chroma sample values.

[0474] In one example, the filter pattern also includes a coefficient for a non-linearity value and a coefficient for a bias value.

[0475] In one example, the one or more pre-calculated luma sample values ​​further include at least one pre-calculated luma sample value obtained by weighted average calculation.

[0476] In one example, the one or more pre-operated luma sample values ​​further include at least one pre-operated luma sample value obtained by applying one of a low pass filter, a high pass filter, or a non-linear filter.

[0477] In one example, the non-linear value comprises a square of a corresponding pre-computed luma sample value obtained by a weighted average operation, and wherein the non-linear value is scaled to a luma sample value range of the video block.

[0478] In one example, the non-linear value includes a value obtained by scaling a sum of a square of a corresponding pre-calculated luma sample value obtained via a weighted average operation and an intermediate chroma value corresponding to a luma sample value range of the video block.

[0479] In one example, the non-linear value comprises a square of a corresponding pre-computed luma sample value obtained based on a sample difference between luma sample values, and wherein the non-linear value is scaled to a luma sample value range of the video block.

[0480] In one example, the non-linear value includes a value obtained by scaling a square of a corresponding pre-operated luma sample value obtained based on a sample difference between luma sample values ​​and a sum of intermediate chroma values ​​corresponding to a luma sample value range of the video block.

[0481] In one example, the filter pattern includes two or more linear weighting coefficients for corresponding two or more pre-operated luma sample values, and wherein: at least two of the two or more pre-operated luma sample values ​​correspond to the same luma position and are obtained using the same or different filters; or at least two of the two or more pre-operated luma sample values ​​correspond to different luma positions and are obtained using the same or different filters.

[0482] In one example, the non-linear value and at least one of the one or more pre-computed luma sample values ​​correspond to the same or different luma locations.

[0483] In one example, one or more of the linear weighting coefficients are switched for different coding units (CUs) or between different chroma components.

[0484] Fig. 22 An exemplary computing system 2200 according to one or more aspects of the present disclosure is shown. The computing system 2200 may include at least one processor 2210. The computing system 2200 may also include at least one storage device 2220. The storage device 2220 may store computer executable instructions that, when executed, cause the processor 2210 to perform the steps of the method described above. The processor 2210 may be a general-purpose processor, or may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such structure. The storage device 2220 may store input data, output data, data generated by the processor 2210, and / or instructions executed by the processor 2210.

[0485] It should be understood that the storage device 2220 may store computer-executable instructions that, when executed, cause the processor 2210 to perform any operations according to the embodiments of the present disclosure.

[0486] Embodiments of the present disclosure may be embodied as computer-readable media such as non-transitory computer-readable media. Non-transitory computer-readable media may include instructions that, when executed, cause one or more processors to perform any operations according to embodiments of the present disclosure. For example, when the instructions are executed, one or more processors may receive a bitstream and perform a decoding operation as described above. For another example, when the instructions are executed, one or more processors may perform an encoding operation and send a bitstream including the encoded video information associated with the above-mentioned predicted chroma samples.

[0487] It should be understood that all operations in the method described above are merely exemplary, and the present disclosure is not limited to any operations in the method or the order of these operations, but should cover all other equivalent transformations under the same or similar concept.

[0488] It should also be understood that all modules in the above described method can be implemented in various ways. These modules can be implemented as hardware, software or a combination thereof. In addition, any module in these modules can be further divided into submodules or combined together functionally.

[0489] The above description is provided so that any person skilled in the art can implement the various aspects described herein. Various modifications of these aspects are obvious to those skilled in the art, and the general principles defined herein can be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein. All structures and functions that are known or will be known to those of ordinary skill in the art and are equivalent to the elements of the various aspects described throughout this disclosure are expressly incorporated herein by reference and are intended to be included in the claims.

Claims

1. A method for decoding video data, comprising: Get video chunks from the bitstream; Obtaining an internal luminance sample value of the video block, an external luminance sample value of an external area of ​​the video block, and an external chrominance sample value of the external area; determining a filter pattern based on the external luma sample values ​​and the external chroma sample values, wherein the filter pattern comprises one or more linear weighting coefficients for corresponding one or more pre-computed luma sample values, and wherein the one or more pre-computed luma sample values ​​comprise at least one pre-computed luma sample value obtained based on a sample difference between external luma sample values; predicting intra chroma sample values ​​of the video block based on the intra luma sample values ​​using the filter mode; and A decoded video block is obtained using the predicted intra chroma sample values.

2. The method according to claim 1, wherein: The filter pattern also includes coefficients for non-linear values ​​and coefficients for bias values.

3. The method according to claim 1, wherein: The one or more pre-calculated luma sample values ​​further include at least one pre-calculated luma sample value obtained by weighted average calculation.

4. The method according to claim 1, wherein: The one or more pre-operated luma sample values ​​further include at least one pre-operated luma sample value obtained by applying one of a low pass filter, a high pass filter, or a non-linear filter.

5. The method according to claim 2, wherein: The non-linear value comprises a square of a corresponding pre-computed luma sample value obtained by a weighted average operation, and wherein the non-linear value is scaled to the luma sample value range of the video block.

6. The method according to claim 2, wherein: The non-linear value includes a value obtained by scaling the square of the corresponding pre-calculated luma sample value obtained through a weighted average operation and a sum of intermediate chroma values ​​corresponding to the luma sample value range of the video block.

7. The method according to claim 2, wherein: The non-linear value comprises the square of the corresponding pre-computed luma sample value obtained based on a sample difference between luma sample values, and wherein the non-linear value is scaled to the luma sample value range of the video block.

8. The method according to claim 2, wherein: The non-linear value includes a value obtained by scaling the square of a corresponding pre-calculated luma sample value obtained based on a sample difference between luma sample values ​​and a sum of intermediate chroma values ​​corresponding to the luma sample value range of the video block.

9. The method according to claim 1, wherein: The filter pattern comprises two or more linear weighting coefficients for corresponding two or more pre-operated luma sample values, and wherein: At least two of the two or more pre-computed luma sample values ​​correspond to the same luma position and are obtained using the same or different filters; or At least two of the two or more pre-operated luma sample values ​​correspond to different luma positions and are obtained using the same or different filters.

10. The method according to claim 1, wherein: At least one of the non-linear value and the one or more pre-computed luma sample values ​​corresponds to the same or different luma locations.

11. The method according to claim 1, wherein: One or more of the linear weighting coefficients are switched for different coding units (CUs) or between different chroma components.

12. A method for encoding video data, comprising: Get the video block; Obtaining an internal luminance sample value of the video block, an external luminance sample value of an external area of ​​the video block, and an external chrominance sample value of the external area; determining a filter pattern based on the external luma sample values ​​and the external chroma sample values, wherein the filter pattern comprises one or more linear weighting coefficients for corresponding one or more pre-computed luma sample values, and wherein the one or more pre-computed luma sample values ​​comprise at least one pre-computed luma sample value obtained based on a sample difference between external luma sample values; predicting intra chroma sample values ​​of the video block based on the intra luma sample values ​​using the filter mode; and A bitstream including the encoded video block is generated by using the predicted intra chroma sample values.

13. The method according to claim 12, wherein: The filter pattern also includes coefficients for non-linear values ​​and coefficients for bias values.

14. The method according to claim 12, wherein: The one or more pre-calculated luma sample values ​​further include at least one pre-calculated luma sample value obtained by weighted average calculation.

15. The method according to claim 12, wherein: The one or more pre-operated luma sample values ​​further include at least one pre-operated luma sample value obtained by applying one of a low pass filter, a high pass filter, or a non-linear filter.

16. The method according to claim 13, wherein: The non-linear value comprises a square of a corresponding pre-computed luma sample value obtained by a weighted average operation, and wherein the non-linear value is scaled to the luma sample value range of the video block.

17. The method according to claim 13, wherein: The non-linear value includes a value obtained by scaling the square of the corresponding pre-calculated luma sample value obtained through a weighted average operation and a sum of intermediate chroma values ​​corresponding to the luma sample value range of the video block.

18. The method according to claim 13, wherein: The non-linear value comprises the square of the corresponding pre-computed luma sample value obtained based on a sample difference between luma sample values, and wherein the non-linear value is scaled to the luma sample value range of the video block.

19. The method according to claim 13, wherein: The non-linear value includes a value obtained by scaling the square of a corresponding pre-calculated luma sample value obtained based on a sample difference between luma sample values ​​and a sum of intermediate chroma values ​​corresponding to the luma sample value range of the video block.

20. The method according to claim 12, wherein: The filter pattern comprises two or more linear weighting coefficients for corresponding two or more pre-operated luma sample values, and wherein: At least two of the two or more pre-computed luma sample values ​​correspond to the same luma position and are obtained using the same or different filters; or At least two of the two or more pre-operated luma sample values ​​correspond to different luma positions and are obtained using the same or different filters.

21. The method according to claim 12, wherein: At least one of the non-linear value and the one or more pre-computed luma sample values ​​corresponds to the same or different luma locations.

22. The method according to claim 12, wherein: One or more of the linear weighting coefficients are switched for different coding units (CUs) or between different chroma components.

23. A computer system comprising: one or more processors; as well as One or more storage devices storing computer-executable instructions that, when executed, cause the one or more processors to perform the operations of the method of any one of claims 1-22.

24. A computer program product storing computer executable instructions which, when executed, cause one or more processors to perform the operations of the method according to any one of claims 1-22.

25. A computer-readable medium storing computer-executable instructions which, when executed, cause one or more processors to receive a bitstream and perform the operations of the method according to any one of claims 1-11 based on the bitstream.

26. A computer-readable medium storing computer-executable instructions that, when executed, cause one or more processors to perform the operations of the method of any one of claims 12-22 and to send a bitstream including encoded video information associated with the predicted chroma samples.

27. A computer-readable medium storing a bit stream, wherein: The bitstream is to be decoded by performing the operations of the method according to any one of claims 1-11.

28. A computer-readable medium storing a bit stream, wherein: The bitstream is obtained by performing the operations of the method according to any one of claims 12-22.

Images