Method and apparatus for hybrid cross-component prediction in video coding systems

Abstract

A method and apparatus for inter-picture CCLM prediction. According to the method, one or more model parameters of one or more cross-color models associated with the first color block and the second color block are determined. A cross-component predictor for the second color block is derived by applying one or more cross-color models to reconstructed or predicted first color pixels corresponding to the first color block. A final predictor for the second color block is derived by using the cross-component predictor or combining the cross-component predictor with a target mode predictor for the second color block. And encoding or decoding the second color block by using prediction data including the final predictor.

Description

[0001] Cross-references

[0002] This application is a non-provisional application based on and claims priority to U.S. Provisional Patent Application No. 63 / 352,343, filed on June 15, 2022. The entire contents of the U.S. Provisional Patent Application are incorporated herein by reference. Technical Field

[0003] The present invention relates to a video coding system, and more particularly to a new video coding tool in the video coding system by mixing a cross-component linear model predictor and an inter-picture mode predictor. Background Art

[0004] Versatile Video Coding (VVC) is the latest international video codec standard developed by the Joint Video Experts Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). The standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Information technology - Codecs for immersive media presentation - Part 3: Versatile video coding, which was published in February 2021. VVC is developed on the basis of its predecessor HEVC (High Efficiency Video Coding) by adding more codec tools to improve codec efficiency and handle various types of video sources including 3-Dimensional (3D) video signals.

[0005] Figure 1AAn exemplary adaptive inter-picture (Inter) / intra-picture (Intra) video coding system including loop processing is shown. For intra-picture prediction, prediction data is derived based on previously encoded and decoded video data in the current picture. For inter-picture prediction 112, motion estimation (Motion Estimation, ME) is performed on the encoder side, and motion compensation (Motion Compensation, MC) is performed based on the results of ME to provide prediction data and motion data derived from other pictures. Switch 114 selects intra-picture prediction 110 or inter-picture prediction 112, and the selected prediction data is provided to adder 116 to form a prediction error, also called a residual. The prediction error is then processed by transform (Transform, T) 118 and quantization (Quantization, Q) 120 in sequence. The transformed and quantized residual is then encoded and decoded by entropy encoder 122 to be included in the video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged together with side information such as motion mode and coding mode associated with intra-picture prediction and inter-picture prediction, and other information such as parameters associated with the loop filter applied to the underlying picture area. Figure 1A As shown, auxiliary information related to intra-picture prediction 110, inter-picture prediction 112 and loop filter 130 is provided to entropy encoder 122. When inter-picture prediction mode is used, one or more reference pictures must also be reconstructed at the encoder end. Therefore, the transformed and quantized residual is processed by inverse quantization (IQ) 124 and inverse transformation (IT) 126 to recover the residual. Then, at reconstruction (REC) 128, the residual is added back to the prediction data 136 to reconstruct the video data. The reconstructed video data can be stored in the reference picture buffer 134 and used for prediction of other frames.

[0006] like Figure 1AAs shown, the incoming video data undergoes a series of processes in the encoding system. The reconstructed video data from REC 128 may suffer various losses due to the series of processes. Therefore, in order to improve the video quality, a loop filter 130 is usually applied to the reconstructed video data before the reconstructed video data is stored in the reference picture buffer 134. For example, a deblocking filter (DF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF) may be used. The loop filter information may need to be incorporated into the bitstream so that the decoder can properly recover the required information. Therefore, the loop filter information is also provided to the entropy encoder 122 for incorporation into the bitstream. Figure 1A In , a loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in a reference picture buffer 134 . Figure 1A The system in is intended to illustrate an exemplary structure of a typical video encoder. It may correspond to a High Efficiency Video Coding (HEVC) system, VP8, VP9, ​​H.264, VVC, or any other video codec standard.

[0007] like Figure 1B As shown, since the decoder only needs inverse quantization 124 and inverse transform 126, the decoder can use similar or partially identical functional blocks as the encoder except for transform 118 and quantization 120. The decoder uses an entropy decoder 140 instead of an entropy encoder 122 to decode the video bitstream into quantized transform coefficients and required coding information (e.g., ILPF information, intra-picture prediction information, and inter-picture prediction information). The intra-picture prediction 150 on the decoder side does not need to perform a pattern search. Instead, the decoder only needs to generate an intra-picture prediction based on the intra-picture prediction information received from the entropy decoder 140. In addition, for inter-picture prediction, the decoder only needs to perform motion compensation 152 based on the inter-picture prediction information received from the entropy decoder 140, without the need for motion estimation.

[0008] According to VVC, similar to HEVC, the input picture is divided into a plurality of non-overlapping square block areas, called coding tree units (CTU). Each CTU can be divided into one or more coding units (CU) of smaller size. The resulting CU segmentation can be square or rectangular. At the same time, VVC divides the CTU into a plurality of prediction units (PU) as units to apply prediction processing, such as inter-picture prediction, intra-picture prediction, etc.

[0009] The present invention proposes various new coding tools to improve the coding efficiency beyond VVC. In particular, a new coding tool involves inter-picture CCLM, which can mix cross-color LM predictors and non-intra-picture mode predictors. Summary of the invention

[0010] The present invention discloses a method and device for prediction in a video coding system. According to the method, input data related to a current block including a first color block and a second color block is received, wherein the input data includes pixel data of the current block to be encoded on the encoder side or encoded data related to the current block to be decoded on the decoder side, and the first color block is encoded and decoded in a target mode, and the target mode refers to an inter-picture mode or an intra-picture block copy mode. One or more model parameters of one or more cross-color models related to the first color block and the second color block are determined. A cross-component predictor of the second color block is derived by applying one or more cross-color models to corresponding reconstructed or predicted first color pixels of the first color block. A final predictor of the second color block is derived by using the cross-component predictor or combining the cross-component predictor with the target mode predictor of the second color block. The second color block is encoded or decoded by using the prediction data including the final predictor.

[0011] In one embodiment, one or more model parameters of one or more cross-color models are derived by using neighboring reconstructed first color samples of the first color block and neighboring reconstructed second color samples of the second color block. In another embodiment, one or more model parameters for one or more cross-color models are derived by using neighboring predicted first color samples of the first color block and neighboring predicted second color samples of the second color block.

[0012] In one embodiment, the cross-component predictor is selected from the cross-component mode set. For example, the cross-component mode set may include a combination of all or any subsets of CCLM_LT mode, CCLM_L mode, CCLM_T mode, MMLM_LT mode, MMLM_L mode and MMLM_T mode. The cross-component predictor can be selected from the cross-component mode set according to an implicit rule. In one embodiment, the implicit rule is related to the block width, block height or block area of ​​the current block. In another embodiment, the implicit rule is inferred to be predefined. In yet another embodiment, the cross-component predictor is selected according to one or more explicit indexes.

[0013] In one embodiment, the target mode predictor is derived using Combined Inter Merge and Intra Prediction (CIIP), CIIP with Template Matching (CIIP-TM), or CIIP with Position Dependent Intra Prediction Combination (CIIP-PDPC).

[0014] In one embodiment, the first color block corresponds to a luma block and the second color block corresponds to a chroma block.

[0015] In one embodiment, the final predictor for the second color block is derived using a weighted sum of the cross-component predictor and the target mode predictor. In one embodiment, one or more weights of the weighted sum of the cross-component predictor and the target mode predictor are selected based on the coding mode of one or more neighboring blocks of the current block. The neighboring block may correspond to at least one of a top neighboring block and a left neighboring block.

[0016] In one embodiment, when encoding or decoding the current block in CIIP, one or more flags are indicated or parsed from the bitstream to indicate whether inter-picture CCLM processing is applied to the current block, wherein the inter-picture CCLM processing includes deriving a cross-component predictor, deriving a final predictor, and encoding or decoding a second color block by using prediction data including the final predictor. When the inter-picture CCLM processing is applied to the current block, the cross-component predictor is inferred to be derived according to CCLM_LT. In another embodiment, when the inter-picture CCLM processing is applied to the current block, the cross-component predictor is inferred to be a target cross-component mode with a minimum boundary matching cost in a set of candidate cross-component modes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1A An exemplary adaptive inter / intra video coding system including in-loop processing is shown.

[0018] Figure 1B Shows Figure 1A The decoder corresponding to the encoder in the figure.

[0019] Figure 2 An example of a directional (angle) mode for intra-screen prediction is shown.

[0020] Figure 3 An example of multiple reference line (MRL) intra-picture prediction is shown, where four reference lines are used for intra-picture prediction.

[0021] Figure 4A An example of Intra Sub-Partition (ISP) is shown, where a block is partitioned into two sub-blocks horizontally or vertically.

[0022] Figure 4B An example of intra-picture sub-partitioning (ISP) is shown, where a block is divided into four sub-blocks horizontally or vertically.

[0023] Figure 5 An example of a matrix weighted intra prediction (MIP) processing flow is shown.

[0024] Figure 6 The reference area of ​​the IBC mode is marked, where each block represents a 64x64 luma sample unit, and the reference area depends on the position of the currently decoded CU.

[0025] Figure 7 The relative sample positions of an M×N chroma block, a corresponding 2M×2N luma block, and their neighboring samples (shown as filled circles and filled triangles) for “type-0” content are shown.

[0026] Fig. 8A An example of a selected template for a current block is shown, where the template includes T rows above the current block and T columns to the left of the current block.

[0027] Figure 8B An example of T=3 is shown, and HoG (Histogram of Gradient) is calculated for the middle line pixels and the middle column pixels.

[0028] Figure 8C Examples of the amplitude (ampl) of the angular intra prediction modes are marked.

[0029] Fig. 9 An example of a blending process is shown where two angular intra-frame modes (M1 and M2) are selected based on the index having the two highest bins of the histogram bins.

[0030] Fig.10 An example of a Template-based Intra Mode Derivation (TIMD) mode is marked, where TIMD implicitly derives the intra prediction mode of a CU using neighboring templates at the encoder and decoder.

[0031] Fig.11 Neighboring blocks used to derive spatial merging candidates for VVC are shown.

[0032] Fig.12 Possible candidate pairs considered for redundancy checking in VVC are shown.

[0033] Fig.13 An example of temporal candidate derivation is shown, where scaled motion vectors are derived from Picture Order Count (POC) distances.

[0034] Fig.14 The position of the temporal candidate selected between candidate C0 and candidate C1 is shown.

[0035] Fig.15 The distance offset from the starting MV in the horizontal and vertical directions according to the Merge Mode with MVD (MMVD) is marked.

[0036] Fig.16A An example of an affine motion field of a block described by motion information of two control points (4 parameters) is shown.

[0037] Fig. 16B An example of an affine motion field of a block described by motion information of three control point motion vectors (6 parameters) is shown.

[0038] Fig.17 An example of block-based affine transform prediction is shown, where the motion vector of each 4×4 luma sub-block is derived from the control point MV.

[0039] Fig.18 An example of deriving inherited affine candidates based on control point MVs of neighboring blocks is shown.

[0040] Fig.19 An example of affine candidate construction by combining the translational motion information of each control point from both spatial and temporal neighbors is shown.

[0041] Fig. 20 An example of affine motion information storage for motion information inheritance is marked.

[0042] Fig.21 An example of derivation of weight values ​​for Combined Inter and Intra Prediction (CIIP) according to coding modes of top neighboring blocks and left neighboring blocks is shown.

[0043] Fig. 22 An example of 64 partitions used in the VVC standard is shown, where the partitions are grouped according to their angles and dashed lines indicate redundant partitions.

[0044] Fig.23 An example of uni-predictive MV selection for geometric partitioning mode is marked.

[0045] Fig.24 An example of blend weight_0 using geometric split mode is marked.

[0046] Fig.25 An example of a GPM blending process based on a discrete ramp function for the blending region around the boundary is shown.

[0047] Fig.26 An example of a GPM mixing process for GPM mixing in ECM 4.0 is shown.

[0048] Fig.27A -C show examples of available IPM candidates: parallel angle patterns for GPM block boundaries (parallel patterns, Fig.27A ), vertical angle mode for GPM block boundaries (vertical mode, Fig.27B ) and planar mode ( Fig.27C ).

[0049] Fig.27D An example of a GPM with intra-picture prediction and inter-picture prediction is shown, where the intra-picture prediction is restricted to reduce the signaling overhead of the IPM and the hardware decoder cost.

[0050] Fig.28A The syntax encoding and decoding for Spatial GPM (SGPM) before using the simplified method is shown.

[0051] Fig.28B An example of simplified syntax encoding and decoding for Spatial GPM (SGPM) is shown.

[0052] Fig.29 An example of a template for a Spatial GPM (SGPM) is shown.

[0053] Fig.30 Templates for luma and chroma are shown to derive model parameters and examples of template matching distortion.

[0054] Fig.31 An example of deriving predictions of Cb and Cr based on predicted samples of Y using a prediction-based CCLM according to an embodiment of the present invention is shown.

[0055] Fig.32 An example of the relationship between cr prediction, cb prediction and JCCLM predictor is shown.

[0056] Fig.33An example of adaptive intra-picture mode selection is shown, where the chroma block is divided into 4 sub-blocks.

[0057] Fig.34A -C shows some possible ways of partitioning the current block and the selection of weights for prediction from the CCLMs associated with these partitions.

[0058] Fig.35 An example of boundary samples used to derive boundary matching costs is shown.

[0059] Fig.36 An example of cross-CU LM ​​is shown, where the blocks have irregular patterns and no angular intra prediction can provide good prediction.

[0060] Fig.37 It is shown that a luma picture region associated with a node contains an irregular pattern and the picture region is divided into blocks for applying inter-picture or intra-picture prediction.

[0061] Fig.38A -B specifies the use of LM mode within the current block ( Fig.38A ) or outside the current block ( Fig.38B ) generates an example of the lower right region.

[0062] Fig.39 A flow chart of a video encoding and decoding system that mixes a linear model predictor with an inter-picture mode predictor according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0063] It is readily understood that, as generally described and illustrated in the accompanying drawings herein, the components of the present invention may be arranged and designed in a variety of different configurations. As shown, the following detailed description of the embodiments of the present system and method is not intended to limit the scope of the present invention as required, but is merely representative of the embodiments selected by the present invention. In the specification, reference to "one embodiment", "embodiment" or similar language means that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing in various places in the specification do not necessarily all refer to the same embodiment.

[0064] In addition, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. However, those of ordinary skill in the art will recognize that the present invention may be implemented without one or more specific details, or using other methods, components, etc. In other cases, in order to avoid confusing the various aspects of the present invention, known structures or operations are not shown or described in detail. The illustrated embodiments of the present invention will be best understood by reference to the accompanying drawings, in which the same parts are represented by the same numbers throughout. The following description is provided by way of example only, and simply illustrates certain selected embodiments of the apparatus and methods consistent with the claimed invention.

[0065] The VVC standard includes various new coding tools to further improve coding efficiency compared to the HEVC standard. Among the various new coding tools, some coding tools related to the present invention are reviewed below.

[0066] Intra-mode codec with 67 intra-prediction modes

[0067] To capture arbitrary edge directions present in natural video, the number of directional intra-frame modes in VVC is expanded from 33 used in HEVC to 65. New directional (angle) modes not in HEVC such as Figure 2 As shown by the red dashed arrows in , the planar mode and DC mode remain unchanged. These more dense directional intra prediction modes are applied to all block sizes and luma intra prediction and chroma intra prediction.

[0068] In order to keep the complexity of the Most Probable Mode (MPM) list generation low, an intra-mode coding method with 6 MPMs is used by considering two available adjacent intra-modes. The following three aspects are considered to construct the MPM list:

[0069] –Default in-frame mode

[0070] – Adjacent Intra-frame Mode

[0071] – In-frame mode derived

[0072] Multi-reference line intra-frame prediction

[0073] Multiple Reference Line (MRL) intra-picture prediction uses more reference lines for intra-picture prediction. Figure 3In , an example of 4 reference lines is described, where the samples of segments A and F are not taken from the reconstructed neighboring samples, but are filled with the closest samples from segments B and E, respectively. HEVC intra-picture prediction uses the nearest reference line (i.e., reference line 0). In MRL, 2 additional lines are used (reference line 1 and reference line 3)

[0074] The index of the selected reference line (mrl_idx) is indicated and used to generate the intra-picture predictor. For reference line idx greater than 0, only additional reference line modes are included in the MPM list, and only the mpm index is indicated, without the remaining modes. The reference line index is indicated before the intra-picture prediction mode, and if a non-zero reference line index is indicated, the planar mode is excluded from the intra-picture prediction model.

[0075] MRL is disabled for blocks of the first line within a CTU to prevent the use of extended reference samples outside the current CTU row. In addition, the Position-Dependent Prediction Combination (PDPC) is disabled when additional lines are used. For MRL mode, the derivation of the DC value in the DC intra-screen prediction mode for a non-zero reference line index is aligned with the derivation of the DC value in the DC intra-screen prediction mode for a reference line index of 0. MRL requires storage of 3 adjacent luminance reference lines of the CTU to generate predictions. The Cross-Component Linear Model (CCLM) tool also requires 3 adjacent luminance reference lines for its downsampling filter. The definition of MRL using the same 3 lines is consistent with CCLM to reduce the storage requirements of the decoder.

[0076] Intra-screen subdivision

[0077] Intra Sub-Partition (ISP) divides the luma intra prediction block vertically or horizontally into 2 or 4 sub-partitions depending on the block size. For example, the minimum block size of ISP is 4x8 (or 8x4). If the block size is larger than 4x8 (or 8x4), the corresponding block is divided by 4 sub-partitions. It has been noted that M×128 (with M≤64) ISP blocks and 128×N (with N≤64) ISP blocks can cause potential problems with a 64×64 Virtual Decoder Pipeline Unit (VDPU). For example, an M×128 CU in a single-tree case has M×128 luma TBs and two corresponding Chroma TB. If the CU uses ISP, the luma TB will be divided into four M×32TBs (only horizontal division is possible), each of which is smaller than a 64×64 block. However, in the current design of ISP, the chroma blocks are not divided. Therefore, the size of the two chroma components will be larger than 32×32 blocks. Similarly, using 128×NCU of ISP, similar scenarios can be created. Therefore, these two situations are problems of the 64×64 decoder pipeline. Based on this, the CU size that can use ISP is limited to a maximum value of 64×64. Figure 4A and Figure 4B Examples of both possibilities are shown. All sub-splits satisfy the condition of having at least 16 samples.

[0078] In ISP, 1xN and 2xN sub-block predictions are not allowed to rely on reconstructed values ​​of previously decoded 1xN and 2xN sub-blocks of the codec block, so that the minimum prediction width of the sub-block becomes four samples. For example, an 8xN (N>4) codec block encoded and decoded using ISP with vertical partitioning is divided into two prediction regions of size 4xN each and four transforms of size 2xN. In addition, a 4xN codec block encoded and decoded using ISP with vertical partitioning is predicted using a complete 4xN block; and four transforms of 1xN each are used. Although transform sizes of 1xN and 2xN are allowed, the transforms in these block 4xN regions can be performed in parallel. For example, when a 4xN prediction region contains four 1xN transforms, there is no transform in the horizontal direction; the transform in the vertical direction can be performed as a single 4xN transform in the vertical direction. Similarly, when a 4xN prediction region contains two 2xN transform blocks, the transform operations of the two 2xN blocks in each direction (horizontally and vertically) can be performed in parallel. Therefore, there is no added delay in processing these smaller blocks compared to processing intra-picture blocks for 4x4 conventional codecs.

[0079] For each sub-partition, the reconstructed samples are obtained by adding the residual signal to the prediction signal. Here, the residual signal is generated by processes such as entropy decoding, inverse quantization and inverse transformation. Therefore, the reconstructed sample values ​​of each sub-partition are available to generate the prediction of the next sub-partition, and each sub-partition is processed consecutively. In addition, the first sub-partition to be processed is the one containing the top left samples of the CU, and then continues to the one below (horizontal partition) or to the right (vertical partition). Therefore, the reference samples used to generate the prediction signal for the sub-partition are only located to the left and above the line. All sub-partitions share the same intra-screen mode.

[0080] Matrix Weighted Intra-frame Prediction

[0081] The Matrix Weighted Intra Prediction (MIP) method is a newly added intra prediction technique in VVC. For predicting samples of a rectangular block of width W and height H, MIP takes as input a line of H reconstructed adjacent boundary samples on the left side of the block and a line of W reconstructed adjacent boundary samples above the block. If reconstructed samples are not available, they are generated as done in traditional intra prediction. The generation of the prediction signal is based on the following three steps, namely, Figure 5 averaging, matrix-vector multiplication, and linear interpolation as shown. A line of H reconstructed adjacent boundary samples 512 located on the left side of the block and a line of W reconstructed adjacent boundary samples 510 located above the block are shown as dot-filled small squares. After the averaging process, the boundary samples are downsampled to the top boundary line 514 and the left boundary line 516. The downsampled samples are provided to the matrix-vector multiplication unit 520 to generate a downsampled prediction block 530. Then, the interpolation process is applied to generate a prediction block 540.

[0082] Averaging adjacent samples

[0083] Among the boundary samples, four samples or eight samples are selected by averaging based on the block size and shape. Specifically, the input boundary bdry is obtained by averaging adjacent boundary samples according to a predefined rule depending on the block size. top and input border bdry left Reduced to smaller boundaries and border Then, the two reduced boundaries and is concatenated into the reduced boundary vector bdry red , which is therefore four times the size for blocks of 4×4 shape and eight times the size for blocks of all other shapes. If mode refers to a MIP mode, the concatenation is defined as follows:

[0084]

[0085] Matrix Multiplication

[0086] Taking the average sample as input, a matrix-vector multiplication is performed, followed by the addition of an offset. The result is a reduced prediction signal for a subsampled set of samples in the original block. In addition to the reduced input vector bdry red , generating a reduced prediction signal pred red , which is the width W red and height H red Here, W red and H redis defined as:

[0087]

[0088]

[0089] The reduced prediction signal pred is calculated by computing the matrix-vector product and adding the offset red :

[0090] pred red =A.bdry red +b.

[0091] Here, A is a matrix with W red ·H red rows and 4 columns for W=H=4, and 8 columns for other cases. b is W red ·H red The matrix A and the offset vector b are taken from the set S 0 , S 1 , S 2 A set in . The index idx=idx(W,H) is defined as follows:

[0092]

[0093] Here, each coefficient of the matrix A is represented with 8 bits of precision. 0 Includes 16 matrices Each matrix has 16 rows and 4 columns, and 16 offset vectors Each offset vector has a size of 16. This set of matrices and offset vectors is for blocks of size 4×4. 1 Includes 8 matrices Each matrix is ​​16 rows and 8 columns, and 8 offset vectors The size of each offset vector is 16. The set S 2 Includes 6 matrices Each matrix is ​​64 rows and 8 columns, and 6 offset vectors The size of each offset vector is 64.

[0094] Interpolation

[0095] The prediction signals at the remaining positions are generated from the prediction signals for the subsampled set by linear interpolation, which is a single-step linear interpolation in each direction. Interpolation is performed first in the horizontal direction and then in the vertical direction, regardless of block shape or block size.

[0096] MIP mode signaling and coordination with other coding tools

[0097] For each Coding Unit (CU) in intra mode, a flag indicating whether the MIP mode is applied is sent. If the MIP mode is to be applied, the MIP mode (predModeIntra) is indicated. For the MIP mode, the transposed flag (isTransposed) that determines whether the mode is transposed and the MIP mode Id (modeId) that determines which matrix will be used for a given MIP mode are derived as follows.

[0098] isTransposed=predModeIntra&1

[0099] modeId=predModeIntra>>1

[0100] The MIP codec mode is coordinated with other codecs by taking into account the following aspects:

[0101] - Enable Low-Frequency Non-Separable Transform (LFNST) for MIP of larger blocks. Here, the planar mode LFNST transform is used

[0102] - Same as the traditional intra-picture prediction mode, perform MIP reference sample derivation

[0103] - For the upsampling step used in MIP prediction, use the original reference samples instead of the downsampled samples

[0104] - Perform clipping before upsampling, not after.

[0105] - Allow MIPs up to 64x64 regardless of the maximum transform size.

[0106] - The number of MIP modes is 32 for sizeld=0, 16 for sizeld=1, and 12 for sizeld=2.

[0107] In-screen block copy

[0108] Intra Block Copy (IBC) is a tool adopted in the HEVC extension of Screen Content Coding (SCC). As we all know, it significantly improves the coding efficiency of screen content materials. Since the IBC mode is implemented as a block-level coding mode, block matching (BM) is performed at the encoder to find the best block vector (or motion vector) for each CU. Here, the block vector is used to indicate the displacement from the current block to the reference block, which has been reconstructed in the current picture. The luminance block vector of the CU encoded and decoded by IBC is integer precision. The chrominance block vector is also rounded to integer precision. When combined with Adaptive Motion Vector Resolution (AMVR), the IBC mode can switch between 1-pixel motion vector precision and 4-pixel motion vector precision. The IBC mode for the CU encoded and decoded by IBC is regarded as a third prediction mode in addition to the intra-picture prediction mode or the inter-picture prediction mode. The IBC mode is applicable to CUs with width and height less than or equal to 64 luminance samples.

[0109] On the encoder side, hash-based motion estimation is performed for IBC. The encoder performs RD checks on blocks with width or height no greater than 16 luma samples. For non-merge mode, a hash-based search is used, with a block vector search performed first. If the hash search does not return a valid candidate, a local search based on block matching is performed.

[0110] In hash-based search, the hash key matching (32-bit CRC) between the current block and the reference blocks is extended to all allowed block sizes. The hash key calculation for each position in the current picture is based on 4×4 sub-blocks. For current blocks of larger sizes, when all hash keys of all 4×4 sub-blocks match the hash keys in the corresponding reference positions, it is decided that the hash key matches the hash key of the reference block. If the hash keys of multiple reference blocks are found to match the hash key of the current block, the block vector cost of each matching reference is calculated and the one with the minimum cost is selected.

[0111] In the block matching search, the search range is set to cover the previous CTU and the current CTU.

[0112] At CU level, IBC mode is indicated with a flag, and it can be indicated as IBC AMVP (Advanced Motion Vector Prediction) mode or IBC skip / merge mode, as follows:

[0113] – IBC skip / merge mode: The merge candidate index is used to indicate which block vector from the list of neighboring candidate IBC codec blocks is used to predict the current block. The merge list includes spatial candidates, HMVP (History based Motion Vector Prediction) candidates, and paired candidates.

[0114] – IBC AMVP mode: Block vector differences are encoded and decoded in the same way as motion vector differences. The block vector prediction method uses two candidates as predictors, one from the left neighbor and one from the top neighbor (if IBC coded). When neither neighbor is available, the default block vector is used as the predictor. A flag is signaled to indicate the block vector predictor index.

[0115] IBC Reference Area

[0116] To reduce memory consumption and decoder complexity, IBC in VVC allows only the reconstructed portion of a predefined area including the area of ​​the current CTU and some areas of the left CTU. Figure 6 The reference area for IBC mode is shown, where each block represents a 64x64 luma sample unit. Depending on the location of the currently encoded CU within the current CTU, the following applies:

[0117] – Using the Current Picture Referencing (CPR) mode, if the current block falls into the upper left 64x64 block of the current CTU ( Figure 6 In case 610 in the figure), in addition to the reconstructed samples in the current CTU, it can also refer to the reference samples in the lower right 64x64 block of the left CTU. (For more details of CPR, please refer to JVET-T2002 (JianleChen, et.al., "Algorithm description for Versatile Video Coding and Test Model11 (VTM 11)", Joint Video Experts Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC1 / SC 29, 20th Meeting, by teleconference, 7–16 October 2020, Document: JVET-T2002)). Using the CPR mode, the current block can also refer to the reference samples in the lower left 64x64 block of the left CTU and the reference samples in the upper right 64x64 block of the left CTU.

[0118] – Using CPR mode, if the current block falls into the upper right 64x64 block of the current CTU ( Figure 6 In case 620 in the figure, in addition to the samples that have been reconstructed in the current CTU, if the brightness position (0, 64) relative to the current CTU has not been reconstructed, the current block can also refer to the reference samples in the lower left 64x64 block and the lower right 64x64 block of the left CTU; otherwise, the current block can also refer to the reference samples in the lower right 64x64 block of the left CTU.

[0119] – Using CPR mode, if the current block falls into the lower left 64x64 block of the current CTU ( Figure 6 In case 630 in the example, in addition to the samples that have been reconstructed in the current CTU, if the luma position (64, 0) relative to the current CTU has not been reconstructed, the current block can also refer to the reference samples in the upper right 64x64 block of the left CTU and the lower right 64x64 block. Otherwise, using the CPR mode, the current block can also refer to the reference samples in the lower right 64x64 block of the left CTU.

[0120] – Using CPR mode, if the current block falls into the lower right 64x64 block of the current CTU ( Figure 6 In case 640), it can only refer to the samples that have been reconstructed in the current CTU.

[0121] This restriction allows the IBC mode to be implemented using local on-chip memory implemented in hardware.

[0122] Joint Coding and Decoding of Chroma Residual

[0123] VVC supports the Joint Coding of Chroma Residual (JCCR) tool, in which the chroma residual is jointly encoded and decoded. The use (activation) of the JCCR mode is indicated by the TU level flag TU_joint_cbcr_recurial_flag, and the selected mode is implicitly indicated by the chroma CBF. If one or both of the chroma CBFs of the TU are equal to 1, the flag tu_joint_ccr_residual_flag exists. In the PPS (Picture Parameter Set) and slice header, the chroma QP offset values ​​are marked for the JCCR mode to distinguish them from the regular chroma QP offset values ​​marked for the regular chroma residual coding and decoding mode. These chroma QP offset values ​​are used to derive the chroma QP values ​​of some blocks encoded and decoded using the JCCR mode. The JCCR mode has 3 sub-modes. When the corresponding JCCR submode (submode 2 in Table 1) is enabled in a TU, the chroma QP offset is added to the luma-derived chroma QP applied during quantization and decoding of the TU. For other JCCR submodes (submode 1 and submode 3 in Table 1), the chroma QP is derived in the same way as for a conventional Cb block or Cr block. The reconstruction process of the chroma residuals (resCb and resCr) from the transmitted transform blocks is shown in Table 1. When the JCCR mode is enabled, a single joint chroma residual block (resJointC[x][y] in Table 1) is indicated, and considering information such as tu_cbf_Cb, tu_cbf_Cr and CSign, which are symbol values ​​specified in the slice header, the residual block of Cb (resCb) and the residual block of Cr (resCr) are derived.

[0124] At the encoder side, the joint chroma components are derived as explained below. Depending on the mode (listed in the table), resJointC{1,2} is generated by the encoder as follows:

[0125] - If mode is equal to 2 (single residual with reconstruction Cb=C, Cr=CSign*C), the joint residual is determined as follows.

[0126] resJointC[x][y]=(resCb[x][y]+CSign*resCr[x][y=) / 2

[0127] - Otherwise, if mode is equal to 1 (single residual with reconstruction Cb=C, Cr=(CSign*C) / 2), the joint residual is determined as follows.

[0128] resJointC[x][y]=(4*resCb[x][y=2*CSign*resCr[x][y]) / 5

[0129] – Otherwise (mode equals 3, i.e. single residual, reconstruction Cr=C, Cb=(CSign*C) / 2), determine the joint residual as follows.

[0130] resJointC[x][y]=(4*resCr[x][y=2*CSign*resCb[x][y]) / 5

[0131] Table 1 Reconstruction of the chroma residual. The value CSign is the sign value (+1 or -1) specified in the slice header, and resJointC is the transmitted residual.

[0132]

[0133]

[0134] The above three joint chroma coding sub-modes are only supported in I slices. In P slices and B slices, only mode 2 is supported. Therefore, in P slices and B slices, the syntax element tu_joint_cbcr_recurial_flag exists only when both chroma cbfs are 1.

[0135] The JCCR mode can be combined with the Chroma Transform Skip (TS) mode (see Section 3.9.3 of JVET-T2002 for more details on the TS mode). To speed up the encoder decision, the JCCR transform selection depends on whether the independent codecs of the Cb and Cr components select DCT-2 or TS as the best transform, and whether there are non-zero coefficients in the independent chroma codecs. Specifically, if one chroma component selects DCT-2 (or TS) and the other component is all zero, or if both chroma components select DCT-2 (or TS), only DCT-2 will be considered in JCCR encoding. Otherwise, if one component selects DCT-2 and the other selects TS, both DCT-2 and TS will be considered in JCCR encoding.

[0136] Cross Component Linear Model (CCLM)

[0137] The main idea behind the CCLM mode (sometimes abbreviated to LM mode) is as follows: a linear model whose parameters are derived from the reconstructed luma and chroma samples adjacent to the block whose chroma components are predicted from the co-located reconstructed luma samples.

[0138] In VVC, the CCLM mode exploits inter-channel correlation by predicting chrominance samples from reconstructed luma samples. The prediction is implemented using a linear model of the following form:

[0139] P(i,j)=a·rec′ L (i,j)+b. (1)

[0140] Among them, P(i,j) represents the predicted chrominance sample in CU, rec′ L (i, j) represents the reconstructed luma samples of the same CU downsampled in case of non-4:4:4 color format. The model parameters a and b are derived based on the reconstructed neighboring luma and chroma samples at the encoder and decoder side without explicit instructions.

[0141] Three CCLM modes are specified in VVC, namely CCLM_LT, CCLM_L and CCLM_T. These three modes differ in the location of the reference samples used for model parameter derivation. Only samples from the top boundary are involved in the CCLM_T mode, while only samples from the left boundary are involved in the CCLM_L mode. In the CCLM_LT mode, samples from the top boundary and the left boundary are used.

[0142] In general, the forecasting process of CCLM model consists of three steps:

[0143] 1) downsample the luma block and its adjacent reconstructed samples to match the size of the corresponding chroma block,

[0144] 2) derivation of model parameters based on reconstructed neighboring samples, and

[0145] 3) Apply model formula (1) to generate chroma intra-picture prediction samples.

[0146] Down-sampling of the luma component: To match the chroma sample positions of a 4:2:0 or 4:2:2 color format video sequence, two types of down-sampling filters can be applied to the luma samples, which have a 2 to 1 down-sampling ratio in both the horizontal and vertical directions. These two filters correspond to "type-0" and "type-2" 4:2:0 chroma format content, respectively, and are obtained by the following formulas.

[0147]

[0148] Based on the SPS level flag information, the two-dimensional 6-tap (i.e., f 2 ) or 5 taps (i.e. f 1) filter is applied to luma samples in the current block and its neighboring luma samples. SPS level refers to sequence parameter set level. An exception occurs if the top line of the current block is a CTU boundary. In this case, a one-dimensional filter [1,2,1] / 4 is applied to the neighboring luma samples to avoid using more than one luma line above the CTU boundary.

[0149] Model parameter derivation process: The model parameters a and b from equation (1) are derived based on the reconstructed adjacent luminance and chrominance samples at the encoder and decoder sides to avoid the need for any labeling overhead. In the initial version of the CCLM mode, a linear minimum mean square error (LMMSE) estimator was used to derive the parameters. However, in the final design, only four samples are involved to reduce the computational complexity. Figure 7 The relative sample positions of an MxN chroma block 710, a corresponding 2Mx2N luma block 720, and their neighboring samples (as depicted by the filled circles and triangles) of "Type-0" content are shown.

[0150] exist Figure 7 In the example of FIG. 1 , four samples used in CCLM_LT mode are shown, which are marked by triangular shapes. They are located at the M / 4 position and the M·3 / 4 position of the top boundary, and at the N / 4 position and the N·3 / 4 position of the left boundary. In CCLM_T mode and CCLM_L mode, the top boundary and the left boundary are extended to the size of (M+N) samples, and the four samples used for model parameter derivation are located at the (M+N) / 8 position, (M+N)·3 / 8 position, (M+N)·5 / 8 position, and (M+N)·7 / 8 position.

[0151] Once four samples are selected, four comparison operations are used to determine the two minimum and two maximum brightness sample values. l Represents the average of the two maximum brightness sample values, X s represents the average of the two minimum brightness sample values. Similarly, let Y l and Y s Represents the average value of the corresponding chrominance sample value. Then, the linear model parameters are obtained according to the following formula:

[0152]

[0153] b=Y s -a·X s . (3)

[0155] In this formula, the division operation for calculating parameter a is implemented using a lookup table. In order to reduce the memory required to store the table, the diff value, which is the difference between the maximum and minimum values, and the parameter a are expressed by an exponential counting system. Here, the diff value is approximately a 4-bit significant part and an exponent. Therefore, the table of 1 / diff includes only 16 elements. Thus, this has the advantages of reducing the computational complexity and reducing the memory size required to store the table.

[0156] MMLM Overview

[0157] As the name indicates, the original CCLM mode uses a linear model to predict chrominance samples from the luma samples of the entire CU, while in MMLM (Multiple Model CCLM), there can be two models. In MMLM, the neighboring luma samples and neighboring chroma samples of the current block are classified into two groups, each of which is used for a training set to derive a linear model (i.e., a specific α and β are derived for a specific group). In addition, the samples of the current luma block are also classified based on the same classification rules used for the neighboring luma samples.

[0158] o Threshold is calculated as the average of adjacent reconstructed luminance samples. Rec′

[0159] Neighboring samples with L[x,y]<=Threshold are classified into group 1; and Rec′L[x,y]>

[0160] Neighboring samples of Threshold are classified into group 2.

[0161] o Accordingly, using the linear model, the prediction of chromaticity is obtained as follows:

[0162]

[0163] Chroma Derived Mode (DM) mode

[0164] For the chroma DM mode, the intra prediction mode of the corresponding (co-located) luma block covering the center position of the current chroma block is directly inherited.

[0165] Decoder Side Intra Mode Derivation (DIMD)

[0166] When DIMD is applied, two intra modes are derived from reconstructed neighboring samples and these two predictors are combined with the planar mode predictor using weights derived from the gradients. DIMD mode is used as an optional prediction mode and is always checked in high complexity RDO mode.

[0167] To implicitly derive the intra prediction mode of a block, texture gradient analysis is performed on the encoder and decoder side. The process starts with an empty Histogram of Gradients (HoG) with 65 entries corresponding to 65 angular modes. The magnitudes of these entries are determined during texture gradient analysis.

[0168] In the first step, DIMD takes T=3 columns and lines of samples from the left and top of the current block, respectively. This area is used as a reference for gradient-based intra prediction mode derivation.

[0169] In the second step, the horizontal Sobel filter and the vertical Sobel filter are applied to all 3×3 window positions, centered on the pixels in the template centerline. At each window position, the Sobel filter calculates the pure horizontal and vertical intensities as G x and G y . Then, the texture angle of the window is calculated as:

[0170] angle=arctan(G x / G y ), (4)

[0171] It can be converted into one of 65 angular intra prediction modes. Once the intra prediction mode index of the current window is derived as idx, the magnitude of its entry in HoG[idx] is updated by adding:

[0172] ampl=|G x |+|G y | (5)

[0173] Fig. 8A -C shows an example of HoG, which is calculated after applying the above operation to all pixel positions in the template. Fig. 8A An example of a selected template 820 for a current block 810 is shown. The template 820 includes T lines above the current block and T columns to the left of the current block. For intra-picture prediction of the current block, an area 830 above and to the left of the current block corresponds to a reconstructed area, and an area 840 below and to the right of the block corresponds to an unavailable area. Figure 8B An example is shown for T=3, and the HoG is calculated for the pixel in the middle line 860 and the pixel in the middle column 862. For example, for pixel 852, a 3x3 window 850 is used. Figure 8C An example of the amplitude (ampl) calculated by formula (16) based on the angle intra-screen prediction mode determined from formula (4) is shown.

[0174] Once the HoG is calculated, the indices with the two highest histogram bins are selected as the two implicitly derived intra-picture prediction modes for the block and further combined with the planar mode as the prediction for DIMD mode. Prediction fusion is applied as a weighted average of the above three predictors. For this purpose, the weight of the plane is fixed to 21 / 64 (~1 / 3). Then, the remaining weight of 43 / 64 (~2 / 3) is shared between the two HoG IPMs, proportional to the amplitude of their HoG bins. Fig. 9 An example of this mixing process is shown. Fig. 9 As shown, two intra-picture modes (M1 912 and M2 914) are selected based on the index with the two highest bars in the histogram bar 910. Three predictors (940, 942 and 944) are used to form a mixed prediction. The three predictors correspond to applying M1, M2 and the planar intra-picture mode (920, 922 and 924, respectively) to the reference pixel 930 to form the corresponding predictor. The three predictors are weighted by corresponding weight factors (ω1, ω2 and ω3) 950. The weighted predictors are added using an adder 952 to generate a mixed predictor 960.

[0175] Furthermore, two implicitly derived intra modes are included in the MPM list so that the DIMD process is performed before the MPM list is constructed. The primary derived intra mode of a DIMD block is stored with the block and used for MPM list construction of neighboring blocks.

[0176] Template-based Intra Mode Derivation (TIMD)

[0177] TIMD mode uses neighboring samples at the encoder and decoder to implicitly derive the intra prediction mode of a CU, rather than signaling the intra prediction model to the decoder. Fig.10 As shown, the prediction samples of the template (1012 and 1014) of the current block 1010 are generated using the reference samples (1020 and 1022) of the template of each candidate mode. The cost is calculated as the SATD (Sum of Absolute Transformed Differences) between the prediction samples and the reconstructed samples of the template. The intra-screen prediction mode with the smallest cost is selected as the DIMD mode and used for the intra-screen prediction of the CU. The candidate mode can be 67 intra-screen prediction modes as in VVC, or can be extended to 131 intra-screen prediction modes. Typically, MPM can provide clues to indicate the directional information of the CU. Therefore, in order to reduce the intra-screen mode search space and utilize the characteristics of the CU, the intra-screen prediction mode can be implicitly derived from the MPM list.

[0178] For each intra-picture prediction mode in the MPM, the SATD between the predicted and reconstructed samples of the template is calculated. The first two intra-picture prediction modes with the smallest SATD are selected as TIMD modes. After applying the PDPC process, the two TIMD modes are fused with weights, and the weighted intra-picture prediction is used to encode and decode the current CU. Position Dependent intra Prediction Combination (PDPC) is included in the derivation of the TIMD mode.

[0179] The costs of the two selection modes are compared to the threshold. In the test, a cost factor of 2 is applied as follows:

[0180] costMode2<2*costMode1.

[0181] If this condition holds, then fusion is applied, otherwise only mode 1 is used. The weight of the modes is calculated based on their SATD costs as follows:

[0182] weight1=costMode2 / (costMode1+costMode2)

[0183] weight2=1-weight1.

[0184] Inter-picture prediction overview

[0185] According to JVET-T2002 Section 3.4 (Jianle Chen, et.al., “Algorithm description for Versatile Video Coding and Test Model 11 (VTM 11), Joint Video Experts Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29, 20th Meeting, bytle conference, 7–16 October 2020, Document: JVET-T2002)), for each inter-predicted CU, the motion parameters include motion vector, reference picture index and reference picture list usage index, as well as additional information for inter-prediction sample generation required by the new coding and decoding features of VVC. Motion parameters can be indicated explicitly or implicitly. When a CU is encoded in skip mode, the CU is associated with one PU and has no valid residual coefficients, no coded motion vector delta or reference picture index. The merge mode is specified based on the motion parameters of the current CU, which are obtained from neighboring CUs, including spatial and temporal candidates, and additional scheduling introduced in VVC. The merge mode can be applied to any inter-predicted CU, not just for skip mode. An alternative to the merge mode is the explicit transmission of motion parameters, where the motion vector, the corresponding reference picture index of each reference picture list and the reference picture list usage flag, as well as other required information, are explicitly indicated per CU.

[0186] In addition to the inter-picture codec features in HEVC, VVC also includes many new and refined inter-picture prediction codec tools, listed below:

[0187] – Extended Consolidated Forecast

[0188] –Merge mode with MVD (MMVD)

[0189] –Symmetric MVD (SMVD) marking

[0190] – Affine motion compensated prediction

[0191] – Subblock-based temporal motion vector prediction (SbTMVP)

[0192] –Adaptive motion vector resolution (AMVR)

[0193] – Motion field storage: 1 / 16th brightness sample MV storage and 8x8 motion field compression

[0194] – Bi-prediction with CU-level weight (BCW)

[0195] – Bi-directional optical flow (BDOF)

[0196] –Decoder side motion vector refinement (DMVR)

[0197] –Geometric partitioning mode (GPM)

[0198] – Combined inter and intra prediction (CIIP)

[0199] The following description provides details of those inter-picture prediction methods specified in VVC.

[0200] Extended Merge Prediction

[0201] In VVC, a merge candidate list is constructed by including the following five types of candidates in order:

[0202] 1) Spatial MVP from spatially adjacent CUs

[0203] 2) Time MVP from the same CU

[0204] 3) History-based MVP from FIFO table

[0205] 4) Paired Average MVP

[0206] 5) Zero MV.

[0207] The size of the merge list is indicated in the Sequence Parameter Set (SPS) header, and the maximum allowed size of the merge list is 6. For each CU encoded in merge mode, the index of the best merge candidate is encoded using truncated unary (TU). The first bin of the merge index is encoded using context, and bypass encoding is used for the remaining bins.

[0208] The derivation process for each category of merge candidates is provided in this section. As performed in HEVC, VVC also supports parallel derivation of merge candidate lists (or merge candidate lists) for all CUs within a certain size region.

[0209] Spatial candidate derivation

[0210] The derivation of spatial merge candidates in VVC is the same as that in HEVC, except that the positions of the first two merge candidates are swapped. Fig.11 Select up to four merge candidates (B) of the current CU 1110 from the candidates at the positions shown. 0 , A 0 , B 1 and A 1 ). The order of derivation is B 0 , A 0 , B 1 , A 1 and B 2 Position B2 is considered only when one or more neighboring CUs of positions B0, A0, B1, A1 are not available (for example, belong to another slice or tile) or are intra-coded. After the candidate at position A1 is added, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving the coding and decoding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the above redundancy check. Instead, only Fig.12 Pairs linked with arrows in . If the corresponding candidate for redundancy check does not have the same motion information, the candidate is only added to the list.

[0211] Time candidate derivation

[0212] In this step, only one candidate is added to the list. Specifically, when deriving the temporal merge candidate for the current CU 1310, based on Fig.13 As shown, the scaled motion vector is derived for the co-located CU 1320 belonging to the co-located reference picture. The reference picture list and reference index used for the derivation of the co-located CU are explicitly indicated in the slice header. Fig.13 As shown by the dotted line in , the scaled motion vector 1330 of the temporal merge candidate is obtained, which is scaled from the motion vector 1340 of the co-located CU using the POC (Picture Order Count) distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero.

[0213] like Fig.14 As shown, the position of the temporal candidate is selected between candidate C0 and candidate C1. If the CU at position C0 is not available, is intra-coded, or is outside the current row of the CTU, position C1 is used. Otherwise, position C0 is used in the derivation of the temporal merge candidate.

[0214] History-based Merge Candidate Derivation

[0215] History-based MVP (HMVP) merge candidates are added to the merge list after spatial MVP and TMVP. In this method, the motion information of the previous codec block is stored in a table and used as the MVP of the current CU. During the encoding / decoding process, a table with multiple HMVP candidates is retained. When a new CTU row is encountered, the table is reset (cleared). Whenever there is a non-sub-block inter-codec CU, the relevant motion information is added to the last entry of the table as a new HMVP candidate.

[0216] The HMVP table size S is set to 6, which indicates that up to 5 history-based MVP (HMVP) candidates can be added to the table. When a new motion candidate is inserted into the table, a constrained first-in-first-out (FIFO) rule is used, where a redundancy check is first applied to find out if there is an identical HMVP in the table. If found, the identical HMVP is deleted from the table, and then all subsequent HMVP candidates are moved forward, and the identical HMVP is inserted into the last entry of the table.

[0217] HMVP candidates can be used in the merge candidate list construction process. Check the nearest few HMVP candidates in the table in order and insert them into the candidate list after the TMVP candidate. Apply redundancy check to HMVP candidates to spatial or temporal merge candidates.

[0218] To reduce the number of redundant,check operations, the following simplifications are introduced:

[0219] 1. Check the redundancy of the last two entries in the table with respect to the A1 spatial candidate and the B1 spatial candidate respectively.

[0220] 2. Once the total number of available merge candidates reaches the maximum allowed number of merge candidates minus 1, the merge candidate list construction process from HMVP is terminated.

[0221] Pairwise average merge candidate derivation

[0222] Using the first two merge candidates, a pairwise average candidate is generated by averaging the predefined candidate pairs in the existing merge candidate lists. The first merge candidate is defined as p0Cand, and the second merge candidate can be defined as p1Cand. The average motion vector is calculated based on the availability of motion vectors for p0Cand and p1Cand, respectively, for each reference list. If both motion vectors are available in one list, they are averaged even if they point to different reference pictures, and its reference picture is set to the one corresponding to p0Cand. If only one motion vector is available, it is used directly; if no motion vector is available, the list is kept invalid. In addition, if the half-pixel interpolation filter index of p0Cand and p1Cand is different, it is set to 0.

[0223] When the merge list is not full after adding pairwise average merge candidates, zero MVPs are inserted at the end until the maximum number of merge candidates is encountered.

[0224] Merge estimated areas

[0225] The merge estimation region (MER) allows independent derivation of the merge candidate list of a CU in the same merge estimation region. Candidate blocks in the same MER as the current CU are not included in the generation of the merge candidate list of the current CU. In addition, the update process of the history-based motion vector prediction sub-candidate list is only updated when (xCb+cbWidth)>>Log2ParMrgLevel is greater than xCb>>Log2ParMrgLevel and (yCb+cbHeight)>>Log2Par MrgLevel is greater than (yCb>>Log2ParMrgLevel). Where (xCb, yCb) is the upper left luminance sample position of the current CU in the picture, and (cbWidth, cbHeight) is the CU size. The MER size is selected on the encoder side and log2_parallel_merge_level_minus2 is indicated in the Sequence Parameter Set (SPS).

[0226] Merge Mode with MVD (MMVD)

[0227] In addition to the merge mode, where the implicitly derived motion information is directly used for prediction sample generation of the current CU, a merge mode with motion vector difference (MMVD) is introduced in VVC. The MMVD flag is signaled immediately after the regular merge flag is sent to specify whether the MMVD mode is used for the CU.

[0228] In MMVD, after a merge candidate (referred to as a basic merge candidate in the present invention) is selected, it is further refined by the indicated MVD information. Further information includes a merge candidate flag, an index specifying the magnitude of motion, and an index indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected as the MV basis. The MMVD candidate flag is indicated to specify which one is used between the first merge candidate and the second merge candidate.

[0229] The distance index specifies motion magnitude information and indicates predefined offsets (1512 and 1522) from the starting point of the L0 reference block 1510 and the starting point of the L1 reference block 1520. Fig.15 As shown, the offset is added to the horizontal component or the vertical component of the starting MV, where small circles of different styles correspond to different offsets from the center. Table 2 specifies the relationship between the distance index and the predefined offset.

[0230] Table 2 Relationship between distance index and predefined offset

[0231]

[0232] The direction index indicates the direction of the MVD relative to the starting point. As shown in Table 2, the direction index can indicate four directions. Note that the meaning of the MVD symbol may vary depending on the information of the starting MV. When the starting MV is a uni-predicted MV or a bi-predicted MV, where both lists point to the same side of the current picture (i.e., both referenced POCs are greater than the POC of the current picture, or both are less than the POC of the current picture), the symbol in Table 3 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bi-predicted MV, where the two MVs point to different sides of the current picture (i.e., one referenced POC is greater than the POC of the current picture, and the other referenced POC is less than the POC of the current picture), and the POC difference in List 0 is greater than the POC difference in List 1, then the symbol in Table 3 specifies the sign of the MV offset added to the List 0 MV component of the starting MV, and the sign of the List 1 MV has the opposite value. Otherwise, if the POC difference in List 1 is greater than the POC difference in List 0, then the symbol in Table 3 specifies the sign of the MV offset added to the List 1 MV component of the starting MV, and the sign of the List 0 MV has the opposite value.

[0233] MVD is scaled based on the difference in POC in each direction. If the POC in the two lists is the same, no scaling is required. Otherwise, Fig.13As shown, if the difference in POC in list 0 is greater than that in list 1, the MVD of list 1 is scaled by defining the POC difference of L0 as td and the POC difference of L1 as tb. If the POC difference of L1 is greater than that of L0, the MVD of list 0 is scaled in the same way. If the starting MV is uni-predicted, the MVD is added to the available MV.

[0234] Table 3 - Sign of MV offset specified by direction index

[0235] Directions to IDX 00 01 10 11 x-axis + - N / A N / A y-axis N / A N / A + -

[0236] Affine Motion Compensated Prediction

[0237] In HEVC, only the translational motion model is used for motion compensation prediction (MCP). In the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensation prediction is applied. Fig.16A As shown in FIG. 1B , the affine motion field of block 1610 is given by Fig.16A Two control points in (4 parameters) or Fig. 16B The motion information description of the three control point motion vectors (6 parameters).

[0238] For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

[0239]

[0240] For the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

[0241]

[0242] Among them, (mv 0x , mv 0y ) is the motion vector of the upper left control point, (mv 1x , mv 1y ) is the upper right corner control point through motion vector, and (mv 2x , mv 2y ) is the motion vector of the lower left control point.

[0243] In order to simplify motion compensated prediction, block-based affine transformation prediction is applied. Fig.17As shown, in order to derive the motion vector of each 4×4 luminance subblock, the motion vector of the center sample of each subblock is calculated according to the above formula and rounded to 1 / 16 fractional precision. Then, a motion compensation interpolation filter is applied to generate a prediction for each subblock using the derived motion vector. The subblock size of the chrominance component is also set to 4×4. The MV of the 4×4 chrominance subblock is calculated as the average of the MVs of the upper left luminance subblock and the lower right luminance subblock in the same 8x8 luminance region.

[0244] Like translational motion inter-picture prediction, there are two affine motion inter-picture prediction modes: affine merge mode and affine AMVP mode.

[0245] Affine Merge Prediction

[0246] AF_MERGE mode can be applied to CUs with width and height greater than or equal to 8. In this mode, the CPMV (Control Point MV) of the current CU is generated based on the motion information of the spatially neighboring CUs. There can be up to five CPMVP (CPMV prediction) candidates, and the index is marked to indicate the one that will be used for the current CU. The following three types of CPVM candidates are used to form the affine merge candidate list:

[0247] – Inherited affine merge candidates inferred from the CPMV of neighboring CUs

[0248] – Constructed affine merge candidate CPMVP derived using the translation MV of the adjacent CU

[0249] – Zero MV

[0250] In VVC, there are at most two inherited affine candidates, which are derived from the affine motion models of neighboring blocks, one from the left neighboring CU and the other from the top neighboring CU. Fig.11 The blocks shown are the same. For the left predictor, the scan order is A0->A1, while for the upper predictor, the scan order is B0->B1->B2. Only the first inherited candidate from each side is selected. No pruning check is performed between two inherited candidates. When a neighboring affine CU is identified, its control point motion vectors are used to derive the CPMVP candidate in the affine merge list of the current CU. Fig.18 As shown, if the adjacent lower left block A of the current block 1810 is encoded and decoded in affine mode, the motion vectors v of the upper left corner, upper right corner and lower left corner of the CU 1820 containing the block A are obtained. 2 , v 3 and v 4 When encoding and decoding block A using a 4-parameter affine model, according to v 2 and v3 Calculate the current CU (ie, v 0 and v 1 ) two CPMVs. When encoding and decoding block A using the 6-parameter affine model, according to v 2 、v 3 and v 4 Calculate the three CPMVs of the current CU.

[0251] The constructed affine candidate is constructed by combining the adjacent translation motion information of each control point. Fig.19 As shown, the motion information of the control point is derived from the specified spatial and temporal neighbors of the current block 1910. CPMV k (k=1,2,3,4) represents the kth control point. 1 , check B2->B3->A2 blocks and use the MV of the first available block. For CPMV 2 , check B1->B0 block, for CPMV 3 , check the A1->A0 block. For CPMV 4 , if TMVP is available, use it as CPMV 4 .

[0252] After obtaining the MVs of the four control points, an affine merge candidate is constructed based on the motion information. The following combinations of control point MVs are used to construct in order:

[0253] {CPMV1,CPMV2,CPMV3},{CPMV1,CPMV2,CPMV4},{CPMV1,CPMV3,CPMV4},{CPMV2,CPMV3,CPMV4},{CPMV1,CPMV2},

[0254] {CPMV1,CPMV3}

[0255] Combinations of 3 CPMVs constitute 6-parameter affine merge candidates, and combinations of 2 CPMVs constitute 4-parameter affine merge candidates. To avoid motion scaling processing, relevant combinations of control point MVs are discarded if the reference indices of the control points are different.

[0256] After checking inherited affine merge candidates and constructed affine merge candidates, if the list is still not full, a zero MV is inserted to the end of the list.

[0257] Affine AMVP prediction

[0258] Affine AMVP mode can be applied to CUs with width and height greater than or equal to 16. An affine flag in the CU level is indicated in the bitstream to indicate whether to use affine AMVP mode, and then another flag is indicated to indicate whether to use 4-parameter affine or 6-parameter affine. In this mode, the difference between the CPMV of the current CU and its predicted sub-CPMVP is indicated in the bitstream. The affine AVMP candidate list size is 2 and is generated by using the following four types of CPVM candidates in order:

[0259] – Inherited affine AMVP candidates inferred from the CPMV of neighboring CUs

[0260] – Constructed affine AMVP candidate CPMVP derived using the translation MV of the adjacent CU

[0261] –Translation MV from neighboring CU

[0262] – Zero MV

[0263] The order of checking the inherited affine AMVP candidates is the same as that of checking the inherited affine merge candidates. The only difference is that for AVMP candidates, only affine CUs with the same reference picture as the current block are considered. When the inherited affine motion predictor is inserted into the candidate list, no pruning is applied.

[0264] Constructed AMVP candidates can be selected from Fig.19 The 4-parameter affine mode is used to encode the first block with the same reference picture as the current CU. 0 and mv 1 When all three CPMVs are available, they are added as a candidate in the affine AMVP list. When the current CU is encoded and decoded with 6-parameter affine mode, and all three CPMVs are available, they are added as a candidate in the affine AMVP list. Otherwise, the constructed AMVP candidate will be set to unavailable.

[0265] If after inserting valid inherited affine AMVP candidates and constructed AMVP candidates, the number of affine AMVP list candidates is still less than 2, then mv 0 、mv 1 and mv 2 Will be added as a translation MV to predict all control point MVs of the current CU. Finally, if the affine AMVP list is still not full, the list is filled with zero MVs.

[0266] Affine motion information storage

[0267] In VVC, the CPMV of an affine CU is stored in a separate buffer. The stored CPMV is only used to generate the inherited CPMVP of the most recently encoded CU in affine merge mode and affine AMVP mode. The sub-block MV derived from the CPMV is used for motion compensation, merging of translation MVs / MV derivation of AMVP lists, and deblocking.

[0268] To avoid an extra CPMV in the image line buffer, the affine motion data inheritance of the CU from the upper CTU is handled differently from the inheritance from the regular neighboring CU. If the candidate CU for affine motion data inheritance is located in the upper CTU line, the lower left sub-block MV and the lower right sub-block MV in the line buffer are used for affine MVP derivation instead of the CPMV. In this way, the CPMV is only stored in the local buffer. If the candidate CU is 6-parameter affine coded, the affine model is downgraded to a 4-parameter model. Fig. 20 As shown, along the top CTU boundary, the bottom left sub-block motion vector and the bottom right sub-block motion vector of the CU are used for affine inheritance of the CU in the bottom CTU. Fig. 20 , lines 2010 and 2012 represent the x and y coordinates of the picture, with the origin (0,0) being at the upper left corner. Legend 2020 shows the meaning of various motion vectors, where arrow 2022 represents CPMV for affine inheritance in the local buffer, arrow 2024 represents sub-block vectors for MC / Merge / Skip / AMVP / Deblocking / TMVP in the local buffer and sub-block vectors for affine inheritance in the row buffer, and arrow 2026 represents sub-block vectors for MC / Merge / Skip / AMVP / Deblocking / TMVP.

[0269] Adaptive Motion Vector Resolution (AMVR)

[0270] In HEVC, when use_integer_mv_flag in the slice header is equal to 0, the motion vector difference (MVD) (between the CU's motion vector and the predicted motion vector) is indicated in units of one quarter of the brightness sample. In VVC, a CU-level adaptive motion vector resolution (AMVR) scheme is introduced. AMVR allows the MVD of the CU to be encoded and decoded with different precisions. Depending on the current CU mode (conventional AMVP mode or affine AVMP mode), the MVD of the current CU can be adaptively selected as follows:

[0271] – Conventional AMVP modes: quarter brightness sampling, half brightness sampling, integer brightness sampling, or quad brightness sampling.

[0272] – Affine AMVP mode: quarter brightness sampling, integer brightness sampling, or one-sixteenth brightness sampling.

[0273] The CU-level MVD resolution indication is conditionally indicated if the current CU has at least one non-zero MVD component. If all MVD components (ie, horizontal and vertical MVD for reference list L0 and reference list L1) are zero, a quarter luma sample MVD resolution is inferred.

[0274] For CUs with at least one non-zero MVD component, a first flag is indicated to indicate whether quarter luma sample MVD precision is used for the CU. If the first flag is 0, no further indication is required and quarter luma sample MVD precision is used for the current CU. Otherwise, a second flag is indicated to indicate that half luma samples or other MVD precision (integer or quad luma samples) are used for regular AMVP CUs. In the case of half luma samples, a 6-tap interpolation filter, instead of the default 8-tap interpolation filter, is used for half luma sample positions. Otherwise, a third flag is indicated to indicate whether integer luma samples or quad luma sample MVD precision is used for regular AMVP CUs. In the case of affine AMVP CUs, a second flag is used to indicate whether integer valued luma samples or one-sixteenth luma sample MVD precision is used. To ensure that the reconstructed MV has the expected precision (quarter luma samples, half luma samples, integer luma samples, or quad luma samples), the motion vector predictor of the CU is rounded to the same precision as the MVD before being added with the MVD. The motion vector predictor is rounded towards zero (ie, a negative motion vector predictor is rounded towards positive infinity and a positive motion vector predictor is rounded towards negative infinity).

[0275] The encoder uses RD check to determine the motion vector resolution of the current CU. In order to avoid always performing four CU-level RD checks for each MVD resolution, the RD check of MVD precision other than quarter brightness samples is only conditionally called in VTM11. For the conventional AVMP mode, the RD cost of quarter brightness sample MVD precision and integer brightness sample MV precision is first calculated. Then, the RD cost of integer brightness sample MVD precision is compared with the RD cost of quarter brightness sample MVD precision to determine whether it is necessary to further check the RD cost of four brightness sample MVD precision. When the RD cost of quarter brightness sample MVD precision is much smaller than the RD cost of integer brightness sample MVD precision, the RD check of four brightness sample MVD precision is skipped. Then, if the RD cost of integer brightness sample MVD precision is significantly greater than the best RD cost among the previously tested MVD precisions, the check of half brightness sample MVD precision is skipped. For the affine AMVP mode, if the affine inter-frame mode is not selected after checking the rate-distortion cost of the affine merge / skip mode, merge / skip mode, quarter luma sample MVD precision regular AMVP mode, and quarter luma sample MVD precision affine AMVP mode, the one-sixteenth luma sample MV precision and 1-pixel MV precision affine inter-frame modes are not checked. In addition, the affine parameters obtained in the one-quarter luma sample MV precision affine inter-frame model are used as the starting search points in the one-sixteenth luma sample and one-quarter luma sample MV precision affine inter-frame modes.

[0276] Bi-Prediction with CU-level Weight (BCW)

[0277] In HEVC, two prediction signals are obtained from two different reference pictures. 0 and P 1 Averaging and / or using two different motion vectors, the bi-prediction signal P bi-pred In VVC, the bi-prediction mode is extended beyond simple averaging to allow for a weighted average of two prediction signals.

[0278] P bi-pred =((8-w)*P 0 +w*P 1 +4)>>3 (8)

[0279] Five weights, w∈{-2,3,4,5,10}, are allowed in weighted average bi-prediction. For each bi-predicted CU, the weight w is determined in one of two ways: 1) for non-merged CUs, the weight index is indicated after the motion vector difference; 2) for merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applied to CUs with 256 or more luma samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency pictures, all 5 weights are used. For non-low-latency pictures, only 3 weights (w∈{3,4,5}) are used. At the encoder, a fast search algorithm is applied to find the weight index without significantly increasing the encoder complexity. These algorithms are summarized as follows. The details are disclosed in the VTM software and document JVET-L0646 (Yu-Chi Su, et.al., “CE4-related: Generalized bi-prediction improvements combined from JVET-L0197 and JVET-L0296”, Joint Video Experts Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29, 12th Meeting: Macao, CN, 3–12 Oct. 2018, Document: JVET-L0646).

[0280] – When combined with AMVR, unequal weights are only conditionally checked for 1-pixel motion vector precision and 4-pixel motion vector precision if the current picture is a low-latency picture.

[0281] – When combined with affine, affine ME will be performed for unequal weights if and only if the affine mode is selected as the current best mode.

[0282] – When the two reference pictures in bidirectional prediction are the same, unequal weights are only checked conditionally.

[0283] – Do not search for unequal weights when certain conditions are met, which depend on the POC distance between the current picture and its reference pictures, the codec QP, and the temporal level.

[0284] The BCW weight index is encoded using a context codec binary followed by a bypass codec binary. The first context codec binary indicates whether equal weights are used; if unequal weights are used, an additional bypass codec binary is indicated to indicate which unequal weights are used.

[0285] Weighted prediction (WP) is a codec tool supported by the H.264 / AVC and HEVC standards to efficiently encode and decode gradient video content. Support for WP has also been added to the VVC standard. WP allows weighting parameters (weights and offsets) to be indicated for each reference picture in the reference picture list L0 and the reference picture list L1. Then, during motion compensation, the weights and offsets of the corresponding reference pictures are applied. WP and BCW are designed for different types of video content. In order to avoid the interaction between WP and BCW, which will complicate the VVC decoder design, if the CU uses WP, the BCW weight index is not indicated, and the weight w is inferred to be 4 (i.e., equal weights are applied). For the merged CU, the weight index is inferred from the neighboring blocks based on the merge candidate index. This can be applied to the normal merge mode and the inherited affine merge mode. For the reconstructed affine merge mode, the affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index of the CU using the constructed affine merge mode is simply set equal to the BCW index of the first control point MV.

[0286] In VVC, CIIP and BCW cannot be applied jointly to a CU. When a CU is encoded and decoded using CIIP mode, the BCW index of the current CU is set to 2 (ie, w=4 for equal weight). Equal weight means the default value of the BCW index.

[0287] Combined Inter and Intra Prediction (CIIP)

[0288] In VVC, when a CU is encoded and decoded in merge mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and if both the CU width and the CU height are less than 128 luma samples, an additional flag is signaled to indicate whether the combined inter / intra prediction (CIIP) mode is applied to the current CU. As the name indicates, the CIIP prediction combines the inter prediction signal with the intra prediction signal. The inter prediction signal P_inter in the CIIP mode is derived using the same inter prediction process applied to the regular merge mode; and the intra prediction signal P_intra is derived after the regular intra prediction process with the planar mode. The intra prediction signal and the inter prediction signal are then combined using a weighted average, where the weighted average is calculated based on the top neighboring blocks and the left neighboring blocks (such as the left neighboring blocks) of the current CU 2110. Fig.21 The weight value wt is calculated as shown below.

[0289] – If top adjacent is available and is intra-coded, set isIntraTop to 1, otherwise set isIntraTop to 0;

[0290] – If the left adjacent image is available and the image is encoded and decoded within the picture, isIntraLeft is set to 1, otherwise isIntralLeft is set to 0;

[0291] – If (isIntraLeft + isIntraTop) is equal to 2, set wt to 3;

[0292] – Otherwise, if (isIntraLeft + isIntraTop) is equal to 1, set wt to 2;

[0293] – Otherwise, set wt to 1.

[0294] The CIIP forecast is formed as follows:

[0295] P CIIP =((4-wt)*P inter +wt*P intra +2)>>2 (9)

[0296] Geometric Partitioning Mode (GPM)

[0297] In VVC, Geometric Partitioning Mode (GPM) is supported for inter-picture prediction, as described in JVET-W2002 (Adrian Browne, et al., Algorithm description for VersatileVideo Coding and Test Model 14 (VTM 14), ITU-T / ISO / IEC Joint Video ExplorationTeam (JVET), 23rd Meeting, by teleconference, 7–16 July 2021, document: document JVET-M2002). Using the CU-level flag, the geometric partitioning mode is marked as a merge mode, while other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. The geometric partitioning mode supports a total of 64 partitions for each possible CU size, w×h=2 m ×2 n, where m,n∈{3…6}, excluding 8x64 and 64x8. GPM mode can be applied to skip or merge CUs whose sizes are within the above limits and have at least two normal merge modes.

[0298] When this mode is used, the CU is divided into two parts by a geometrically positioned straight line at a certain angle. In VVC, there are a total of 20 angles and 4 offset distances used for GPM, which is reduced from 24 angles in the early draft. The position of the dividing line is mathematically derived from the angle parameters and offset parameters of the specific partition. In VVC, if Fig. 22 As shown, there are 64 partitions in total, where the partitions are grouped according to their angles, and the dashed lines represent redundant partitions. Each part of the geometric partition in the CU is inter-predicted using its own motion; only uni-prediction is allowed for each partition, i.e., each part has a motion vector and a reference index. Fig. 22 , each line corresponds to the boundary of a partition. The partitions are grouped according to their angles. For example, partition group 2210 includes three vertical GPM partitions (i.e., 90 degrees). Partition group 2220 includes four inclined GPM partitions at a small angle to the vertical direction. In addition, partition group 2230 includes three vertical GPM partitions (i.e., 270 degrees), which are similar to the partitions of group 2210, but with opposite directions. As with traditional dual prediction, single prediction motion constraints are applied. To ensure that only two motion compensated predictions are required for each CU. The unified prediction motion for each partition is derived through the processing described later.

[0299] If geometric partitioning mode is used for the current CU, a geometric partitioning index indicating the selected partitioning mode (angle and offset) for the geometric partitioning and two merge indices (one for each partition) are further indicated. The number of maximum GPM candidate sizes is explicitly indicated in the SPS, and the syntax binarization for the GPM merge indices is specified. After predicting each part of the geometric partitioning, the sample values ​​along the edges of the geometric partitioning are adjusted using a hybrid process with adaptive weights using the processing described later. This is the prediction signal for the entire CU, and, as in other prediction modes, the transform and quantization process will be applied to the entire CU. Finally, using the processing described later, the motion field of the CU predicted using the geometric partitioning mode is stored.

[0300] Single prediction candidate list construction

[0301] The uniprediction candidate list is derived directly from the merge candidate list constructed according to the extended merge prediction process. In the geometric uniprediction candidate list, n is represented as the index of the uniprediction motion. The LX motion vector (X = 0 or 1, i.e. LX = L0 or L1) of the nth extended merge candidate, where X is equal to the parity of n, is used as the nth uniprediction motion vector of the geometric partition mode. These motion vectors are Fig.23 In the case where the corresponding LX motion vector of the n-th extended merge candidate does not exist, the L(1-X) motion vector of the same candidate is used as the uni-prediction motion vector of the geometric partition mode.

[0302] Blending along geometric split edges

[0303] After predicting each part of the geometric partition using its own motion, blending is applied to the two prediction signals to derive samples around the geometric partition edges. The blending weights for each position of the CU are derived based on the distance between each position and the partition edge.

[0304] Two integer mixing matrices (W0 and W1) are used for the GPM mixing process. Fig.24 As shown, the weights in the GPM mixing matrix contain a value range of [0, 8] and are derived based on the displacement from the sample location to the GPM segmentation boundary 2440.

[0305] Specifically, Fig.25 As shown, the weights are given by a discrete ramp function with a displacement and two thresholds, where the two endpoints of the ramp (i.e., -τ and τ) correspond to Fig.24 Line 2442 and line 2444 in.

[0306] Here, the threshold τ defines the width of the GPM blending region and is chosen as a fixed value in VVC. In other words, as in JVET-Z0137 (Han Gao, et.al., "Non-EE2: Adaptive Blending for GPM", Joint Video Experts Team (JVET) of ITU-T SG 16WP 3and ISO / IEC JTC 1 / SC 29, 26th Meeting, bytle conference, 20–29 April 2022, JVET-Z0137), the blending strength or blending region width θ is fixed for all different contents.

[0307] The weight values ​​in the blend mask can be given by the following ramp function:

[0308]

[0309] With a fixed pixel value θ=2 in the current ECM (VVC) design, the ramp function can be quantized as:

[0310] ω m,n =Clip3(0,8,(d(m,n)+32+4)>>3) (11)

[0311] The distance from position (x, y) to the segmentation edge is derived as:

[0312]

[0313] Where i,j are the angle and offset indices of the geometric segmentation, which depend on the indicated geometric segmentation index. x , j and ρ y,j The sign of depends on the angle index i.

[0314] Fig.26 An example of GPM mixing according to ECM 4.0 (Muhammed Coban, et.al., “Algorithm description of Enhanced Compression Model 4 (ECM 4)”, Joint Video Experts Team (JVET) of ITU-TSG 16WP 3 and ISO / IEC JTC 1 / SC 29, 26th Meeting, by teleconference, 20–29 April 2022, JVET-Y2025) is described. Fig.26 In , the size of the mixed region on each side of the segmentation boundary is indicated by θ. The weights for each part of the geometric segmentation are derived as follows:

[0315] wIdxL(x,y)=partIdx? 32+d(x,y):32-d(x,y) (16)

[0316]

[0317] w 1 (x,y)=1-w 0 (x,y) (18)

[0318] partIdx depends on the angle index i. Fig.24 The weight w is shown 0 An example of where the angle 2410 and offset ρ i 2420 is indicated for GPM index i, point 2430 corresponds to the center of the block. Line 2440 corresponds to the GPM segmentation boundary

[0319] Motion field storage for geometric partitioning patterns

[0320] Mv1 from the first part of the geometric partition, Mv2 from the second part of the geometric partition, and the combined MV of Mv1 and Mv2 are stored in the motion field of the geometric partition mode codec CU.

[0321] The type of stored motion vector for each individual position in the motion field is determined as:

[0322] sTypee=abs(motionldx)<32?2:

[0323] (motionIdx≤0?(1-partIdx):partIdx) (19)

[0324] Wherein, motionIdx is equal to d(4x+2,4y+2), which is recalculated according to formula (7). partIdx depends on the angle index i.

[0325] If sType is equal to 0 or 1, then Mv0 or Mv1 is stored in the corresponding motion field, otherwise, if sTType is equal to 2, the combined MV from Mv0 and Mv2 is stored. The combined MV is generated using the following process:

[0326] 1) If Mv1 and Mv2 are from different reference picture lists (one from L0 and the other from L1), then Mv1 and Mv2 are simply combined to form a bi-prediction motion vector.

[0327] 2) Otherwise, if Mv1 and Mv2 are from the same list, only the uni-predicted motion Mv2 is stored.

[0328] Multi-Hypothesis Prediction (MHP)

[0329] In the multi-hypothesis inter-picture prediction mode (JVET-M0425), in addition to the conventional bi-prediction signal, one or more additional motion compensated prediction signals are indicated. The resulting overall prediction signal is obtained by weighted superposition per sample. Using the bi-prediction signal p_bi and the first additional inter-picture prediction signal / hypothesis h_3, the resulting prediction signal is obtained as follows:

[0330] p 3 =(1-α)p bi +αh 3 (20)

[0331] The weight factor α is specified by a new syntax element add_hyp_weight_idx according to the following mapping (Table 4):

[0332] Table 4 maps α to add_hyp_weight_idx

[0333]

[0334]

[0335] Similar to the above, more than one additional prediction signal may be used. The resulting overall prediction signal is iteratively accumulated together with each additional prediction signal.

[0336] p n+1 =(1-α n+1 ) n +a n+1 h n+1 (twenty one)

[0337] The resulting overall prediction signal is obtained as the final p n (i.e., the p with the largest index n n ). For example, at most two additional prediction signals may be used (ie, n is limited to 2).

[0338] The motion parameters for each additional prediction hypothesis can be explicitly indicated by specifying the reference index, MV predictor index and MV difference, or implicitly indicated by specifying the merge index. A separate multi-hypothesis merge flag distinguishes between these two indication modes.

[0339] For AMVP inter-mode, MHP is applied only when unequal weights in BCW are selected in bi-prediction mode. Details of MHP for VVC can be found in JVET-W2025 (Muhammed Coban, et.al., "Algorithm description of Enhanced Compression Model 2 (ECM 2)", Joint Video Experts Team (JVET) of ITU-TSG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 23rd Meeting, by teleconference, 7–16 July 2021, Document: JVET-W2025).

[0340] GPM Extensions

[0341] Several variants of the GPM mode (JVET-W0097 (Zhipin Deng, et.al., "AEE2-related: Combination of EE2-3.3, EE2-3.4 and EE2-3.5", Joint Video Experts Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29, 23rd Meeting, by teleconference, 7–16 July 2021, Document: JVET-W0097) and JVET-Y0065 (Yoshitaka Kidani, et.al., "EE2-3.1: GPM with inter and intra prediction (JVET-X0166)", Joint Video Experts Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29, 25th Meeting, byteleleconference, 12–21 January 2022, Document: JVET-Y0065)) have been proposed to improve the encoding and decoding efficiency of the GPM mode in VVC. These methods are included in the Exploration Experiment (EE2) for further evaluation, and their main technical aspects are described as follows:

[0342] EE2-3.3 Regarding GPM with MMVD (GPM-MMVD): 1) Additional MVD is added to the existing GPM merge candidates; 2) MVD is marked in the same way as MMVD in VVC, i.e., a distance index plus a direction index; 3) Two flags are marked to control whether MMVD is applied to each GPM partition respectively.

[0343] EE2-3.4-3.5 Regarding GPM and template matching (GPM-TM): 1) Template matching is extended to GPM mode by refining the GPM MV based on the left neighboring samples and the top neighboring samples of the current CU; 2) Template samples are selected according to the GPM partition direction; 3) A single flag is marked to jointly control whether template matching is applied to the MVs of two GPM partitions.

[0344] JVET-W0097 proposes a combination of EE2-3.3, EE2-3.4 and EE2-3.5 to further improve the encoding and decoding efficiency of the GPM mode. Specifically, in the proposed combination, the existing designs in EE2-3.3, EE2-3.4 and EE2-3.5 remain unchanged, while the following modifications are further applied to coordinate the two encoding and decoding tools:

[0345] 1) GPM-MMVD and GPM-TM are enabled specifically for one GPM CU. This is done by first indicating the GPM-MMVD syntax. When both GPM-MMVD control flags are equal to false (i.e., GPM-MMVD is disabled for both GPM partitions), the GPM-TM flag is indicated to indicate whether pattern matching is applied to both GPM partitions. Otherwise (at least one GPM-MMVD flag is equal to true), the GPM-TM flag target value is inferred to be false.

[0346] 2) The GPM merge candidate list generation methods in EE2-3.3 and EE2-3.4-3.5 are directly combined, in such a way that the MV pruning scheme in EE2-3.4.3.5 (in which the MV pruning threshold is adaptive based on the current CU size) is applied instead of the default MV pruning scheme applied in EE2-3.3; in addition, as in EE2-3.4-3.5, a plurality of zero MVs are added until the GPM candidate list is completely filled.

[0347] In JVET-Y0065, in GPM with inter- and intra-prediction (or named GPM intra), the final prediction samples are generated by weighting the inter-prediction samples and intra-prediction samples of each GPM separation area. The inter-prediction samples are derived through the same scheme as the GPM in the current ECM, while the intra-prediction samples are derived through the intra-prediction mode (IPM) candidate list and the index indicated from the encoder. The IPM candidate list size is predefined to be 3. Fig.27A As shown in Figure 1-C, the available IPM candidates are the parallel angle mode (parallel mode) for the GPM block boundary, the vertical angle mode (vertical mode) for the GPM block boundary, and the planar mode. Fig.27D The GPM with inter-picture prediction and intra-picture prediction shown is restricted in the proposed method to reduce the IPM signaling overhead and avoid the increase of intra-picture prediction circuit size on the hardware decoder. In addition, direct motion vector and IPM storage for GPM mixed regions is introduced to further improve the codec performance.

[0348] Space GPM

[0349] Similar to inter-picture GPM, spatial GPM (SGPM) includes a partition mode and two related intra-picture prediction modes. Fig.28A As shown in , if these modes are directly indicated in the bitstream, significant overhead bits will be generated. In order to more efficiently express the necessary segmentation and prediction information in the bitstream, a candidate list is used, and only the candidate index will be indicated in the bitstream. Fig.28B As shown, each candidate in the list can derive a combination of a partition mode and two intra-picture prediction modes.

[0350] The template is used to generate the candidate list. The template has the shape of Fig.29 As shown. Fig.29 As shown, for each possible combination of a partition mode and two intra prediction modes, a prediction is generated for the template by extending the partition weights to the template. These combinations are arranged in ascending order of their SATD between the prediction and reconstruction of the template. The length of the candidate list is set equal to 16, and these candidates are considered as the most likely SGPM combination for the current block. Both the encoder and the decoder construct the same candidate list based on the template.

[0351] To reduce the complexity of creating the candidate list, the number of possible partition modes and the number of possible intra-picture prediction modes are pruned. In the following tests, 26 of the 64 partition modes are used, and only the MPMs of the 67 intra-picture prediction modes are used.

[0352] Overlapped Motion Compensation (OBMC)

[0353] When OBMC is applied, as described in JVET-L0101 (Zhi-Yi Lin, et al. "CE10.2.1:OBMC", Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 12th Meeting: Macao, CN, 3–12 Oct. 2018, Document: JVET-L0101), the top boundary pixels and left boundary pixels of the CU are refined using motion information of neighboring blocks with weighted prediction.

[0354] The conditions under which OBMC should not be applied are as follows:

[0355] ●When OBMC is disabled at SPS level;

[0356] ●When the current block has intra-picture mode or IBC mode;

[0357] ●When the current block applies LIC;

[0358] ●When the current luminance block area is less than or equal to 32.

[0359] Sub-block boundary OBMC is performed by applying the same blending to the top, left, bottom and right sub-block boundary pixels using the motion information of neighboring sub-blocks. It is enabled for sub-block based codec tools:

[0360] ●Affine AMVP mode;

[0361] ●Affine merge mode and subblock-based temporal motion vector prediction (SbTMVP);

[0362] ●Sub-block based bilateral matching.

[0363] In order to improve the coding and decoding efficiency of cross-color prediction, various techniques are disclosed below.

[0364] Reconstructed adjacent sample preprocessing

[0365] When deriving the model parameters, the reconstructed neighboring samples of the first and second components are used. Take the CCLM described in the overview section as an example. The first component is luminance and the second component is cb or cr. In order to improve the model performance, the reconstructed neighboring samples are preprocessed before becoming the input for deriving the model parameters.

[0366] Fig.30 An example of reconstructed neighboring samples that are preprocessed before becoming input for deriving model parameters is shown, where a neighboring region 3010 of a luma block 3012 and a neighboring region 3020 of a chroma (cb or cr) block 3012 are preprocessed before being provided to a model parameter derivation block 3030.

[0367] In one embodiment, the reconstructed neighboring samples of the first component are pre-processed.

[0368] In one embodiment, the reconstructed neighboring samples of the second component are pre-processed.

[0369] In another embodiment, only the reconstructed neighboring samples of one of the first component and the second component are pre-processed.

[0370] In one embodiment, the pre-processing method may be any one or any combination of (but not limited to) the following processes: 3x3 or 5x5 filtering, biasing, clipping, filtering or clipping similar to ALF or CCALF, filtering similar to SAO, filter sets (e.g., ALF sets)

[0371] In another embodiment, the first component is any one of brightness, cb and cr. For example, when the first component is brightness, the second component is cb or cr. For another example, when the first component is cb, the second component is brightness or cr. For another example, when the first component is cr, the second component is brightness or cb. For another example, when the first component is brightness, the second component is based on a weighted combination of cb and cr.

[0372] In one embodiment, the preprocessing method of one component (e.g., cr) depends on another component (e.g., cb). For example, the selection of the preprocessing method of cb is derived based on the label / bitstream, and cr follows the selection of cb. For another example, assuming that there is a high correlation between cb and cr, the selection of the preprocessing method of cr is as follows:

[0373] -cb reconstruction (no preprocessing) plus cb residual is considered golden (i.e. the target that guides the processing)

[0374] -Based on the preprocessing reconstruction of cb and the preprocessing method of golden selection cr

[0375] - For example, if the preprocessing reconstruction of CB is very similar to gold, then use the preprocessing method of CB as the preprocessing method of CR.

[0376] In another embodiment, the pre-processing method is applied immediately after reconstructing neighboring samples of the first and / or second component.

[0377] In another embodiment, a pre-processing method is applied to the reconstructed neighboring samples before generating the model parameters for the current block.

[0378] Prediction Sample Post-processing

[0379] After CCLM is applied to the current block, a prediction of the current block is generated and can be further adjusted using post-processing methods. The post-processing methods can be (but are not limited to) any one or any combination of the following processes: 3x3 or 5x5 filtering, biasing, clipping, filtering or clipping similar to ALF or CCALF, SAO-like filtering, filter sets (e.g., ALF sets).

[0380] In one embodiment, the current block refers to luma, cb and / or cr. For example, when LM (e.g., the proposed inverse LM described in the subsequent part of the present invention) is used to generate luma prediction, post-processing is applied to luma. For another example, when CCLM is used to generate chroma prediction, post-processing is applied to chroma.

[0381] In another embodiment, post-processing is applied when the block size (width and / or height) is greater than a threshold.

[0382] In another embodiment, the post-processing method of one component (e.g., cr) depends on another component (e.g., cb). For example, the selection of the post-processing method of cb is derived based on the label / bitstream, and cr follows the selection of cb. For another example, assuming that there is a high correlation between cb and cr, the selection of the post-processing method of cr is as follows:

[0383] -cb predictions (no post-processing) plus cb residuals are considered golden

[0384] - Post-processing prediction based on cb and post-processing method of golden selection cr

[0385] - For example, if the post-processing predictions of CB are very similar to gold, then use the post-processing method of CB as the post-processing method of CR.

[0386] Delta predicted LM

[0387] This section proposes a new LM method. Different from the CCLM previously disclosed in the background section, the inputs for deriving model parameters are the predicted samples of the first component (used as X) and the delta samples between the reconstructed samples and the predicted samples of the first component (used as Y). The derived parameters and the initial predicted samples of the second component can determine the current predicted samples of the second component. For example, the predictors of cb and cr can be calculated based on the following formulas:

[0388] delta_cb=alpha*initial_pred_cb+beta, pred_cb=initial_pred_cb+delta_cb,

[0389] delta_cr=alpha*initial_pred_cr–beta, pred_cr=initial_pred_cr+delta_cr.

[0390] For another example, the predictors of cb and cr can be calculated based on the following formula:

[0391] delta_cb=alpha*initial_pred_cb+beta, pred_cb=initial_pred_cb+delta_cb,

[0392] delta_cr=-alpha*initial_pred_cr+beta,, pred_cr=initial_pred_cr+delta_cr.

[0393] Embodiments of the prediction-reconstruction (pred-reco) LM may be used for the delta prediction (delta-pred) LM.

[0394] Prediction-Reconstruction LM

[0395] This section proposes a new LM method. Different from the CCLM previously disclosed in the background section, the inputs for deriving model parameters are the predicted samples of the first component (used as X) and the reconstructed samples of the first component (used as Y). The derived parameters and the initial predicted samples of the second component can determine the current predicted samples of the second component. For example, the predictors of cb and cr can be calculated based on the following formulas:

[0396] Pred_cb=alpha*initial_pred_cb+beta

[0397] Pred_cr=alpha*initial_pred_cr–beta

[0398] As another example, the predictors for cb and cr can be calculated based on:

[0399] Pred_cb=alpha*initial_pred_cb+beta,

[0400] Pred_cr=-alpha*initial_pred_cr+beta.

[0401] In one embodiment, the first component is luminance and the second component is cb or cr.

[0402] In another embodiment, the first component is cb and the second component is cr.

[0403] In another embodiment, the first component is weighted cb and cr and the second component is luminance, where the inverse LM is applied. For example, the input to derive the model parameters is the weighted predictions of cb and cr and the weighted reconstructed samples of cb and cr.

[0404] In a sub-embodiment, the weights of (cb, cr) may be equal.

[0405] In another sub-embodiment, the weight of (cb, cr) may be (1, 3) or (3, 1). Taking (3, 1) as an example, the weighting formula may be:

[0406] weighted_pred=(3*pred_cb+1*pred_cr+offset)>>2,

[0407] weighted_reco=(3*reco_cb+1*reco_cr+offset)>>2.

[0408] In another embodiment, the initial predicted samples of the second component are generated through the chroma DM.

[0409] In another embodiment, the initial prediction samples of the second component are generated by one or more conventional intra prediction modes (eg, angular intra prediction model, DC, planar).

[0410] Prediction-based LM

[0411] The present invention proposes a new LM method. Unlike the CCLM described above, in which the model parameters are applied to the reconstructed samples of the first component, the derived model parameters of the present invention are used for the predicted samples of the first component to obtain the predicted samples of the second component or the third component. Fig.31 shows an example of a prediction-based LM, where the predicted samples from luma (i.e., pred′ L (i,j)) is used to predict the chrominance signal (i.e., P(i,j)):

[0412] P(i,j)=a·pred′ L (i,j)+b

[0413] Although Fig.31 The example in derives chrominance prediction based on luma, but the invention can derive a prediction for any color component based on another color component. In one embodiment, the first component is luma.

[0414] In one sub-embodiment, the predicted samples of the first component are downsampled using a downsampling filter. For example, the downsampling filter follows the original LM design. For another example, the downsampling filter will not access adjacent predicted / reconstructed samples. At the boundary of the current block, if the adjacent samples are required to be input samples of the downsampling filter, the padded predicted values ​​from the current block boundary are used.

[0415] In another embodiment, the second component is Cb.

[0416] In another embodiment, the third component is Cr.

[0417] The following shows the flow of prediction-based inter-picture CCLM.

[0418] ● Improved inter-frame chroma prediction by linearly predicting chroma samples from luma samples

[0419] ●The linear prediction method can be one of the following

[0420] –CCLM_LT,CCLM_L,CCLM_T

[0421] –MMLM_LT,MMLM_L,MMLM_T

[0422] ●Steps:

[0423] – Step 1: Infer a linear model through adjacent reconstructed luma and chroma samples

[0424] – Step 2: Apply the derived linear model to the current luma predicted sample to obtain the current chroma predicted sample

[0425] -pred CCLM (i,j)=α·pred′ L (i,j)+β

[0426] -pred L ′(i,j): downsampled current brightness predicted sample

[0427] Use padding at the boundaries inside the current block

[0428] In the above context of the present invention, CCLM_LT, CCLM_L, CCLM_T, MMLM_LT, MMLM_L and MMLM_T are all referred to as LM modes.

[0429] Joint LM

[0430] Different from the previously published CCLM, a joint linear model is proposed to share a single model for the chrominance components (cb and cr).

[0431] In one embodiment, the parameters of the derived single model include alpha and beta.For example, the predictors of cb and cr can be calculated based on the luma reconstructed samples and the parameters.

[0432] Pred_cb=alpha*reco_luma+beta,

[0433] Pred_cr=alpha*reco_luma–beta.

[0434] For another example, the predictors of cb and cr can be calculated as

[0435] Pred_cb=alpha*reco_luma+beta,

[0436] Pred_cr=-alpha*reco_luma+beta.

[0437] In another embodiment, when deriving model parameters, luminance, cb and cr are used. The luminance part remains unchanged, and the chrominance part changes. For example, before being used as input for deriving model parameters, the reconstructed adjacent samples of cb and cr are weighted. The weighting method can be any one of the methods described in the section of JCCLM method 1 / -method 2 or any combination thereof.

[0438] In another embodiment, when deriving the model parameters, the luma component and one chroma component are used. For example, luma and cb are used to determine the model parameters.

[0439] In another embodiment, instead of using adjacent reconstructed samples, adjacent residuals are used to derive model parameters. Then, the joint residual of cb and cr is derived as follows:

[0440]

[0441] In one sub-embodiment, if JCCR is applied, the LM parameters of cb and cr are the same (ie, joint LM is applied).

[0442] In another sub-embodiment, the adjacent residuals of chrominance are weighted sums of adjacent cb residuals and cr residuals.

[0443] In another sub-embodiment, if joint LM is applied, then JCCR is inferred to be enabled.

[0444] In another sub-embodiment, when joint LM is used, the prediction of the current chroma block is generated via the chroma DM mode.

[0445] In another sub-embodiment, when joint LM is used, an initial prediction of the current chroma block is generated through the chroma DM mode, and a final prediction of the current chroma block is generated based on the initial prediction and resiC (eg, initial prediction+resiC).

[0446] Residual LM

[0447] Instead of using adjacent reconstructed samples, adjacent residuals are used to derive model parameters. Then, the joint residual of the current chrominance block is derived as follows (cb and cr have their own models respectively):

[0448]

[0449] In one embodiment, a prediction of the current chroma block (denoted as pred_c) is generated by the chroma DM, and a reconstruction of the current chroma block is formed by pred_c + resi_c.

[0450] In another embodiment, an initial prediction of the current chroma block is generated through the chroma DM mode, and based on the initial prediction and the resi C Generate the final prediction for the current chrominance block. (e.g., initial prediction + resi C ).

[0451] JCCLM (JCCR with CCLM) - Method 1

[0452] JCCLM-Method 1 is proposed as a new LM derivation scheme. Unlike the CCLM previously disclosed in the background section, neighboring luminance reconstructed samples and weighted reconstructed neighboring cb and cr samples are used as inputs X and Y for model derivation. In the present invention, the derived model is called JCCLM, and the model parameters are called JCCLM parameters. Then, based on the JCCLM parameters and the reconstructed samples of the same luminance block, the JCCLM predictor is determined. Finally, the predictions of cb and cr are calculated through the JCCLM predictor.

[0453] In one embodiment, the weights used to generate the weighted reconstructed neighboring cb and cr samples may be (1, -1) of (cb, cr).

[0454] In another embodiment, the weights used to generate the weighted reconstructed neighboring cb and cr samples may be (1 / 2, 1 / 2) of (cb, cr).

[0455] In another embodiment, the predictions of cb and cr are calculated as follows:

[0456] pred_cb=1*JCCLM_predictor, pred_cr=-1*JCCLM_predictor+k

[0457] In another sub-embodiment, k can be any positive value, for example, k=512.

[0458] In another sub-embodiment, k varies with the bit depth. For example, if the bit depth is 10, then k=512.

[0459] In another sub-embodiment, k is predefined in the standard or depends on a block, SPS, PPS and / or picture level indication.

[0460] In another embodiment, the predictions of cb and cr are calculated as follows:

[0461] pred_cb=1*JCCLM_predictor, pred_cr=1*JCCLM_predictor.

[0462] In another embodiment, when the weights used to generate the weighted reconstructed neighboring cb and cr samples are (1, -1) of (cb, cr), the predictions of cb and cr are calculated as follows:

[0463] pred_cb=1*JCCLM_predictor, pred_cr=-1*JCCLM_predictor+k

[0464] In the above equation, the value of k can refer to the above sub-embodiment. In another embodiment, when the weight used to generate the weighted reconstructed adjacent cb and cr samples is (1 / 2, 1 / 2) of (cb, cr), the prediction of cb and cr is calculated as follows.

[0465] pred_cb=1*JCCLM_predictor, pred_cr=1*JCCL M_predicter

[0466] In another embodiment, when JCCLM is applied, residual coding automatically uses JCCR.

[0467] JCCLM (JCCR with CCLM) - Method 2

[0468] JCCLM-Method 2 is proposed as a new LM derivation scheme. Different from the CCLM previously disclosed in the background section, two models are used to generate the prediction of the current block. The derivation process of these two models and their corresponding predictors are shown below:

[0469] - JCCLM: Neighboring luma reconstructed samples and weighted reconstructed neighboring cb and cr samples are used as inputs X and Y for model derivation. In the present invention, the derived model is called JCCLM and the model parameters are called JCCLM parameters. Then, the JCCLM predictor is determined based on the JCCLM parameters and the reconstructed samples of the co-located luma block.

[0470] - Cb_CCLM: Neighboring luma reconstructed samples and neighboring cb reconstructed samples are used as inputs X and Y for model derivation. In the present invention, the derived model is called cb_CCLM and the model parameters are called cb_CCLM parameters. Then, the cb_CCLM predictor is determined based on the cb_CCLM parameters and the reconstructed samples of the co-located luma block.

[0471] Finally, the predictions of cb and cr are calculated through the JCCLM predictor and the cb_CCLM predictor. Fig.32An example of the relationship between the cr prediction 3210, the cb prediction 3220 and the JCCLM predictor 3230 is shown.

[0472] In one embodiment, the weights used to generate the weighted reconstructed neighboring cb and cr samples may be (1 / 2, 1 / 2) of (cb, cr).

[0473] In another embodiment, the prediction of cb is calculated as follows:

[0474] pred_cb = cb_CCLM_predictors.

[0475] In another embodiment, the prediction of cr is calculated as follows:

[0476] pred_cr=2*JCCLM_predictor-cb_CCLM_prespector

[0477] In another embodiment, when JCCLM is applied, residual coding automatically uses JCCR.

[0478] Multiple assumptions for CCLM predictions

[0479] In addition to the CCLM previously disclosed in the background section (for cb, the model parameters are derived from brightness and cb; for cr, the model parameters are derived from brightness or cr), the present invention also discloses more CCLM variants. Some examples are given below.

[0480] - In one variant, cr prediction is derived by:

[0481] o Derive model parameters by using the neighboring reconstructed samples of cb and cr as input X and Y for model derivation

[0482] oThen, the cr predictions are generated using the derived model parameters and the cb reconstructed samples.

[0483] - In another variant, MMLM is used.

[0484] - In yet another variant, the model parameters for cb (or cr) prediction are derived from a plurality of co-located luma blocks.

[0485] Each CCLM method is suitable for different scenarios. For some complex features, combined predictions can produce better performance. Therefore, multi-hypothesis CCLM is disclosed to mix predictions from multiple CCLM methods. The CCLM methods to be mixed can be from (but not limited to) the above CCLM methods. A weighting scheme is used for mixing.

[0486] In one embodiment, the weights of different CCLM methods are predefined at the encoder and decoder.

[0487] In another embodiment, the weights vary based on the distance between the sample (or region) location and the reference sample location.

[0488] In another embodiment, the weights depend on neighboring codec information.

[0489] In another embodiment, the weight index is indicated or parsed. The code words can be fixed or adaptively changed. For example, the code words change with a template-based method.

[0490] Adaptive in-frame mode selection

[0491] As video codecs improve, more codecs are created. The syntax overhead of selecting a codec becomes an issue. Several simple methods can be used to reduce syntax overhead. For example, larger blocks can use the same codec mode. For another example, multiple components (such as CB and CR) can share the same codec mode.

[0492] However, using these simple methods, the accuracy / performance of intra-frame prediction decreases. Possible reasons are as follows:

[0493] - Intra-picture prediction is highly correlated with neighboring reference samples. When a single intra-picture prediction mode is used for the entire block, the intra-picture prediction model may be suitable for those samples close to the reference sample, but may not be suitable for those samples far from the reference sample.

[0494] -When processing CR, a reconstruction of CB and luminance is generated, which can be used to select the CR codec mode.

[0495] In this section, it is proposed to adaptively change the intra prediction mode of one or more samples or sub-blocks within the current block depending on the previous encoding / decoding of the components.

[0496] In one embodiment, the performance of different codec modes is determined by reconstruction of previously encoded / decoded components. The better mode is then used for the remaining components (components that are subsequently encoded and decoded). For example, for cb, if the prediction from a conventional intra-picture prediction mode (e.g., corner intra-picture prediction model, DC, planar) is better than the prediction from the LM mode (e.g., "better" means similar to the reconstruction of cb.), then the conventional intra-picture prediction mode is preferred for cr.

[0497] In a sub-embodiment, the proposed method may be sub-block based. For example, a chroma block is divided into several sub-blocks. For each sub-block, if for cb, the prediction of the sub-block from the LM mode is better than the sub-block prediction from a conventional intra-picture prediction mode (e.g., corner intra-picture prediction model, DC, planar) (e.g., "better" means similar reconstruction of cb and reduced residual of cb), then the LM mode is preferred for the corresponding sub-block of cr. For example, Fig.33 As shown, the chrominance block is divided into 4 sub-blocks. If sub-blocks 1 and 2 of the cb block 3310 use the LM mode to have a better prediction result, sub-blocks 1 and 2 of the cr block 3320 also use the LM mode.

[0498] In another embodiment, adaptive changing of rules may be performed at the encoder and / or decoder and no additional syntax is required.

[0499] Inverse LM

[0500] For the CCLM mode previously disclosed in the background section, the luma reconstructed samples are used to derive the predictors in the chroma blocks. In the present invention, inverse LM is proposed to use the chroma information to derive the predictors in the luma blocks. When inverse LM is supported, chroma is encoded / decoded (signed / parsed) before luma.

[0501] In one embodiment, the chroma information refers to the chroma reconstructed samples. When deriving the model parameters of the inverse LM, the reconstructed adjacent chroma samples are used as X and the reconstructed adjacent luminance samples are used as Y. In addition, the reconstructed samples in the chroma block (co-located with the current luminance block) and the derived parameters are used to generate the predictor in the current luminance block. Optionally, the "information" in this embodiment can refer to the prediction samples.

[0502] In one embodiment, chrominance refers to cb and / or cr components.

[0503] In one sub-embodiment, only one of the cb and cr information is used.

[0504] In another sub-embodiment, the chrominance information comes from cb and cr. For example, adjacent reconstructed cb and cr samples are weighted and then used as input to derive model parameters. For another example, reconstructed cb and cr samples in a chrominance block (co-located with the current luma block) are weighted and then used to derive the predictor in the current luma block.

[0505] In another embodiment, for the current luma block, the prediction (generated by the proposed inverse LM) may be combined with one or more hypotheses of prediction (generated by one or more other intra-picture prediction modes).

[0506] In one sub-embodiment, “other intra-picture prediction modes” may refer to angular intra-picture prediction model, DC, planar, MIP, ISP, MRL, any other existing intra-picture model (supported in HEVC / VVC) and / or any other intra-picture prediction model.

[0507] In another sub-embodiment, when combining multiple hypotheses of prediction, the weight of each hypothesis can be fixed or adaptively changed. For example, equal weight is applied to each hypothesis. For another example, the weight varies with neighboring codec information, sample position, block width, height, prediction mode or area. Some examples of the use of neighboring codec information are as follows:

[0508] -A possible rule related to sample location is described as follows:

[0509] o As the sample position moves further away from the reference sample, the weight of predictions from other intra-picture prediction modes decreases.

[0510] Another possible rule related to adjacent codec information is described as follows:

[0511] o When more neighboring blocks (left, above, top-left, top-right, and / or bottom-left) are encoded and decoded using a specific mode (e.g., mode A), the weight of the prediction from mode A becomes higher.

[0512] Another possible rule related to sample location is described as follows:

[0513] o The current block is split into several regions. Sample positions in the same region share the same weight. If the current region is close to the reference L, the weight of the prediction from other intra-picture prediction modes is higher than the weight of the prediction from CCLM. Some possible ways to split the current block are shown below (e.g. Fig.34A -Dotted line in C):

[0514] Fig.34A (The ratio of width to height is close to or exactly 1:1): Consider the distance between the current area and the left and top reference L neighbors.

[0515] Fig.34B (width > n*height, where n can be any positive integer): Consider the distance between the current region and the top reference L neighbor.

[0516] Fig.34C (height > n*width, where n can be any positive integer): Consider the distance between the current region and the left reference L neighbor.

[0517] CCLM of inter-screen blocks

[0518] As mentioned before, the cross-color tool is used for intra-picture blocks to improve the chroma intra-picture prediction. For inter-picture blocks, the chroma prediction may not be as accurate as the luma prediction. Possible reasons are listed below:

[0519] - The motion vectors for chroma components are inherited from luma (chroma does not have its own motion vectors).

[0520] - Fewer codec tools are set up to improve inter-picture chroma prediction.

[0521] Therefore, it is suggested to use one or more cross-color tools as an optional way to encode and decode inter-picture blocks. The following takes the cross-color tool as CCLM as an example. The proposed method is not limited to using CCLM as a cross-color tool only, and can be applied to and / or combined with all or any subsets of cross-color tools. Using the proposed method, the chrominance prediction of the inter-picture block can be improved based on the corresponding brightness. According to the CCLM for the inter-picture block (referred to as the inter-picture CCLM in the present invention), the corresponding brightness block is encoded and decoded in the inter-picture mode, that is, using motion compensation and one or more motion vectors to access the previously reconstructed brightness block in one or more previously encoded and decoded reference frames. Compared with the inter-picture prediction based on the previously reconstructed chrominance block in one or more previously encoded and decoded reference frames, the cross-color linear mode based on the brightness of the inter-picture codec can provide better prediction. The CCLM for the intra-picture mode has been described previously. The previously described CCLM processing can be applied here. However, while conventional CCLM utilizes reconstructed luminance blocks where reference samples (used to predict or reconstruct the luminance blocks) are located in the current frame, CCLM inter-frame mode utilizes reconstructed or predicted luminance blocks where reference samples (used to predict or reconstruct the luminance blocks) are located in one or more previously encoded and decoded reference frames.

[0522] In one embodiment, for chroma components, in addition to or instead of the original inter-picture prediction (generated by motion compensation from the target inter-picture mode), one or more hypotheses of cross-color prediction (generated by any cross-color tool such as CCLM and / or any other LM mode) are used to form the current prediction. Cross-color tool refers to the use of more than one color information to generate one or more cross-color predictions by using a predefined generation method. The use of more than one color information and the predefined generation method of CCLM are described in the subsection entitled CCLM. For CCLM, as shown in equation (1), model parameters a and b are derived based on the reconstructed neighboring luma and chroma samples, and the derived model parameters are applied to the reconstructed luma samples in the co-located luma block. In this case, the number of model parameters (including only a and b) is 2, and for generating the predicted value of (i, j) in the current chroma block, only one down-sampled co-located luma sample of (i, j) is used and combined with the model parameters. The present invention is not limited to setting the number of model parameters to 2 and using only one downsampled co-located luma sample for generating predicted values ​​at positions in chroma blocks. The number of model parameters may be any predefined number, the co-located luma samples may not be downsampled, and / or the number of co-located luma samples (to be combined with model parameters) may be any predefined number. For more variants of the cross-color tool, one or more color information includes reconstructed samples of a co-located first color (e.g., luma and / or first chroma component) block, reconstructed samples of a predefined neighboring area of ​​the co-located first color block, and / or reconstructed samples of a predefined neighboring area of ​​a second color (e.g., current chroma component) block. For example, a predefined neighboring area of ​​the co-located first color block and a predetermined neighboring area of ​​the second color block are used to derive model parameters, and the derived model parameters are applied to the reconstructed samples of the co-located first color block to generate a cross-color prediction of the current second color block. For the optional prediction-based cross-color tool, the one or more color information may be any subset of the above information, and / or may also include predicted samples using a target inter-picture mode for a co-located first color block, predicted samples from a target intra-picture mode from a predefined neighboring area of ​​a co-located first color (e.g., luminance) block, and / or predicted samples using a target inter-picture mode for a predefined neighboring area of ​​a second color (e.g., current chrominance component) block. For example, the predicted samples using the target inter-picture mode for the co-located first color block and the predicted samples using the target inter-picture mode for the second color block are used to derive model parameters, and the derived model parameters are applied to the reconstructed samples of the co-located first color block to generate a cross-color prediction for the current second color block. The use of the one or more color information may depend on the block width, block height, and / or block area.If the block width / block height / block area is less than a predefined threshold, in addition to using the mentioned samples (e.g., predicted samples of the co-located first color block and predicted samples of the second color block) to derive model parameters, neighboring predicted or reconstructed samples are also used. The predefined neighboring area may include a top neighboring area with N rows, a left neighboring area with M columns, an upper left neighboring area with M×N samples, or any subset of the above areas. More cross-color tools can be found in the paragraphs mentioning the term "LM" in this disclosure.

[0523] In one sub-embodiment, the current prediction is a weighted sum of the inter-picture prediction and the CCLM prediction. The weights are designed based on neighboring codec information, sample positions, block width, height, mode or area. Some examples are shown below:

[0524] - In one example, for small blocks (eg, area < threshold), the weight for CCLM prediction is higher than that for inter-picture prediction.

[0525] - In another example, when most of the neighboring coded blocks are intra-picture blocks, the weight for CCLM prediction is higher than that for inter-picture prediction.

[0526] - In yet another example, the weight is a fixed value that is used for the entire block.

[0527] In another embodiment, the inter-picture prediction may be generated by any of the above inter-picture modes. For example, the inter-picture mode may be a regular merge mode. For another example, the inter-picture mode may be a CIIP mode. For another example, the inter-picture mode may be a CIIP PDPC. For another example, the inter-picture mode may be a GPM or any GPM variant (e.g., GPM intra-picture).

[0528] In one sub-embodiment, the normal merge mode is to select a merge candidate from a merge candidate list having a marked merge index.

[0529] In another sub-embodiment, the normal merge mode may be MMVD.

[0530] In another sub-embodiment, the LM mode used in inter-picture CCLM is prediction-based LM.

[0531] In another embodiment, inter-picture CCLM is supported only when any one (or more than one) of the predefined inter-picture modes is used for the current block, or inter-picture CCRM is supported when any one (or more than one) of the enable flags of the predefined inter-picture modes is indicated as enabled. Supporting inter-picture CCLM means that the prediction of the current block can be selected between applying inter-picture CCRM or not applying inter-picture CCLM.

[0532] In one sub-embodiment: one or more hypotheses of the prediction (generated by CCLM and / or any other LM mode) are mixed with the original inter-frame prediction. When inter-frame CCLM is applied, the prediction of the current block is generated by:

[0533] Mixing the chrominance prediction from existing inter-picture modes with the prediction from LM

[0534] oMixing: Predfinal=(wmerge*PredInter+wLM*PredLM+2)>>2

[0535] oWeighted rule: w Inter and w LM ,

[0536] For example:

[0537] ●If the top and left sides are both within the frame, then (w Inter , w LM )=(1,3)

[0538] ● Otherwise, if one of the top and left sides is within the frame, then (w Inter , w LM )=(2,2)

[0539] ●Otherwise, (w Inter , w LM )=(3,1)

[0540] ■For another example, the weights follow the CIIP weights.

[0541] ●For example, pred Inter = Inter-picture prediction after OBMC (if OBMC is used)

[0542] ●For example, pred Inter =Inter-picture prediction before OBMC (OBMC can be applied after blending)

[0543] In another sub-embodiment, the original inter-picture prediction is replaced by one or more hypotheses of the prediction (generated by CCLM and / or any other LM mode).

[0544] When inter-picture CCLM is not applied, the prediction of the current block comes from the original inter-picture prediction.

[0545] In another embodiment, whether to apply CCLM between pictures depends on a flag, which can be located at TU / TB, CU / CB, PU / PB or CTU / CTB.

[0546] In one sub-embodiment, one or more flags are indicated in the bitstream to indicate whether CCLM inter-picture is applied. For example, the flag is contextually coded. For another example, only one context is used to code the flag. For another example, multiple contexts are used to code the flag, and the selection of the context depends on the block width, block height, block area, or neighboring mode information.

[0547] In another sub-embodiment, when the flag indicates that CCLM inter-frame is applied, an additional flag is used to select one or more LMs from the total candidate LM modes (e.g., CCLM_LT, CCLM_L, CCLM_T, MMLM_L, or any subset / extension from the above modes). For example, if one LM mode is selected, the LM prediction is generated by the selected LM. For another example, if more than one LM mode is selected, the LM prediction is generated by mixing assumptions of predictions from multiple LM modes. For another example, the additional flag refers to an index in the bitstream that can be truncated unary codec with context and / or without context.

[0548] In another sub-embodiment, when the flag indicates that CCLM inter-frame is applied, one or more LMs are implicitly selected (or predefined) from the total candidate LM modes (e.g., CCLM_LT, CCLM_L, CCLM_T, MMLM_L or any subset / extension from the above modes) for CCLM inter-frame.

[0549] For example, CCLM_LT is used to generate LM prediction for inter-picture CCLM. For another example, MMLM_LT is used to generate LM prediction for inter-picture CCLM. For another example, as shown below, the predefined rule depends on the block width, block height or block area.

[0550] - Boundary matching settings (used as predefined rules) can be applied only if the block width, block height or block area is larger than a threshold.

[0551] - Boundary matching settings (used as predefined rules) can be applied only if the block width, block height or block area is smaller than a threshold.

[0552] - When the block width, block height or block area is smaller than a threshold, the selected LM mode is inferred to be any one (or more than one) LM mode from the total candidate LM modes.

[0553] oThe selected LM mode is fixed to CCLM_LT.

[0554] oThe selected LM mode is fixed to MMLM_LT.

[0555] For another example, the predefined rules depend on the boundary matching settings. The details of the boundary matching settings will be described later in the boundary matching settings section. The candidate mode used in the boundary matching settings section refers to each candidate LM mode for CCLM inter-picture. The prediction from the candidate mode used in the boundary matching settings section refers to the prediction generated by each candidate LM mode, or refers to a hybrid prediction from each candidate LM mode and the original inter-picture mode.

[0556] In another embodiment, CCLM inter-picture is supported only when the size condition of the current block is met.

[0557] In one sub-embodiment, the size condition is that the block width, block height or block area is greater than a predefined threshold value. The predefined threshold value may be a positive integer, such as 8, 16, 32, 64, 128, 256, etc.

[0558] In another sub-embodiment, the size condition is that the block width, block height or block area is less than a predefined threshold. The predefined threshold can be a positive integer, such as 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, etc.

[0559] In another embodiment, the inter-picture mode used in the inter-picture block depends on the enable flag. For example, if the inter-picture mode is conventional merging, the enable flag is called the conventional merging flag. For example, if the inter-picture mode is CIIP, the enable flag is called the CIIP flag. For another example, if the inter-picture mode is CIIP PDPC, the enable flag is called the CIIPPDPC flag. For another example, an enable flag indicated as enabled (e.g., a flag value equal to 1) means that the corresponding inter-picture mode is applied to the current block. For another example, an enable flag indicated as disabled (e.g., a flag value equal to 0) means that the corresponding inter-picture mode is not applied to the current block. For another example, the enable flag is indicated in the bitstream and / or inferred in some cases. For another example, the enable flag target indication depends on the block width, block height or block area.

[0560] In another embodiment, the prediction from the inter-picture can be adjusted by neighboring reconstructed samples and a predefined weighting scheme. For example, when encoding and decoding the current block in merge mode, the prediction from the merge is mixed with the neighboring reconstructed samples. For another example, the proposed scheme is enabled according to the CIIP PDPC flag, wherein the CIIP PDP flag is indicated when the CIIP flag is indicated as enabled. For another example, the predefined weighting scheme follows the PDPC weights.

[0561] An example describing the detailed processing is as follows:

[0562] —Use the above R x,-1 and left R-1,y Reconstructed samples of the conventional merge mode are refined to refine the inter-picture predictor

[0563] The derivation of -nScaIe and wT&wL is the same as that in the in-plane mode

[0564] -wT=32>>((y'<<1)>>nScale)

[0565] -wL=32>>((x'<<1)>>nScale)

[0566] -nScale=(floorLog2(width)+floorLog2(height)-2)>>2;

[0567] -CIIP PDPC:

[0568] - If LMCS is enabled, the inter-picture predictor is computed in the mapped domain

[0569] -Pred(x,y)=((((wT×R x，-1 +wL×R -1，y + 32 )>>6)<<6)+(64-wT-wL)×Fwd(predInter(x,y))+32)>>6

[0570] Otherwise, compute the inter-picture predictor in the original domain

[0571] -Pred(x,y)=((((wT×R x，-1 +wL×R -1，y +32)>>6)<<6)+(64-wT-wL)×predInter(x, y)+32)>>6

[0572] Otherwise, compute the inter-picture predictor in the original domain

[0573] -Pred(x,y)=((((wT×R x，-1 +wL×R -1，y +32)>>6)<<6)+(64-wT-wL)×predInter(x, y)+32)>>6

[0574] - When the CIIP flag is true, the CIIP PDPC flag is further indicated to indicate whether the CIIP PDP is used.

[0575] In another embodiment, the original inter-picture prediction (generated by motion compensation) is used for luma and the prediction for chroma components is generated by CCLM and / or any other LM mode.

[0576] In one sub-embodiment, the current CU is considered to be an inter-CU, an intra-CU, or a new prediction mode (neither intra nor inter).

[0577] The above proposed method can also be applied to IBC blocks ("inter-picture" in this section can be changed to IBC or any other non-intra-picture mode). For example, the target inter-picture mode is changed to the target IBC mode. As mentioned above, with respect to the mechanism of intra-block copying, the prediction generated by the target mode is copied from the reconstructed samples in the current frame using a block vector, wherein the block vector is indicated / parsed at the encoder / decoder and / or is implicitly derived. For example, the block vector is implicitly determined by finding the best prediction block from the reconstructed part of the current frame using template matching within a predefined search range. For example, the template is an L-shaped neighboring area, and template matching means finding the template that is most similar to the template of the current block in the search range. When the proposed method is applied to blocks encoded and decoded in the target IBC mode, for chroma components, the block vector prediction can be combined or replaced by cross-color prediction.

[0578] Boundary-matching setting

[0579] When using the boundary matching setting, such as Fig.35 As shown in the figure, the boundary matching cost of the candidate mode refers to the discontinuity measurement (including top boundary matching and / or left boundary matching) between the current prediction generated from the candidate mode (i.e., the predicted samples in the current block) and the adjacent reconstruction (i.e., the reconstructed samples in one or more adjacent blocks), where pred i，j Refers to the predicted block, reco i，j Refers to the adjacent reconstructed block, and block 3510 (as shown in the bold box) corresponds to the current block. Top boundary matching refers to the comparison between the current top predicted sample and the adjacent top reconstructed sample, while left boundary matching refers to the comparison between the current left predicted sample and the adjacent left reconstructed sample.

[0580] In another sub-embodiment, the candidate pattern with the smallest boundary matching cost is applied to the current block.

[0581] In another embodiment, the boundary matching costs of cb and cr can be added to become the boundary matching cost of chrominance, thereby sharing the selected candidate modes of cb and cr. Therefore, the selected candidate modes of cb and cr will be the same.

[0582] In another embodiment, the selected candidate modes of cb and cr depend on the boundary matching costs of cb and cr, respectively, and thus the selected candidate modes of cb and cr may be the same or different.

[0583] In one embodiment, the predefined subset of the current prediction is used to calculate the boundary matching cost. The n lines of the top boundary in the current block and / or the m lines of the left boundary in the current block are used. In addition, the n2 lines of the top adjacent reconstruction and / or the m2 lines of the left adjacent reconstruction are used.

[0584] In the example of calculating the boundary matching cost, n=2, m=2, n2=2, and m2=2:

[0585]

[0586] In the above formula, the weights (a, b, c, d, e, f, g, h, i, j, k, l) can be any positive integers, for example, a=2, b=1, c=1, d=2, e=1, f=1, g=2, h=1, i=1, j=2, k=1 and l=1.

[0587] In another example of calculating the boundary matching cost, n=2, m=2, n2=1 and m2=1:

[0588]

[0589] In the above formula, the weights (a, b, c, g, h, and i) can be any positive integers, for example, a=2, b=1, c=1, g=2, h=1, and i=1.

[0590] In yet another example of calculating the boundary matching cost, n=1, m=1, n2=2 and m2=2:

[0591]

[0592] In the above formula, the weights (d, e, f, j, k and l) can be any positive integers, for example, d=2, e=1, f=1, j=2, k=1 and l=1.

[0593] In yet another example of calculating the boundary matching cost, n=1, m=1, n2=1 and m2=1:

[0594]

[0595] In the above formula, the weights (a, c, g, and i) can be any positive integers, for example, a=1, c=1, g=1, and i=1.

[0596] In yet another example of calculating the boundary matching cost, n=2, m=1, n2=2 and m2=1:

[0597]

[0598] In the above formula, the weights (a, b, c, d, e, f, g and i) can be any positive integers, for example, a=2, b=1, c=1, d=2, e=1, f=1, g=1 and i=1.

[0599] In yet another example of calculating the boundary matching cost, n=1, m=2, n2=1 and m2=2:

[0600]

[0601] In the above formula, the weights (a, c, g, h, i, j, k and l) can be any positive integers, for example, a=1, c=1, g=2, h=1, i=1, j=2, k=1 and l=1.

[0602] The following examples of n and m are also applicable to n2 and m2.

[0603] For example, n can be any positive integer, such as 1, 2, 3, 4, etc.

[0604] For example, m can be any positive integer, such as 1, 2, 3, 4, etc.

[0605] For example, n and / or m vary with block width, height, or area. In one embodiment, as the block gets larger, m becomes larger (e.g., area>threshold 2).

[0606] oThreshold 2 = 64, 128 or 256.

[0607] o When area > threshold 2, m is increased to 2 (originally, m is 1).

[0608] o When area > threshold 2, m is increased to 4 (originally, m is 1 or 2).

[0609] For example, as the block is taller, m becomes larger and / or n becomes smaller (e.g., height > threshold 2*width).

[0610] oThreshold 2 = 1, 2 or 4.

[0611] o When height > threshold 2*width, m is increased to 2 (originally, m is 1).

[0612] o When height > threshold 2*width, m is increased to 4 (originally, m is 1 or 2).

[0613] In another embodiment, n becomes larger for larger blocks (area>threshold 2).

[0614] oThreshold 2 = 64, 128 or 256.

[0615] o When area > threshold 2, n is increased to 2. (Originally, n is 1.)

[0616] o When area > threshold 2, n is increased to 4. (Originally, n is 1 or 2.)

[0617] In another embodiment, for wider blocks (width>threshold 2*height), n becomes larger and / or m becomes smaller. For example,

[0618] oThreshold 2 = 1, 2 or 4.

[0619] o When width > threshold 2*height, n is increased to 2 (originally, n is 1).

[0620] o When width > width2*height, n is increased to 4 (originally, n is 1 or 2).

[0621] Cross-CU LM

[0622] Compared with the conventional intra-picture prediction modes (e.g., angular intra-picture prediction mode, DC and planar), Fig.36 As shown, the advantage of LM mode is to predict irregular patterns, where the block has an irregular pattern and no corner intra-picture prediction mode can provide good prediction. However, using LM mode, the luma block 3610 can provide good prediction for the chroma block 3620.

[0623] For the encoding / decoding of irregular patterns in inter-pictures, the distribution of intra-picture and inter-picture encoding / decoding modes can be as follows: For some areas (highly correlated with neighboring areas), the intra-picture mode is used. For other areas, the inter-picture mode is preferred.

[0624] To handle the above situation, a cross-CU LM ​​mode is proposed. Based on the observation of the current CU ancestor node, the LM mode is adopted. For example, if the ancestor node contains irregular patterns (e.g., partly intra-picture and partly inter-picture), the blocks belonging to the ancestor node are encoded / decoded using the LM mode. With the proposed method, the CU-level on / off flag of the LM mode is not required. Fig.37 The luminance image area associated with the node containing the irregular pattern is marked. According to the irregular pattern, the area associated with the node is divided into a plurality of luminance blocks. The luminance block (dashed line block) where the irregular pattern occupies a significant part of the block is processed as an intra-screen block; otherwise, the luminance block (dashed line block) is processed as an inter-screen luminance block.

[0625] In one embodiment, the block-level on / off flag of the LM mode is defined / indicated at the ancestor node level. For example, when the flag at the ancestor node indicates that cross-CU LM ​​is enabled, the CUs belonging to the ancestor node (i.e., those split from the ancestor node) use LM. For another example, when the flag at the ancestor node indicates that cross-CU LM ​​is disabled, the CUs belonging to the ancestor node (i.e., those split from the ancestor node) do not use LM.

[0626] In another embodiment, the ancestor node refers to a CTU.

[0627] In another embodiment, whether to enable cross-CU LM ​​is implicitly derived from analysis of block properties of ancestor nodes.

[0628] In this section, CU can be changed to any block. For example, it can be PU.

[0629] LM auxiliary angle / plane mode

[0630] For traditional intra prediction modes (e.g., corner intra prediction model, DC, and planar), the reference samples come from the top and left neighboring reconstructed samples. Therefore, for the bottom right sample in the current block, the accuracy of intra prediction decreases. In this section, LM is used to improve the prediction from traditional intra prediction modes.

[0631] In one embodiment, the prediction of the current block is formed by a weighted sum of one or more hypotheses of prediction from a traditional intra-picture prediction mode and one or more hypotheses of prediction from a LM mode. In one sub-embodiment, both are applied with equal weights. In another sub-embodiment, the weights vary with adjacent codec information, sample position, block width, height, mode, or area. For example, when the sample position is away from the upper left region, the weight of the prediction from the traditional intra-picture prediction mode decreases. More weighting schemes can be referred to the "Inverse LM" section.

[0632] In another embodiment, it is proposed to use LM mode to generate a lower right region within or near the current block. When performing intra-picture prediction, the reference samples can be based not only on the original left and top neighboring reconstructed samples, but also on the proposed right and bottom LM predicted samples. An example is shown below.

[0633] - Reconstruct the co-located luma block before intra-prediction of the chroma block.

[0634] -“The adjacent luma reconstructed samples of the co-located luma block” and “the adjacent chroma reconstructed samples of the current chroma block” are used to derive LM parameters.

[0635] - The "reconstructed samples of the co-located luma block" with the derived parameters are used to obtain the lower right LM predicted samples of the current chroma block. The lower right area of ​​the current chroma block can be Figure 38A-Figure 38B Any subset of the area shown. Fig.38A An example in which the lower right area 3812 is located outside and adjacent to the current chroma block 3810 is marked. Fig.38B An example where the lower right area 3822 is within the current chroma block 3820 is marked.

[0636] - Generate prediction of the current block bidirectionally by referring to the original L neighboring region (the original upper left region, obtained using the traditional intra prediction mode) and the proposed inverse L region (obtained using LM).

[0637] In one sub-embodiment, the predictor from the original upper left region is combined with the predictor from the lower left region using weights. In one example, equal weights are applied to both. In another example, the weights vary with neighboring codec information, sample position, block width, height, mode, or area. For example, when the sample position is far away from the upper left region, the weight of the prediction from the traditional intra-picture prediction mode is reduced.

[0638] In another embodiment, the proposed method can be applied to inverse LM. Then, when performing luma intra prediction, the final prediction is bidirectional, which is similar to the above example of chroma blocks.

[0639] In another embodiment, after segmentation is performed to understand the curve pattern of luminance, the proposed LM-assisted angle / plane mode assists chrominance in obtaining the correct bending angle.

[0640] The methods proposed in the present invention may be enabled and / or disabled according to implicit rules (eg, block width, height or area) or according to explicit rules (eg, syntax at block, slice, picture, SPS or PPS level).

[0641] The term "block" in the present invention may refer to TU / TB, CU / CB, PU / PB, or CTU / CTB.

[0642] The term "LM" in the present invention can be regarded as a CCLM / MMLM mode or any other extension / variant of CCLM (for example, the CCLM extension / variant proposed in the present invention). Although the linear model for cross-component prediction has been used to illustrate the embodiments of the present invention shown above, the present invention is not limited to linear models. On the contrary, any cross-component (cross-color) prediction model can be used to implement the present invention. An example in which the cross-component mode is a convolutional cross-component mode (Convolutional Cross-component Mode, CCCM) is shown below. The CCCM mode can be regarded as an optional mode of CCLM. When the optional mode is applied to the current block, cross-component information with one or more models, including linear terms and / or nonlinear terms, is used to generate chrominance prediction. Similar to CCLM, the optional mode of CCCM can follow the template selection of CCLM, and therefore, the CCCM family includes CCCM_LT CCCM_L and / or CCCM_T. In addition, the optional mode of CCCM uses a single model or a multi-model variant of CCCM.

[0643] The method proposed in the present invention (for CCLM) can be used for any other LM mode.

[0644] Any combination of the methods proposed in the present invention can be applied. In the example of combining the proposed method in the cross-color tool to improve inter-block or intra-block replication, and the above-mentioned sub-division scheme, when the block width, block height or block area of ​​the current second color (chrominance) block is greater than a predefined threshold, the current chrominance block is first divided into sub-divisions, and each sub-division can use the method proposed in the subsection entitled CCLM of inter-screen blocks to generate hypotheses for cross-color prediction. For the sub-division of the current chrominance block, the model parameters are derived by using the reconstructed samples adjacent to the current sub-division, the reconstructed samples adjacent to the co-located first color block, the predicted samples in the current sub-division (generated by the target mode), the predicted samples in the co-located first color block (generated by the target mode) or the reconstructed samples, or any subset or extension of the above samples. As mentioned in the sub-division scheme in the subsection entitled intra-screen sub-division, the minimum sub-division size is predefined, and the width, height or area of ​​the sub-division cannot be less than the predefined sub-division size. When the current block is split into sub-divisions and the minimum sub-division size is met, the division is terminated.

[0645] Any of the cross-component methods proposed above (e.g., CCLM and CCLM inter-picture methods) can be implemented in an encoder and / or a decoder. For example, any of the proposed methods can be implemented in an intra-picture / inter-picture coding / decoding module of an encoder (e.g., Figure 1A Intra-picture prediction 110 and inter-picture prediction 112 in the image processing module), motion compensation module (e.g. Figure 1BOptionally, any of the proposed methods may be implemented as a circuit coupled to an intra-picture / inter-picture coding / decoding module and / or a motion compensation module of an encoder and a merge candidate derivation module of a decoder.

[0646] Fig.39 A flowchart of an exemplary video codec system that mixes a linear model predictor with an inter-picture mode predictor according to an embodiment of the present invention is shown. The steps shown in the flowchart can be implemented as executable program codes on one or more processors (e.g., one or more CPUs) on the encoder side. The steps shown in the flowchart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to execute the steps in the flowchart. According to the method, in step 3910, input data related to a current block including a first color block and a second color block is received. The input data includes pixel data of the current block to be encoded on the encoder side or encoded data associated with the current block to be decoded on the decoder side; wherein the first color block is encoded and decoded in a target mode, and the target mode refers to an inter-picture mode or an intra-picture block copy mode. In step 3920, one or more model parameters of one or more cross-color models related to the first color block and the second color block are determined. In step 3930, a cross-component predictor for the second color block is derived by applying one or more cross-color models to the reconstructed or predicted first color pixels corresponding to the first color block. In step 3940, a final predictor for the second color block is derived by using the cross-component predictor or combining the cross-component predictor with the target mode predictor for the second color block. In step 3950, the second color block is encoded or decoded by using the prediction data including the final predictor.

[0647] The flowcharts shown in the present invention are used to illustrate examples of video encoding according to the present invention. Without departing from the spirit of the present invention, a person of ordinary skill in the art can modify each step, reorganize the steps, separate a step, or combine the steps to implement the present invention. In the present invention, specific syntax and semantics have been used to illustrate different examples to implement embodiments of the present invention. Without departing from the spirit of the present invention, a person of ordinary skill in the art can implement the present invention by replacing the syntax and semantics with equivalent syntax and semantics.

[0648] The above description enables the ordinary general knowledge of the art to implement the present invention in the content of a specific application and its requirements. For the ordinary knowledge of the art, the various deformations of the described embodiments will be apparent, and the general principles defined herein can be applied in other embodiments. Therefore, the present invention is not limited to the specific embodiments shown and described, but will be given the maximum scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are described to thoroughly understand the present invention. Nevertheless, it will be understood by the ordinary knowledge of the art that the present invention can be put into practice.

[0649] The embodiments of the present invention as described above can be implemented in various hardware, software codes or a combination of the two. For example, an embodiment of the present invention can be a circuit integrated in a video compression chip, or a program code integrated into video compression software to perform the processing described herein. An embodiment of the present invention can also be a program code executed on a digital signal processor (Digital Signal Processor, DSP) to perform the processing described herein. The present invention can also include several functions executed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array (FieldProgrammable Gate Array, FPGA). According to the present invention, these processors can be configured to perform specific tasks by executing machine-readable software code or firmware code that defines the specific method implemented by the present invention. The software code or firmware code can be developed in different programming languages ​​and different formats or styles. The software code can also be compiled for different target platforms. However, different code formats, styles and languages ​​of software codes and other forms of configuration codes for performing the tasks of the present invention will not deviate from the spirit and scope of the present invention.

[0650] The invention may be implemented in other specific forms without departing from its spirit or essential characteristics. The examples described are illustrative in all respects and are not restrictive. Therefore, the scope of the invention is indicated by the additional claims rather than the foregoing description. The meaning of the claims and all changes within the same scope should be included within their scope.

Claims

1. A color picture prediction method, comprising: Receiving input data related to a current block including a first color block and a second color block, wherein the input data includes pixel data of the current block to be encoded at an encoder side or encoded data related to the current block to be decoded at a decoder side, wherein the first color block is encoded or decoded in a target mode, the target mode being an inter-picture mode or an intra-picture block copy mode; determining one or more model parameters of one or more cross-color models associated with the first color block and the second color block; deriving a cross-component predictor for the second color block by applying the one or more cross-color models to corresponding reconstructed or predicted first color pixels of the first color block; deriving a final predictor for the second color block by using the cross-component predictor or combining the cross-component predictor with a target mode predictor for the second color block; as well as The second color block is encoded or decoded by using the prediction data including the final predictor.

2. The method of claim 1, wherein: The one or more model parameters of the one or more cross-color models are derived by using the adjacent reconstructed first color samples of the first color block and the adjacent reconstructed second color samples of the second color block.

3. The method of claim 1, wherein: The one or more model parameters of the one or more cross-color models are derived by using the neighboring predicted first color samples of the first color block and the neighboring predicted second color samples of the second color block.

4. The method of claim 1, wherein: The cross-component predictor is selected from a set of cross-component patterns.

5. The method of claim 4, wherein: The cross-component mode set includes a combination of all or any subsets of CCLM_LT mode, CCLM_L mode, CCLM_T mode, MMLM_LT mode, MMLM_L mode and MMLM_T mode.

6. The method of claim 4, wherein: The cross-component predictor is selected from the cross-component pattern set according to a hint rule.

7. The method of claim 6, wherein: The hint rule is related to the block width, block height or block area of ​​the current block.

8. The method of claim 6, wherein: The implicit rule is inferred to be predefined.

9. The method of claim 4, wherein: The cross-component predictor is selected based on one or more explicit indices.

10. The method of claim 1, wherein: The target mode predictor is derived using Combined Inter Merge and Intra Prediction (CIIP), CIIP with Template Matching (CIIP-TM), or CIIP with Position Dependent IntraPrediction Combination (CIIP PDPC).

11. The method of claim 1, wherein: The first color block corresponds to a luminance block, and the second color block corresponds to a chrominance block.

12. The method of claim 1, wherein: The final predictor for the second color block is derived using a weighted sum of the cross-component predictor and the target mode predictor.

13. The method of claim 12, wherein: One or more weights of the weighted sum of the cross-component predictor and the target mode predictor are selected according to the coding mode of one or more neighboring blocks of the current block.

14. The method of claim 13, wherein: The one or more neighboring blocks correspond to at least one of a top neighboring block and a left neighboring block.

15. The method of claim 1, wherein: When the current block is encoded and decoded with Combined Inter Merge and Intra Prediction (CIIP), one or more flags are marked or parsed from the bitstream to indicate whether cross-component processing is applied to the current block; The cross-component processing includes deriving a cross-component predictor, deriving a final predictor, and encoding or decoding the second color block by using prediction data including the final predictor.

16. The method of claim 15, wherein: When the cross-component processing is applied to the current block, the cross-component predictor is inferred to be derived according to a predefined cross-component mode.

17. The method of claim 15, wherein: When the cross-component processing is applied to the current block, the cross-component predictor is inferred as the target cross-component mode with the minimum boundary matching cost in the set of candidate cross-component modes.

18. A color picture prediction device, comprising: one or more electronic devices or processors, Arranged as: Receiving input data related to a current block including a first color block and a second color block, wherein the input data includes pixel data of the current block to be encoded at an encoder side or encoded data related to the current block to be decoded at a decoder side, wherein the first color block is encoded and decoded in a target mode, the target mode being an inter-picture mode or an intra-picture block copy mode; determining one or more model parameters of one or more cross-color models associated with the first color block and the second color block; deriving a cross-component predictor for the second color block by applying the one or more cross-color models to corresponding reconstructed or predicted first color pixels of the first color block; deriving a final predictor for the second color block by using the cross-component predictor or combining the cross-component predictor with a target mode predictor for the second color block; as well as The second color block is encoded or decoded by using the prediction data including the final predictor.

Images