Prediction improvements with convolutional models

Abstract

A method for deriving and using a cross-component model to improve intra or inter prediction when encoding or decoding a block of pixels is provided. A video codec receives pixel block data to be encoded or decoded as a current block of a current picture of a video. A video codec generates an initial predictor for a current block based on a reference block. A video codec receives pixels of a first set of component samples within or adjacent to a reference block. The video codec receives pixels of a second set of component samples within or adjacent to the current block. The video codec derives a model based on the component samples of the first and second sets. The video codec applies the resulting model to the initial predictor to generate an improved predictor. A video codec decodes or encodes a current block by using the improved predictor.

Description

[0001] Cross-references

[0002] This disclosure is part of a non-provisional application claiming the benefit of priority to U.S. Provisional Patent Application No. 63 / 340,978, filed on May 12, 2022, the contents of which are incorporated herein by reference. Technical Field

[0003] The present invention generally relates to video coding. In particular, the present disclosure relates to a method for encoding a block by inter-prediction refinement. Background Art

[0004] Unless otherwise indicated herein, the methods described in this section are not prior art in the scope of the following patent claims and are not admitted to be prior art by inclusion of this section.

[0005] High-Efficiency Video Coding (HEVC) is an international video coding standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). HEVC is a hybrid block-based motion compensation DCT transform coding architecture. The basic unit for compression (called coding unit (CU)) is a 2Nx2N square block, and each CU can be recursively divided into four smaller CUs until a predefined minimum size is reached. Each CU contains one or more prediction units (PUs).

[0006] Versatile video coding (VVC) is the latest international video coding standard developed by the Joint Video Experts Team (JVET) of ITU-T SG16 WP3 and ISO / IECJTC1 / SC29 / WG11. The input video signal is predicted based on the reconstructed signal derived from the coded and decoded picture area. The prediction residual signal is processed by block transform. The transform coefficients are quantized and entropy encoded together with other side information in the bitstream. The reconstructed signal is generated based on the prediction signal and the reconstructed residual signal after the dequantized transform coefficients are inversely transformed. The reconstructed signal is further processed by in-loop filtering to remove coding artifacts. The decoded pictures are stored in a frame buffer for prediction of future pictures in the input video signal.

[0007] In VVC, the coded picture is partitioned into non-overlapping square block areas represented by associated coding tree units (CTUs). The leaf nodes of the coding tree correspond to coding units (CUs). The coded picture can be represented by a set of slices, each slice including an integer number of CTUs. Individual CTUs in a slice are processed in raster scan order. Bidirectional prediction (B) slices can be decoded using intra prediction or inter prediction with up to two motion vectors and reference indices to predict sample values ​​for each block. Prediction (P) slices are decoded using intra prediction or inter prediction with up to one motion vector and reference index to predict sample values ​​for each block. Intra (I) slices are decoded using only intra prediction.

[0008] A quadtree (QT) with a nested multi-type-tree (MTT) structure can be used to partition a CTU into one or more non-overlapping codec units (CUs) to accommodate various local motion and texture features. The CU can be further divided into smaller CUs using one of the following five partition types: quadtree partitioning, vertical binary tree partitioning, horizontal binary tree partitioning, vertical center-side ternary tree partitioning, and horizontal center-side ternary tree partitioning.

[0009] Each CU contains one or more prediction units (PUs). The prediction unit, together with the associated CU syntax, is used as a basic unit to signal predictor information. A specified prediction process is used to predict the values ​​of associated pixel samples within the PU. Each CU may contain one or more transform units (TUs) for representing prediction residual blocks. A transform unit (TU) includes a transform block (TB) of luminance samples and two corresponding transform blocks of chrominance samples, and each TB corresponds to a residual block of samples from one color component. An integer transform is applied to the transform block. The horizontal values ​​of the quantized coefficients are entropy encoded in the bitstream together with other auxiliary information. The terms coding tree block (CTB), coding block (CB), prediction block (PB), and transform block (TB) are defined to specify a 2D sample array of a color component associated with a CTU, CU, PU, ​​and TU, respectively. Therefore, a CTU includes a luminance CTB, two chrominance CTBs, and associated syntax elements. Similar relationships are valid for CUs, PUs, and TUs.

[0010] For each inter-predicted CU, motion parameters consisting of a motion vector, a reference picture index, and a reference picture list usage index, as well as additional information, are used for inter-predicted sample generation. The motion parameters can be signaled explicitly or implicitly. When a CU is encoded and decoded in skip mode, the CU is associated with one PU and has no significant residual coefficients, no coded motion vector delta, or reference picture index. A merge mode is specified, whereby the motion parameters of the current CU are obtained from neighboring CUs, including spatial candidates and temporal candidates, as well as additional scheduling introduced in VVC. The merge mode can be applied to any inter-predicted CU. An alternative to the merge mode is the explicit transmission of motion parameters, in which motion vectors, corresponding reference picture indices for each reference picture list, and reference picture list usage flags, as well as other required information, are explicitly signaled per CU. Summary of the invention

[0011] The following summary is illustrative only and is not intended to be limiting in any way. That is, the following summary is provided to introduce the concepts, highlights, benefits, and advantages of the new and non-obvious technologies described herein. Selected but not all embodiments are further described in the detailed description below. Therefore, the following summary is not intended to determine the essential characteristics of the claimed subject matter, nor is it intended to determine the scope of the claimed subject matter.

[0012] Some embodiments of the present disclosure provide a method for deriving and using a cross component model to improve intra-frame prediction or inter-frame prediction when encoding or decoding a block. A video codec receives data of a pixel block of a current block of a current picture to be encoded or decoded as a video. The video codec generates an initial predictor of the current block based on a reference block. The video codec receives pixels of a first group of component samples within or adjacent to the reference block. The reference block can be identified by a motion vector, a block vector, or an intra-frame prediction mode of the current block. The video codec receives pixels of a second group of component samples within or adjacent to the current block. The video codec derives a model based on the first group and the second group of component samples. The derived model can be linear or nonlinear. The derived model can be a convolution model including an MxN filter. The video codec applies the derived model to the initial predictor to generate an improved predictor. The video codec decodes or encodes the current block by using the improved predictor.

[0013] In some embodiments, the first set of component samples are neighboring samples of a reference block, and the second set of component samples are neighboring samples of a current block. In some embodiments, the first set of component samples are luma samples of a reference block, and the second set of component samples are luma samples of the current block. In some embodiments, the first set of component samples are chroma samples of a reference block, and the second set of component samples are chroma samples of the current block.

[0014] In some embodiments, the video codec derives a plurality of candidate models based on the first set of component samples and the second set of component samples, and selects a candidate model from the plurality of candidate models. In some embodiments, the video codec signals or receives an indication (e.g., a syntax element) in a bitstream for selecting the candidate model from the plurality of candidate models. In some embodiments, the candidate model is implicitly selected based on a setting (e.g., a template matching setting, a boundary matching setting, or a model accuracy setting) used to encode and decode the current block.

[0015] In some embodiments, the video codec signals or receives an indication in a bitstream for determining whether to use an initial predictor or an improved predictor to encode or decode a current block. In some embodiments, the video codec implicitly determines whether to use an initial predictor or an improved predictor to encode or decode the current block based on a setting used to encode the current block (e.g., a template matching setting, a boundary matching setting, or a model accuracy setting).

[0016] In some embodiments, a weighted sum of the improved predictor and the initial predictor is used as a final predictor for reconstructing the current block. In some embodiments, a first weight assigned to the initial predictor and a second weight assigned to the improved predictor are determined based on one or more prediction modes used to encode one or more neighboring blocks of the current block for generating a weighted sum. In some embodiments, the first weight and the second weight are determined based on a first number of neighboring blocks encoded in a first prediction mode and a second number of neighboring blocks encoded in a second prediction mode. (The first prediction mode and the second prediction mode may be any of intra, inter, AMVP, merge mode, etc.). BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1A to Figure 1B A current block and its reference blocks are conceptually illustrated.

[0018] FIG. 2A to FIG. 2E Deriving a convolution model based on a reference sample and a current sample is conceptually illustrated.

[0019] FIG. 3A to FIG. 3E The application of the resulting convolutional model to improve an initial prediction into an improved prediction is illustrated.

[0020] Figure 4 The use of data pairs of corresponding reference samples and current samples to derive a linear model for cross-component prediction is conceptually illustrated.

[0021] Figure 5 Corresponding reference samples and current samples used to derive the MxN filters used to define the cross-component linear model are illustrated.

[0022] Figure 6 An example video encoder is illustrated that can use a cross-component model to improve inter-prediction or intra-prediction.

[0023] Figure 7 A portion of a video encoder that implements improvements to inter-prediction or intra-prediction through a cross-component model is illustrated.

[0024] Figure 8 The process of using a predictor improved by a cross-component model when encoding a block is conceptually illustrated.

[0025] Fig. 9 An example video decoder is illustrated that can use a cross-component model to improve inter-prediction or intra-prediction.

[0026] Fig.10 A portion of a video decoder that implements improvements to inter-prediction or intra-prediction through a cross-component model is illustrated.

[0027] Fig.11 The process of using a predictor improved by a cross-component model when decoding a block is conceptually illustrated.

[0028] Fig.12 An electronic system for implementing some embodiments of the present disclosure is conceptually illustrated. DETAILED DESCRIPTION

[0029] In the following detailed description, a large number of specific details are explained by way of example in order to provide a thorough understanding of the relevant teachings. Any changes, derivations and / or extensions based on the teachings described herein are within the scope of protection of the present invention. In some examples, in order to avoid unnecessary confusion of aspects of the teachings of the present invention, known methods, procedures, components and / or circuits related to one or more exemplary embodiments disclosed herein are described at a relatively high level without details.

[0030] I. Improving Predictions with Convolution Models

[0031] To improve video compression efficiency, video codecs can use video coding tools or prediction modes using predictors. Inter codecs can use temporal information to predict the current block. Intra codecs can refer to spatial neighboring samples to predict the current block. Cross-component codecs can use cross-component information to generate cross-component predictions of the current block.

[0032] In some embodiments, for a current block that references a reference block as a predictor for inter prediction or intra prediction, two groups (a first group and a second group) of samples of color components (e.g., YCbCr or RGB) from the current block and the reference block are used to derive a convolutional model. The convolutional model is used to improve the inter prediction or intra prediction of the current block. The first group of samples is in or adjacent to the reference block (also referred to as reference samples). The second group of samples is in or adjacent to the current block (also referred to as current samples). The convolutional model can be derived by using the first group of samples and the second group of samples. The convolutional model can be used to generate cross-component predictions based on the first (group) of color components or the second (group) of color components.

[0033] Figure 1A to Figure 1B The current block 110 and its reference block are conceptually illustrated. The current block 110 is in the current picture 100. The reference block 111 is a predictor of the current block 110 according to inter prediction or intra prediction.

[0034] Figure 1A The reference block 111 is illustrated as a predictor of the current block 110 under inter prediction. As shown, the motion vector (MV) 120 of the current block 110 identifies the reference block 111 in the reference picture 101 as a predictor for inter prediction of the current block.

[0035] Figure 1B The reference block 111 is illustrated as a predictor of the current block 110 under intra prediction. As shown, the intra prediction direction or mode 121 or block vector (BV) identifies or derives the reference block 111 from samples of the current picture 100 as a predictor for intra prediction of the current block 110.

[0036] like Figure 1A to Figure 1BAs shown, the current template region 130 includes reconstructed samples adjacent to the current block 110 in the current picture 100. The current template region 130 has a corresponding or collocated reference template region 131 adjacent to the reference block 111 in the current picture 100 (for intra prediction) or the reference picture 101 (for inter prediction). The current sample can be obtained from within the current block 110 and the current template region 130 or around the current block 110 and the current template region 130. The reference sample can be obtained from within the reference block 111 and the reference template region 131 or around the current block 110 and the current template region 130. In some embodiments, the reference sample and the current sample are used to derive a convolution model.

[0037] The convolution model may be a cross-component model. For example, in some embodiments, the convolution model may be constructed using reference samples and current samples of the first color component in the juxtaposed region (e.g., between the Y sample in the reference template 131 and the Y sample in the current template 130), and used to predict samples of the second color component in the juxtaposed region (e.g., an improved prediction from Cr or Cb of the reference block 111 to Cr or Cb of the current block 110). For another example, in some embodiments, the convolution / cross-component model may be constructed using corresponding cross-component reference samples and / or current samples of the first color component and the second color component of the first region (e.g., Y versus Cr / Cb in the reference block 111 or the current template 130), and used for cross-component prediction in the second region (e.g., from Y to Cr / Cb in the current block 110).

[0038] The convolution model may also be used in a non-cross-component manner. For example, a convolution model derived based on reference samples and current samples of a first color component in and around a first block (e.g., between a Y sample in a reference template 131 and a Y sample in a reference block 111) may be used to predict samples of the same first color component in a second block (e.g., from a Y sample in a current template 130 to a Y sample in a current block 110). However, in the following sections, the term "cross-component model" is used interchangeably with the term "convolution model".

[0039] FIG. 2A to FIG. 2E Deriving a convolution model based on a reference sample and a current sample is conceptually illustrated. Figure 2A The derivation process 201 of deriving the convolution model 210 using the reference sample 211 and the current sample 212 is shown. The reference sample 211 is the Y sample in the reference block 111, and the current sample 212 is the Y sample in the current block 110. Figure 2BThe derivation process 202 of the convolution model 220 using the reference sample 221 and the current sample 222 is shown. The reference sample 221 is the Y sample in the reference block 111 (or the current template 130), and the current sample 222 is the Cr / Cb sample in the reference block 111 (or the current template 130). Figure 2C The derivation process 203 of deriving the convolution model 230 using the reference sample 231 and the current sample 232 is shown. The reference sample 231 is a Y sample in the reference template 131, and the current sample 232 is a Y sample in the reference block 111. Figure 2D The derivation process 204 of deriving the convolution model 240 using the reference sample 241 and the current sample 242 is shown. The reference sample 241 is the Y sample in the reference template 131, and the current sample 242 is the Y sample in the current template 130. Figure 2E A derivation process 205 is shown for deriving a convolution model 250 using a reference sample 251 and a current sample 252. The reference sample 251 is a Y sample in the reference block 111, and the current sample 252 is a Cr / Cb sample in the reference block 111. The derivation of the cross-component / convolution model will be further described in Section B below.

[0040] FIG. 3A to FIG. 3E The application of the resulting convolutional model to improve an initial prediction into an improved prediction is illustrated. Figure 3A It is shown that an initial prediction 311 is improved into an improved prediction 312 using the convolutional model 210. The initial prediction 311 is the Cr or Cb of the reference block 111, while the improved prediction 312 is used to reconstruct the Cr or Cb of the current block 110. Figure 3B It is shown that the initial prediction 321 is improved into the improved prediction 322 using the convolutional model 220. The initial prediction 321 is the reconstructed Y sample of the current block 110, while the improved prediction 322 is used to reconstruct the Cr / Cb in the current block 110. Figure 3C The improvement of the initial prediction 331 into the improved prediction 332 using the convolutional model 230 is shown. The initial prediction 331 is the reconstructed Y samples in the current block 130, while the improved prediction 332 is used to reconstruct the Y samples in the current block 110. Figure 3D It is shown that the initial prediction 341 is improved into the improved prediction 342 using the convolutional model 240. The initial prediction 341 is the Y samples in the reference block 111, while the improved prediction 342 is used to reconstruct the Y samples in the current block 110. Figure 3EThe use of the convolutional model 250 to improve the initial prediction 351 into an improved prediction 352 is shown. The initial prediction 351 is the Y sample in the current block 110, and the improved prediction 352 is used to reconstruct the Cr / Cb samples in the current block 110. Improving the prediction by using the cross-component / convolutional model will be further described below in Sections A to C.

[0041] A. Selecting the input of the convolutional model

[0042] In some embodiments, different color components of reference samples taken within or around the reference block 111 (including the reference template 131) and different color components of current samples taken within or around the current block 110 (including the current template 130) may be used as the first set of samples and the second set of samples, respectively, to derive a cross-component model.

[0043] For example, the first group is the Y (luminance) component of the reference block 111, and the second group is the Y component of the current block 110; or, the first group is the Cb of the reference block 111, and the second group is the Cb of the current block 110; or, the first group is the Cr of the reference block 111, and the second group is the Cr of the current block 110.

[0044] In some embodiments, the samples in the first group are neighboring samples of the reference block 111 (in the reference template area 121), and the samples in the second group are neighboring samples of the current block 110 (in the current template area 120). In some embodiments, the samples in the first group are samples in the reference block 111, and the samples in the second group are samples in the current block 110.

[0045] In some embodiments, the samples in the first group or the second group may be integer or fractional position samples. For example, the integer position samples in the first group or the second group may be 1-precision, 2-precision, 4-precision, or 8-precision position samples; or, for example, the fractional position samples in the first group or the second group may be 1 / 2-precision, 1 / 4-precision, 1 / 8-precision, or 1 / 16-precision position samples.

[0046] In some embodiments, the samples in the first group or the second group may be reconstructed samples or original samples. In some embodiments, the samples in the first group or the second group may be in a "reshape domain". In some embodiments, the samples in the first group or the second group are neighboring samples of a reference block or a current block after applying a gradient filter (e.g., [-1, 0, 1; -2, 0, 2; -1, 0, 1], [1, 2, 1; 0, 0, 0; -1, -2, -1], [0, 1, 2; -1, 0, 1; -2, -1, 0], [0, -1, -2; 1, 0, -1; 2, 1, 0] or [0, -1, 0; -1, 4, -1; 0, -1, 0]). In some embodiments, the gradient filter applied to the first group of samples and the second group of samples is the same.

[0047] In some embodiments, the above-mentioned features may be combined to form samples in the first group and the second group. For example, the first group of samples may be neighboring reconstructed integer Y samples of the reference block 111 (e.g., the reference template region 121), and the second group of samples may be neighboring reconstructed integer Y samples of the current block 110 (e.g., the current template region 120). For another example, the first group of samples are reconstructed integer Y samples in the reference block 111, and the second group of samples are original integer Y samples of the current block 110.

[0048] B. Deriving the cross-component model

[0049] As mentioned, the samples in the first group and the second group are used as reference samples and current samples, respectively, to derive a linear or nonlinear convolution model for cross-component prediction. In some embodiments, the convolution model is a linear model, and the model parameters to be derived include scale (denoted as alpha or) and offset (denoted as beta or) parameters. Figure 4 The use of data pairs of corresponding reference samples and current samples to derive a linear model for cross-component prediction is conceptually illustrated.

[0050] In some embodiments, MxN filter shapes are used to define model parameters of the convolutional / cross-component model. Figure 5 The corresponding reference samples and current samples used to derive the MxN filter for defining the cross-component linear model are illustrated. In the figure, the reference samples (first group) are denoted as Y (i,j) , and the current sample (the second group) is represented as Y'. (i,j) The position of can be in integer or fractional position. The neighboring samples of the reference block 111 (concatenated with the neighboring samples of the current block 110) correspond to Y (i,j), where i = 0, ±1, ±2, ..., and j = 0, ±1, ±2, ..., and the phase of Y' is mapped to Y (0,0) .

[0051] For a 1x2 filter (ie, M=1, N=2), the cross-component model is Y′=a (0,0) ×Y (0,0) +a (0,1) ×Y (0,1) +b, where a (0,0) and a (0,1) is the scale parameter, and b is the offset. For the case of a 2x2 filter (ie, M=2, N=2), the cross-component model is Among them, a (m,n) is the scale parameter, and b is the offset. For the case of a 3x2 filter (ie, M=3, N=2), the cross-component model is Among them, a (m,n) is the scale parameter, and b is the offset. Model parameters (i.e., a (m,n) and b) can have values ​​that are positive, negative or zero.

[0052] For some embodiments, the general model for a linear model using an MxN filter shape is formulated as Among them, Y (m,n) is the neighboring sample of the reference block, a (m,n) is the scale parameter, and b is the offset. Model parameters (i.e., a (m,n) and b) can be positive or negative values ​​or 0. If a neighboring sample of a reference block is unavailable, other samples or default sample values ​​may be used to replace the unavailable sample.

[0053] In some embodiments, nonlinear terms can be integrated into the convolutional model (thus, transforming the model into a nonlinear model). For example, clipping, bit-depth-based constants representing sample values, or Min / Max operations can be integrated into the convolutional model.

[0054] In some embodiments, a higher degree model is used instead of a linear model to predict chrominance samples. (m,n) x , where x>1. For another example, the exponential term Y of the mean of the neighboring samples (0,0) .

[0055] In some embodiments, the model parameters are derived at the encoder side and signaled to the decoder side to reduce decoder complexity or coding dependency. For example, a scale parameter or an offset parameter or a correction term thereof may be explicitly signaled in the bitstream.

[0056] In some embodiments, more than one model parameter may be signaled to the decoder side, but only a subset of the model parameters may be signaled, while the remaining model parameters may be derived implicitly (either through neighboring samples or signaled correction terms). For example, if two model parameters are used for the current block, only one model parameter may be signaled, while the other model parameter may be derived implicitly through neighboring samples. For another example, if MxN model parameters are used for the current block, k model parameters out of the MxN model parameters may be signaled (M×N>k≥1), while the other model parameters may be derived implicitly through neighboring samples. For another example, a video codec may signal correction terms to improve the implicitly derived parameters, while the other parameters may be further improved based on the corrected parameters.

[0057] C. Apply the resulting model to improve the prediction sample

[0058] The derived model parameters are used to improve the prediction samples of the current block. In the following, the predictor (e.g., inter prediction or intra prediction) of the current block is denoted as P. An improved predictor denoted as P' is obtained by applying the derived cross-component / convolutional model to the initial or unimproved predictor P.

[0059] In some embodiments, if the linear model uses a 1x2 filter shape, the improved prediction at position (i, j) within the current block is formulated as P′ (i,j) =a (0,0) ×P (i,j) +a (0,1) ×P (i,j+1) +b, where P (i,j) and P (i,j+1) is the unimproved prediction sample a (0,0) 、a (0,1) , and b are model parameters derived either explicitly signaled or implicitly through neighboring samples of the reference block and the current block.

[0060] In some embodiments, if the convolutional / cross-component model uses a 3x2 filter shape, the improved prediction at position (i, j) inside the current block is formulated as Among them, P (i+m,j+n) is the unimproved prediction sample a (m,n), and b are model parameters derived either explicitly signaled or implicitly through neighboring samples of the reference block and the current block.

[0061] In some embodiments, if the convolutional / cross-component model uses an MxN filter shape, the improved prediction at position (i, j) within the current block is formulated as Among them, P (i+m,j+n) is the unimproved prediction sample a (m,n) , and b is a model parameter explicitly signaled or implicitly derived from neighboring samples of the reference block and the current block. In some embodiments, if an unimproved prediction sample is not available, the video codec may replace the unavailable sample with other samples or a default sample value.

[0062] In some embodiments, the samples in P are integer position samples. In some embodiments, if the prediction samples of the current block are in fractional precision positions, the samples at the nearest integer position of the fractional precision positions are used for P.

[0063] In some embodiments, if a gradient filter (e.g., [-1, 0, 1; -2, 0, 2; -1, 0, 1], [1, 2, 1; 0, 0, 0; -1, -2, -1], [0, 1, 2; -1, 0, 1; -2, -1, 0], [0, -1, -2; 1, 0, -1; 2, 1, 0], or [0, -1, 0; -1, 4, -1; 0, -1, 0]) is applied to samples in the first group or the second group, the video codec can apply the same gradient filter to the prediction samples of the current block before applying the model parameters. Note that if a (unimproved) prediction sample is not available, the video codec can replace the unavailable sample with another sample or a default sample value.

[0064] In some embodiments, the predictor of P and the predictor of P' are mixed into the final predictor of the current block and formulated as:

[0065] P final (i,j)=(w1×P (i,j) +w2×P′ (i,j) +c)>>d

[0066] Wherein, w1 and w2 are weighting parameters. In some embodiments, w1 and w2 are set according to the position (i, j). Each position may have a different w1 and w2. In some embodiments, w1 and w2 for the current block are consistent. In some embodiments, w1 and w2 are determined based on neighboring blocks. In some embodiments, when the number of neighboring blocks encoded and decoded in the first prediction mode is greater than a threshold or greater than the number of neighboring blocks encoded and decoded in a second different prediction mode, w2 is greater than w1. Neighboring blocks may refer to top and left neighbors, or any 4x4 blocks around the left and top sides of the current block. The first prediction mode and the second prediction mode may be a mode with the proposed prediction improvement method, intra-frame, inter-frame, AMVP or merge mode, even if the first prediction mode and the second prediction mode should be different.

[0067] In some embodiments, more than one candidate model may be used to improve the current predictor based on different settings of the filter shape or different groupings of neighboring samples.

[0068] In some embodiments, an explicit rule is used to decide whether to enable or disable a cross-component coding tool, and / or an explicit rule is used to decide a candidate model when a cross-component coding tool is enabled. For example, a flag may be signaled / parsed at the block level to determine whether to apply a cross-component coding tool to the current block (e.g., if the flag is true, the tool for the current block is enabled; if the flag is false, the tool for the current block is disabled). If more than one candidate model is available for the current block, another syntax may be signaled to indicate which candidate model is selected for application.

[0069] In some embodiments, an implicit rule is used to determine whether to enable or disable a cross-component coding tool, and / or when a cross-component coding tool is enabled, an implicit rule is used to determine a candidate mode. The implicit rule may depend on a template-matching setting, a boundary-matching setting, or a model accuracy setting. The template matching setting is described below in Section D. The boundary matching setting is described below in Section E. The model accuracy setting is described below in Section F.

[0070] In some embodiments, the hinting rules for intra- and inter-coded blocks can be unified. For example, when template settings are used as hinting rules, the derivation process of template settings for inter-blocks is unified with the process for intra-blocks (e.g., TIMD blocks).

[0071] In some embodiments, the thresholds used in template matching and / or boundary matching and / or model accuracy may depend on block size, sequence resolution, neighboring blocks, and / or QP. In some embodiments, the MV of the current block signaled or implicitly derived in the proposed method has integer precision. In some embodiments, if the MV of the current block signaled or implicitly derived in the proposed method is rounded / clipped to integer precision, the predictor (P) of the current block comes from the integer precision MV.

[0072] D. Template-Matching

[0073] When the template matching setting is used, model parameters of each candidate mode are derived based on reference samples of the reference block and the template of the current block, and then the derived model parameters are executed on the template (neighboring area) of the current block.

[0074] Step 1: Take the reconstructed samples on the template of the current block as golden data.

[0075] Step 2: For each candidate mode, apply the obtained model parameters to the template of the corresponding luminance block to obtain the predicted samples within the template of the current chrominance block.

[0076] Step 3: For each candidate mode, calculate the distortion between the golden data and the predicted sample on the template.

[0077] Step 4: Determine the mode of the current block according to the calculated distortion amount. In some embodiments, the candidate mode with the smallest distortion amount is used for the current block.

[0078] In some embodiments, regarding the enabling condition of the codec tool, when the minimum distortion amount is less than a predefined threshold, the codec tool can be applied to the current block. For example, the predefined threshold can be T*template area, where T can be any floating value or 1 / N. (N can be any positive integer), and the template area can be set to template width*current block height+template height*current block width.

[0079] For another example, the predefined threshold is the amount of distortion between the reconstructed samples of the template of the current block and the predicted samples of the template generated according to the default mode (the original mode not improved by the cross-component model). When cross-component prediction is used to improve inter-frame prediction, the default mode is the original inter-frame mode, which can be any of the conventional merge candidate, AMVP candidate, affine candidate, GPM candidate or merge candidate. In some embodiments, the template size can be adjusted. The adjustment of the template size will be described together with the boundary matching setting.

[0080] E. Boundary matching settings

[0081] For some embodiments, when using the boundary matching setting, the boundary matching cost of the candidate mode refers to the discontinuity measurement (including top boundary matching and / or left boundary matching) between the current prediction (prediction sample within the current block) and the neighboring reconstruction (reconstructed samples within one or more neighboring blocks) generated from the candidate mode. Top boundary matching refers to the comparison between the current top prediction sample and the neighboring top reconstructed sample, while left boundary matching refers to the comparison between the current left prediction sample and the neighboring left reconstructed sample. In some embodiments, the candidate mode with the smallest boundary matching cost is applied to the current block.

[0082] In some embodiments, with respect to the enabling condition of the coding tool (predictor improvement by the cross-component model), the coding tool can be applied to the current block when the minimum boundary matching cost is less than a predefined threshold. For example, the predefined threshold is the boundary matching cost from the default mode (the original mode that is not improved by the proposed coding tool). When cross-component prediction is used to improve inter-frame prediction, the default mode is the original inter-frame mode, which can be any of a conventional merge candidate, an AMVP candidate, an affine candidate, a GPM candidate, or a merge candidate.

[0083] In some embodiments, a predefined subset of the current prediction is used to calculate the boundary matching cost, for example, by using n rows of the top boundary within the current block and / or m rows of the left boundary within the current block. (In addition, n2 rows of top neighboring reconstructed and / or m2 rows of left neighboring reconstructed are used).

[0084] The following examples of n and m can also be applied to n2 and m2. For example, n can be any positive integer, such as 1, 2, 3, 4, etc. For another example, m can be any positive integer, such as 1, 2, 3, 4, etc. One possible way is that n and / or m change with block width, height, or area. For example, for larger blocks (area>threshold2), m becomes larger. Threshold2=64, 128, or 256. When area>threshold2, m increases to 2. (Initially, m is 1). When area>threshold2, m increases to 4. (Initially, m is 1 or 2). Another possible way is that for taller blocks (height>thrershold2*width), m becomes larger and / or n becomes smaller. Threshold2=1, 2, or 4. When height>thrershold2*width, m increases to 2. (Initially, m is 1). When height>thrershold2*width, m increases to 4. (Initially, m is 1 or 2). Another possible way is that for larger blocks (area>threshold2), n becomes larger. Threshold2=64, 128 or 256. When area>threshold2, n increases to 2. (Initially, n is 1). When area>threshold2, n increases to 4. (Initially, n is 1 or 2). Another possible way is that for wider blocks (width>threshold2*height), n becomes larger and / or m becomes smaller. Threshold2=1, 2 or 4. When width>threshold2*height, n increases to 2. (Initially, n is 1). When width>threshold2*height, n increases to 4. (Initially, n is 1 or 2).

[0085] F. Model Accuracy Setting

[0086] When the template matching setting is used, model parameters of each candidate mode are derived based on reference samples of the reference block and the template of the current block, and then the derived model parameters are executed on the template (neighboring area) of the current block.

[0087] Step 1: Take the reconstructed samples on the template of the current block as golden data.

[0088] Step 2: For each candidate mode, apply the derived model parameters to the reconstructed / predicted samples in the template of the corresponding reference block to obtain the predicted samples in the template of the current block.

[0089] Step 3: For each candidate mode, calculate the distortion between the golden data and the predicted sample on the template. In some embodiments, the template used in the distortion calculation is a template for model parameter derivation. For example, the model can use both the left template and the top template to calculate the distortion. For another example, the model can use the left template to calculate the distortion. For another example, the model can use the above template to calculate the distortion.

[0090] Step 4: Determine the mode of the current block based on the calculated distortion amount. In some embodiments, the candidate mode with the minimum distortion amount is used for the current block. In some embodiments, regarding the enabling condition of the encoding tool, the encoding tool can be applied to the current block when the minimum distortion amount is less than a predefined threshold. For example, the predefined threshold is T*template area, T can be any floating value or 1 / N (N can be any positive integer), and the template area can be set to template width*current block height+template height*current block width.

[0091] For example, the predefined threshold is the amount of distortion between the reconstructed samples of the template of the current block and the predicted samples of the template generated according to the default mode. When cross-component prediction is used to improve inter-frame prediction, the default mode is the original inter-frame mode, which can be any of a normal, merge candidate, AMVP candidate, affine candidate, GPM candidate or merge candidate.

[0092] Prediction improvements for cross-component models (or convolutional models) may be enabled and / or disabled based on implicit rules (e.g., block width, height, or area) or based on explicit rules (e.g., syntax at the block, tile, slice, picture, SPS, or PPS level). For example, reordering may be applied when the block area is less than a threshold. The term "block" herein may refer to TU / TB, CU / CB, PU / P, predefined area, CTU / CTB, etc.

[0093] Any combination of the methods proposed in the present invention may be applied. Any of the methods proposed above may be implemented in an encoder and / or a decoder. For example, any of the proposed methods may be implemented in an inter / intra / prediction module of an encoder and / or an inter / intra / prediction module of a decoder. Alternatively, any of the proposed methods may be implemented as a circuit connected to an inter / intra / prediction module of an encoder and / or an inter / intra / prediction module of a decoder to provide the information required by the inter / intra / prediction module.

[0094] II. Example Video Encoder

[0095] Figure 6An example video encoder 600 is illustrated in which a cross-component model can be used to improve inter-frame prediction or intra-frame prediction. As shown, the video encoder 600 receives an input video signal from a video source 605 and encodes the signal into a bitstream 695. The video encoder 600 has several components or modules for encoding the signal from the video source 605, and the several components or modules include at least some components selected from the following items: a transform module 610, a quantization module 611, an inverse quantization module 614, an inverse transform module 615, an intra-picture estimation module 620, an intra-frame prediction module 625, a motion compensation module 630, a motion estimation module 635, an in-loop filter 645, a reconstructed picture buffer 650, an MV buffer 665, and an MV prediction module 675 and an entropy encoder 690. The motion compensation module 630 and the motion estimation module 635 are parts of the inter-frame prediction module 640.

[0096] In some embodiments, modules 610 to 690 are modules of software instructions being executed by one or more processing units (e.g., processors) of a computing device or electronic device. In some embodiments, modules 610 to 690 are modules of hardware circuits implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 610 to 690 are illustrated as separate modules, some of these modules may be combined into a single module.

[0097] The video source 605 provides a raw video signal representing pixel data of each video frame without compression. A subtractor 608 calculates the difference between the raw video pixel data of the video source 605 and the predicted pixel data 613 from the motion compensation module 630 or the intra-frame prediction module 625 as a prediction residual 609. The transform module 610 converts the difference (or residual pixel data or residual signal 608) into transform coefficients (e.g., by performing a discrete cosine transform (DCT)). The quantization module 611 quantizes the transform coefficients into quantized data (or quantized coefficients) 612, which are encoded into a bitstream 695 by the entropy encoder 690.

[0098] The inverse quantization module 614 de-quantizes the quantized data (or quantized coefficients) 612 to obtain transform coefficients, and the inverse transform module 615 performs inverse transform on the transform coefficients to generate a reconstructed residual 619. The reconstructed residual 619 is added to the predicted pixel data 613 to generate a reconstructed pixel data 617. In some embodiments, the reconstructed pixel data 617 is temporarily stored in a line buffer (not illustrated) for intra-picture prediction and spatial MV prediction. The reconstructed pixels are filtered by the in-loop filter 645 and stored in the reconstructed picture buffer 650. In some embodiments, the reconstructed picture buffer 650 is a storage device external to the video encoder 600. In some embodiments, the reconstructed picture buffer 650 is a storage device internal to the video encoder 600.

[0099] The intra-picture estimation module 620 performs intra-prediction based on the reconstructed pixel data 617 to generate intra-prediction data. The intra-prediction data is provided to the entropy encoder 690 to be encoded into a bitstream 695. The intra-prediction module 625 also generates predicted pixel data 613 using the intra-prediction data.

[0100] The motion estimation module 635 performs inter-frame prediction by generating MVs for reference pixel data of previously decoded frames stored in the decoded picture buffer 650. These MVs are provided to the motion compensation module 630 to generate predicted pixel data.

[0101] Instead of encoding the complete actual MV in the bitstream, the video encoder 600 generates a predicted MV using MV prediction, and encodes the difference between the MV used for motion compensation and the predicted MV as residual motion data and stores it in the bitstream 695 .

[0102] The MV prediction module 675 generates a predicted MV based on a reference MV generated for encoding a previous video frame (i.e., a motion compensated MV used to perform motion compensation). The MV prediction module 675 retrieves the reference MV from the previous video frame from the MV buffer 665. The video encoder 600 stores the MV generated for the current video frame in the MV buffer 665 as a reference MV for generating the predicted MV.

[0103] The MV prediction module 675 uses the reference MV to create a predicted MV. The predicted MV can be calculated by spatial MV prediction or temporal MV prediction. The entropy encoder 690 encodes the difference (residual motion data) between the predicted MV and the motion compensation MV (MCMV) of the current frame into the bitstream 695.

[0104] The entropy encoder 690 encodes various parameters and data into a bitstream 695 by using an entropy coding technique such as context-adaptive binary arithmetic coding (CABAC) or Huffman coding. The entropy encoder 690 encodes various header elements, flags, along with quantized transform coefficients 612 and residual motion data as syntax elements into the bitstream 695. The bitstream 695 is in turn stored in a storage device or sent to a decoder through a communication medium such as a network.

[0105] The in-loop filter 645 performs a filtering or smoothing operation on the reconstructed pixel data 617 to reduce coding artifacts, especially coding artifacts at the boundaries of blocks. In some embodiments, the filtering or smoothing operation performed by the in-loop filter 645 includes a deblocking filter (DBF), a sample adaptive offset (SAO), and / or an adaptive loop filter (ALF).

[0106] Figure 7 The portion of the video encoder 600 that implements improvements to inter prediction or intra prediction through a cross-component model is illustrated. When the current block is encoded by inter prediction, the motion estimation module 635 provides an MV used by the motion compensation module 630 to identify a reference block in a reference picture. When the current block is encoded by intra prediction, the intra prediction estimation module 620 provides an intra mode or BV used by the intra prediction module 625 to identify a reference block in the current picture. In some embodiments, the reference block is used as an initial predictor for the current block.

[0107] To derive the cross-component (or convolution) model 710, the model builder 705 retrieves component samples of pixels in and / or around the current block and component samples of pixels in and / or around the reference block from the reconstructed picture buffer 650 to use as reference samples (first set) and current samples (second set). The selection of reference samples and current samples is described in Section IA above. The derivation of the cross-component model is described in Section IB above.

[0108] The cross-component model 710 is applied to the initial predictor 715 (provided by the motion compensation module 630 or the intra-frame prediction module 625) to generate an improved predictor 725. The use of the cross-component model 710 to improve the predictor is described in the above section IC. Samples of the improved predictor 725 can be used as predicted pixel data 613. In some embodiments, the improvement of intra-frame prediction or inter-frame prediction by the cross-component model can be explicitly or implicitly enabled or disabled (so that the samples of the initial predictor are used as predicted pixel data 613), and the entropy encoder 690 can signal a syntax element to indicate this situation. In some embodiments, the prediction generator 720 combines the initial predictor 715 and the improved predictor 725 (e.g., as a weighted sum) into the predicted pixel data 613.

[0109] Figure 8 The process 800 of using a predictor improved by a cross-component model when encoding a block is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing the encoder 600 perform the process 800 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing the encoder 600 performs the process 800.

[0110] An encoder receives (at block 810) data to be encoded as a current block of pixels in a current picture of a video.

[0111] The encoder generates (at block 820) an initial predictor for the current block based on the reference block. The reference block may be identified by a motion vector, a block vector, or an intra prediction mode of the current block.

[0112] The encoder receives (at block 830) pixels of a first set of component samples within or adjacent to a reference block. The encoder receives (at block 840) pixels of a second set of component samples within or adjacent to a current block. In some embodiments, the first set of component samples are adjacent samples of the reference block, and the second set of component samples are adjacent samples of the current block. In some embodiments, the first set of component samples are luma samples of the reference block, and the second set of component samples are luma samples of the current block. In some embodiments, the first set of component samples are chroma samples of the reference block, and the second set of component samples are chroma samples of the current block.

[0113] The encoder derives (at block 850) a model based on the first set of component samples and the second set of component samples. The derived model may be linear or nonlinear (e.g., with nonlinear terms such as clipping). The derived model may be a convolutional model including an MxN filter.

[0114] The encoder applies (at block 860) the derived model to the initial predictor to generate an improved predictor. In some embodiments, the encoder derives a plurality of candidate models based on the first set of component samples and the second set of component samples, and selects a candidate model from the plurality of candidate models. In some embodiments, the encoder signals an indication (e.g., a syntax element) in the bitstream for selecting the candidate model from the plurality of candidate models. In some embodiments, the candidate model is implicitly selected based on a setting (e.g., a template matching setting, a boundary matching setting, or a model accuracy setting) used to encode the current block.

[0115] The encoder encodes (at block 870) the current block using the improved predictor to generate a prediction residual and reconstruct the current block. In some embodiments, the encoder signals an indication to the bitstream for determining whether to use the initial predictor or the improved predictor to encode the current block. In some embodiments, the encoder implicitly determines whether to use the initial predictor or the improved predictor to encode the current block based on a setting used to encode the current block (e.g., a template matching setting, a boundary matching setting, or a model accuracy setting).

[0116] In some embodiments, a weighted sum of the improved predictor and the initial predictor is used as a final predictor for reconstructing the current block. In some embodiments, a first weight assigned to the initial predictor and a second weight assigned to the improved predictor for generating a weighted sum are determined based on one or more prediction modes used to encode one or more neighboring blocks of the current block. In some embodiments, the first weight and the second weight are determined based on a first number of neighboring blocks encoded in a first prediction mode and a second number of neighboring blocks encoded in a second prediction mode. (The first prediction mode and the second prediction mode may be any of intra, inter, AMVP, merge mode, etc.).

[0117] III. Example Video Decoder

[0118] In some embodiments, the encoder may signal (or generate) one or more syntax elements in a bitstream so that the decoder may parse the one or more syntax elements from the bitstream.

[0119] Fig. 9An example video decoder 900 is illustrated in which a cross-component model can be used to improve inter-frame prediction or intra-frame prediction. As shown, the video decoder 900 is an image decoding or video decoding circuit that receives a bitstream 995 and decodes the content of the bitstream into pixel data of a video frame for display. The video decoder 900 has several components or modules for decoding the bitstream 995, including some components selected from the following items: an inverse quantization module 911, an inverse transform module 910, an intra-frame prediction module 925, a motion compensation module 930, an in-loop filter 945, a decoded picture buffer 950, an MV buffer 965, an MV prediction module 975, and a parser 990. The motion compensation module 930 is part of the inter-frame prediction module 940.

[0120] In some embodiments, modules 910 to 990 are modules of software instructions being executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 910 to 990 are modules of hardware circuits implemented by one or more ICs of an electronic device. Although modules 910 to 990 are illustrated as separate modules, some of these modules may be combined into a single module.

[0121] The parser 990 (or entropy decoder) receives the bitstream 995 and performs initial parsing according to the syntax defined by the video coding or image coding standard. The parsed syntax elements include various header elements, flags, and quantized data (or quantized coefficients) 912. The parser 990 parses out the various syntax elements by using entropy coding techniques such as context-adaptive binary arithmetic coding (CABAC) or Huffman coding.

[0122] The inverse quantization module 911 dequantizes the quantized data (or quantized coefficients) 912 to obtain transform coefficients, and the inverse transform module 910 performs an inverse transform on the transform coefficients 916 to generate a reconstructed residual signal 919. The reconstructed residual signal 919 is added to the predicted pixel data 913 from the intra-frame prediction module 925 or the motion compensation module 930 to generate decoded pixel data 917. The decoded pixel data is filtered by the in-loop filter 945 and stored in the decoded picture buffer 950. In some embodiments, the decoded picture buffer 950 is a storage device external to the video decoder 900. In some embodiments, the decoded picture buffer 950 is a storage device internal to the video decoder 900.

[0123] The intra prediction module 925 receives intra prediction information from the bitstream 995 and generates predicted pixel data 913 from the decoded pixel data 917 stored in the decoded picture buffer 950 based on the intra prediction information. In some embodiments, the decoded pixel data 917 is also stored in a line buffer (not illustrated) for intra picture prediction and spatial MV prediction.

[0124] In some embodiments, the contents of the decoded picture buffer 950 are used for display. The display device 955 retrieves the contents of the decoded picture buffer 950 for direct display, or retrieves the contents of the decoded picture buffer to a display buffer. In some embodiments, the display device receives pixel values ​​from the decoded picture buffer 950 via pixel transfer.

[0125] The motion compensation module 930 generates predicted pixel data 913 from the decoded pixel data 917 stored in the decoded picture buffer 950 according to the motion compensation MV (MC MV). These motion compensation MVs are decoded by adding the residual motion data received from the bitstream 995 to the predicted MV received from the MV prediction module 975.

[0126] The MV prediction module 975 generates a predicted MV based on a reference MV generated for decoding a previous video frame (e.g., a motion compensated MV used to perform motion compensation). The MV prediction module 975 retrieves the reference MV of the previous video frame from the MV buffer 965. The video decoder 900 stores the motion compensated MV generated for decoding the current video frame in the MV buffer 965 as a reference MV for generating the predicted MV.

[0127] The in-loop filter 945 performs filtering or smoothing operations on the decoded pixel data 917 to reduce coding artifacts, especially coding artifacts at the boundaries of blocks. In some embodiments, the filtering or smoothing operations performed by the in-loop filter 945 include a deblocking filter (DBF), a sample adaptive offset (SAO), and / or an adaptive loop filter (ALF).

[0128] Fig.10The portion of the video decoder 900 that implements improvements to inter-prediction or intra-prediction through a cross-component model is illustrated. When the current block is encoded and decoded through inter-prediction, the entropy decoder 990 provides an MV used by the motion compensation module 930 to identify a reference block in a reference picture. When the current block is encoded and decoded through intra-prediction, the entropy decoder 990 provides an intra-mode or BV used by the intra-prediction module 925 to identify a reference block in the current picture. In some embodiments, the reference block is used as an initial predictor 1015 of the current block.

[0129] To derive the cross-component (or convolution) model 1010, the model builder 1005 retrieves component samples of pixels in and / or around the current block and in and / or around the reference block from the decoded picture buffer 950 to use as reference samples (first set) and current samples (second set). The selection of reference samples and current samples is described in Section IA above. The derivation of the cross-component model is described in Section IB above.

[0130] The cross-component model 1010 is applied to the initial predictor 1015 (provided by the motion compensation module 930 or the intra-frame prediction module 925) to generate an improved predictor 1025. The use of the cross-component model 1010 to improve the predictor is described in the above section IC. Samples of the improved predictor 1025 can be used as the predicted pixel data 913. In some embodiments, the improvement of the cross-component model to intra-frame prediction or inter-frame prediction (so that the samples of the initial predictor 715 are used as the predicted pixel data 913) can be enabled or disabled explicitly or implicitly, and the entropy decoder 990 can receive a syntax element to indicate this situation. In some embodiments, the prediction generator 1020 can combine the initial predictor 1015 and the improved predictor 1025 (e.g., as a weighted sum) as the predicted pixel data 913.

[0131] Fig.11 The process 1100 of using a predictor improved by a cross-component model when decoding a block is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing the decoder 900 perform the process 1100 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing the decoder 900 performs the process 1100.

[0132] The decoder receives (at block 1110) pixel data to be decoded as a current block in a current picture of a video.

[0133] The decoder generates (at block 1120) an initial predictor for the current block based on the reference block. The reference block may be identified by a motion vector, a block vector, or an intra prediction mode of the current block.

[0134] The decoder receives (at block 1130) pixels of a first set of component samples within or adjacent to a reference block. The decoder receives (at block 1140) pixels of a second set of component samples within or adjacent to a current block. In some embodiments, the first set of component samples are adjacent samples of the reference block and the second set of component samples are adjacent samples of the current block. In some embodiments, the first set of component samples are luma samples of the reference block and the second set of component samples are luma samples of the current block. In some embodiments, the first set of component samples are chroma samples of the reference block and the second set of component samples are chroma samples of the current block.

[0135] The decoder derives (at block 1150) a model based on the first set of component samples and the second set of component samples. The derived model may be linear or nonlinear (e.g., with nonlinear terms such as clipping). The derived model may be a convolutional model including an MxN filter.

[0136] The decoder applies (at block 1160) the derived model to the initial predictor to generate an improved predictor. In some embodiments, the decoder derives a plurality of candidate models based on the first set of component samples and the second set of component samples, and selects a candidate model from the plurality of candidate models. In some embodiments, the decoder receives an indication (e.g., a syntax element) from the bitstream for selecting the candidate model from the plurality of candidate models. In some embodiments, the candidate model is implicitly selected based on a setting (e.g., a template matching setting, a boundary matching setting, or a model accuracy setting) for decoding the current block.

[0137] The decoder reconstructs (at block 1170) the current block using the improved predictor. The decoder may then provide the reconstructed current block for display as part of the reconstructed current picture. In some embodiments, the decoder receives an indication from the bitstream for determining whether to use the initial predictor or the improved predictor to decode the current block. In some embodiments, the decoder implicitly determines whether to use the initial predictor or the improved predictor to decode the current block based on a setting used to decode the current block (e.g., a template matching setting, a boundary matching setting, or a model accuracy setting).

[0138] In some embodiments, a weighted sum of the improved predictor and the initial predictor is used as a final predictor for reconstructing the current block. In some embodiments, a first weight assigned to the initial predictor and a second weight assigned to the improved predictor are determined based on one or more prediction modes used to encode one or more neighboring blocks of the current block for generating a weighted sum. In some embodiments, the first weight and the second weight are determined based on a first number of neighboring blocks encoded in a first prediction mode and a second number of neighboring blocks encoded in a second prediction mode. (The first prediction mode and the second prediction mode may be any of intra, inter, AMVP, merge mode, etc.).

[0139] IV. Example Electronic Systems

[0140] Many of the above features and applications can be implemented as software processes, which are specified as sets of instructions recorded on a computer readable storage medium (also referred to as a computer readable medium). When these instructions are executed by one or more computing units or processing units (e.g., one or more processors, processor cores, or other processing units), the instructions cause the processing units to perform the actions represented by the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, random access memory (RAM) chips, hard disks, erasable programmable read only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. Computer readable media do not include carriers and telecommunications signals connected wirelessly or wired.

[0141] In this specification, the term "software" is meant to include firmware in a read-only memory or an application stored in a magnetic storage device, which can be read into the memory for processing by the processor. At the same time, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program, while retaining different software inventions. In some embodiments, multiple software inventions can be implemented as independent programs. Finally, any combination of independent programs that implement the software inventions described herein together is within the scope of the present invention. In some embodiments, when installed to operate on one or more electronic systems, the software program defines one or more specific machine implementations that execute and implement the operations of the software program.

[0142] Fig.12 An electronic system 1200 for implementing some embodiments of the present disclosure is conceptually illustrated. The electronic system 1200 may be a computer (e.g., a desktop computer, a personal computer, a tablet computer, etc.), a phone, a PDA, or any other type of electronic device. Such an electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 1200 includes: a bus 1205, a processing unit 1210, a graphics processing unit (GPU) 1215, a system memory 1220, a network 1225, a read-only memory 1230, a permanent storage device 1235, an input device 1240, and an output device 1245.

[0143] The bus 1205 collectively represents all system, peripheral, and chipset buses that communicatively connect the various internal devices of the electronic system 1200. For example, the bus 1205 communicatively connects the processing unit 1210 with the GPU 1215, the read-only memory 1230, the system memory 1220, and the permanent storage device 1235.

[0144] From these various memory units, the processing unit 1210 retrieves instructions to be executed and data to be processed in order to perform the processes of the present disclosure. In different embodiments, the processing unit can be a single processor or a multi-core processor. Some instructions are passed to and executed by the GPU 1215. The GPU 1215 can offload various calculations or supplement the image processing provided by the processing unit 1210.

[0145] Read-only-memory (ROM) 1230 stores static data and instructions used by processing unit 1210 and other modules of the electronic system. On the other hand, permanent storage device 1235 is a read-write memory device. This device is a non-volatile memory unit that stores instructions and data even when electronic system 1200 is turned off. Some embodiments of the present disclosure use a mass storage device (such as a magnetic disk or optical disk and its corresponding disk drive) as permanent storage device 1235.

[0146] Other embodiments use removable storage devices (such as floppy disks, flash memory devices, etc., and their corresponding disk drives) as permanent storage devices. Similar to permanent storage device 1235, system memory 1220 is a read-write storage device. However, unlike storage device 1235, system memory 1220 is a volatile read-write memory, such as random access memory. System memory 1220 stores some of the instructions and data used by the processor at run time. In some embodiments, processes according to the present disclosure are stored in system memory 1220, permanent storage device 1235, and / or read-only memory 1230. For example, according to some embodiments, various memory units include instructions for processing multimedia clips. Processing unit 1210 retrieves instructions to be executed and data to be processed from these various memory units in order to perform the processes of some embodiments.

[0147] The bus 1205 is also connected to an input device 1240 and an output device 1245. The input device 1240 enables a user to transmit information and select commands to the electronic system. The input device 1240 includes an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"), a camera (e.g., a webcam), a microphone, or a similar device for receiving voice commands, etc. The output device 1245 displays an image generated by the electronic system or outputs data in other ways. The output device 1245 includes a printer and a display device, such as a cathode ray tube (CRT) or a liquid crystal display (LCD), and a speaker or similar audio output device. Some embodiments include a device such as a touch screen, which serves as both an input and output device.

[0148] Finally, if Fig.12 As shown, bus 1205 also couples electronic system 1200 to network 1225 via a network adapter (not shown). In this way, the computer can be a network of computers (such as a local area network (LAN), a wide area network (WAN)), or an intranet, or a network such as the Internet. Any or all components of electronic system 1200 can be used in conjunction with the present disclosure.

[0149] Some embodiments include electronic components such as a microprocessor, a storage device and a memory that stores computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as a computer-readable storage medium, a machine-readable medium, or a machine-readable storage medium). Some examples of such computer-readable media include: RAM, ROM, read-only compact disc (CD-ROM), recordable compact disc (CD-R), rewritable compact disc (CD-RW), read-only digital versatile disc (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable A computer readable medium may store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer codes include machine code such as generated by a compiler, and files including high-level code that is executed by a computer, electronic component, or microprocessor using an interpreter.

[0150] Although the above discussion primarily involves microprocessors or multi-core processors that execute software, many of the features and applications described above are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuits themselves. In addition, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM devices.

[0151] As used in this specification and any claims of this application, the terms "computer", "server", "processor", and "memory" all refer to electronic or other technical devices. These terms exclude people or groups of people. For the purposes of this specification, the terms "display" or "displaying" refer to displaying on an electronic device. As used in this specification and any claims of this application, the terms "computer-readable medium", "computer-readable media (media)" and "machine-readable medium" are entirely limited to tangible physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, and any other temporary signals.

[0152] Although the present disclosure has been described with reference to many specific details, those skilled in the art will recognize that the present disclosure may be implemented in other specific forms without departing from the spirit of the present disclosure. Figure 8 and Fig.11 ) conceptually illustrate the processes. The specific operations of these processes may not be performed in the exact order shown and described. Specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different implementations. Moreover, the process may be implemented using several sub-processes or as part of a larger macro process. Therefore, it will be understood by those skilled in the art that the present disclosure is not limited by the foregoing illustrative details, but is defined by the appended claims.

[0153] Additional Notes

[0154] The subject matter described herein sometimes represents different components, which are contained in or connected to other different components. It is understood that the described structure is only an example and can actually be implemented by many other structures to implement the same function. Conceptually, any arrangement of components that implement the same function is actually "associated" in order to implement the desired function. Therefore, regardless of the structure or intermediate components, any two components combined to implement a specific function are considered to be "interrelated" to implement the desired function. Similarly, any two associated components are considered to be "operably connected" or "operably coupled" to each other to implement a specific function. Any two components that can be associated with each other are also considered to be "operably coupled" to each other to implement a specific function. Specific examples of operable connections include, but are not limited to, physically pairable and / or physically interacting components, and / or wirelessly interactive and / or wirelessly interactive components, and / or logically interactive and / or logically interactive components.

[0155] Furthermore, with respect to the use of substantially any plural and / or singular terms, those of ordinary skill in the art can translate from the plural to the singular and / or from the singular to the plural as appropriate to the context and / or application. For clarity, the various singular / plural permutations are expressly provided herein.

[0156] In addition, it will be understood by those skilled in the art that, in general, the terms used in the present invention, especially in the claims, and the subject matter of the claims, are generally used as "open" terms, for example, "including" should be interpreted as "including but not limited to", "having" should be interpreted as "at least having", "including" should be interpreted as "including but not limited to", etc. It will be further understood by those skilled in the art that if a specific number of claim contents are intended to be introduced, it will be explicitly indicated in the claim, and will not be displayed in the absence of such contents. For example, to aid understanding, the claims may include the phrases "at least one" and "one or more" to introduce the contents of the claims. However, the use of these phrases should not be interpreted as implying the use of the indefinite article "a" or "an" to introduce the contents of the claims, thereby limiting any particular patent scope. Even when the same claim includes the introductory phrases "one or more" or "at least one", the indefinite article, such as "a" or "an", should be interpreted to mean at least one or more, and the same is true for the use of the explicit description used to introduce the claims. Furthermore, even when an introductory content explicitly refers to a specific number, one of ordinary skill in the art will recognize that such content should be interpreted as indicating the number of references, for example, "two references" without other modifications means at least two references, or two or more references. Furthermore, where expressions similar to "at least one of A, B, and C" are used, they are generally expressed so that one of ordinary skill in the art can understand the expression, for example, "a system includes at least one of A, B, and C" will include but is not limited to a system with A alone, a system with B alone, a system with C alone, a system with A and B, a system with A and C, a system with B and C, and / or a system with A, B, and C, etc. One of ordinary skill in the art will further understand that any separated words and / or phrases represented by two or more alternative terms, whether in the specification, claims, or drawings, should be understood to include the possibility of one of these terms, one of them, or both of these terms. For example, "A or B" should be understood as the possibility of "A", or "B", or "A and B".

[0157] As can be seen from the foregoing, various embodiments have been described herein for illustrative purposes and that various modifications may be made without departing from the scope and spirit of the invention. Therefore, the various embodiments disclosed herein are not intended to be limiting, with the claims indicating the true scope and spirit.

Claims

1. A video encoding and decoding method, comprising: Receiving data of a pixel block to be encoded or decoded as a current block of a current picture of a video; generating an initial predictor of the current block based on a reference block; Receiving pixels of a first set of component samples within or adjacent to the reference block; Receiving pixels of a second set of component samples within or adjacent to the current block; deriving a model based on the first and second groups of component samples; applying the derived model to the initial predictor to generate an improved predictor; as well as The current block is decoded or encoded by using the improved predictor.

2. The video encoding and decoding method as claimed in claim 1, characterized in that: in, The first group of component samples are neighboring samples of the reference block, and the second group of component samples are neighboring samples of the current block.

3. The video encoding and decoding method as claimed in claim 1, characterized in that: in, The first set of component samples are luma samples of the reference block, and the second set of component samples are luma samples of the current block.

4. The video encoding and decoding method as claimed in claim 1, characterized in that: in, The first set of component samples are neighboring chroma samples of the reference block, and the second set of component samples are neighboring chroma samples of the current block.

5. The video encoding and decoding method as claimed in claim 1, characterized in that: in, The resulting model includes nonlinear terms.

6. The video encoding and decoding method as claimed in claim 1, characterized in that: in, The resulting model consists of MxN filters.

7. The video encoding and decoding method as claimed in claim 1, characterized in that: in, A weighted sum of the improved predictor and the initial predictor is used as a final predictor for reconstructing the current block.

8. The video encoding and decoding method as claimed in claim 7, characterized in that: in, A first weight assigned to the initial predictor and a second weight assigned to the improved predictor are determined based on one or more prediction modes used to encode one or more neighboring blocks of the current block for generating the weighted sum.

9. The video encoding and decoding method as claimed in claim 8, characterized in that: in, The first weight and the second weight are determined based on a first number of neighboring blocks encoded in a first prediction mode and a second number of neighboring blocks encoded in a second prediction mode.

10. The video encoding and decoding method as claimed in claim 1, characterized in that: in, Deriving the model includes: deriving a plurality of candidate models based on the first and second groups of component samples, and selecting a candidate model from the plurality of candidate models.

11. The video encoding and decoding method as claimed in claim 10, characterized in that: Also includes: An indication is received from a bitstream or signaled to the bitstream for selecting the candidate model from the plurality of candidate models.

12. The video encoding and decoding method as claimed in claim 10, characterized in that: in, The candidate model is implicitly selected based on settings used to encode or decode the current block.

13. The video encoding and decoding method as claimed in claim 1, further comprising: An indication is received from a bitstream or signaled to the bitstream for deciding whether to use the initial predictor or the improved predictor to encode or decode the current block.

14. The video encoding and decoding method as claimed in claim 1, wherein: Also includes: Whether to use the initial predictor or the improved predictor to encode or decode the current block is implicitly determined based on settings for encoding or decoding the current block.

15. An electronic device, comprising: The video codec circuit is configured to perform operations including: Receiving data of a pixel block to be encoded or decoded as a current block of a current picture of a video; generating an initial predictor of the current block based on a reference block; Receiving pixels of a first set of component samples within or adjacent to the reference block; Receiving pixels of a second set of component samples within or adjacent to the current block; deriving a model based on the first and second groups of component samples; applying the derived model to the initial predictor to generate an improved predictor; and decoding or encoding the current block by using the improved predictor.

16. A video decoding method, the video decoding method comprising: Receiving data of a block of pixels to be decoded as a current block of a current picture of a video; generating an initial predictor of the current block based on a reference block; Receiving pixels of a first set of component samples within or adjacent to the reference block; Receiving pixels of a second set of component samples within or adjacent to the current block; deriving a model based on the first set of component samples and the second set of component samples; applying the derived model to the initial predictor to generate an improved predictor; as well as The current block is reconstructed by using the improved predictor.

17. A video encoding method, the video encoding method comprising: Receiving data of a block of pixels to be encoded as a current block of a current picture of a video; generating an initial predictor of the current block based on a reference block; Receiving pixels of a first set of component samples within or adjacent to the reference block; Receiving pixels of a second set of component samples within or adjacent to the current block; deriving a model based on the first and second groups of component samples; applying the derived model to the initial predictor to generate an improved predictor; as well as The current block is encoded by using the improved predictor.

18. A video encoding and decoding method, the video encoding and decoding method comprising: Receiving data of a pixel block to be encoded or decoded as a current block of a current picture of a video; generating an initial predictor of the current block based on a reference block; Receiving pixels of a first group and a second group of component samples in or adjacent to the current block or the reference block, wherein at least one of the pixels of the first group and the second group of component samples is from the reference block; deriving a model based on the first set of component samples and the second set of component samples; applying the derived model based on the initial predictor to generate an improved predictor; and The current block is decoded or encoded by using the improved predictor.

19. The video encoding and decoding method as claimed in claim 18, characterized in that: in, The first set of component samples are luma samples of the reference block, and the second set of component samples are chroma samples of the reference block.

Images