Unified cross-component model derivation

Abstract

A video codec receives a selection of one of a plurality of codec tools. The video codec specifies a regression data set according to the selection of a codec tool. The specified regression dataset is selected from component samples within or adjacent to the reference block or the current block. The video codec specifies a configuration of a convolution model according to the selection of a codec tool. The convolution model is configurable to support each of the coding and decoding tools. And the video codec obtains parameters of the convolution model by applying a regression data set according to specified configuration. The video codec applies the convolution model with the generated parameters to a reference component sample set to obtain a prediction component sample set. The video codec encodes or decodes the current block by using the prediction component samples.

Description

Technical Field

[0001] The present invention relates to video coding and decoding, and more particularly to a method for coding and decoding pixels by a cross-component model. Background Art

[0002] Unless otherwise indicated herein, the methods described in this section are not prior art to the claims listed below and are not admitted to be prior art by inclusion in this section.

[0003] High Efficiency Video Coding (HEVC) is an international video coding standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). HEVC is based on a hybrid block-based motion compensated DCT-like transform coding architecture. The basic unit of compression, called a coding unit (CU), is a 2Nx2N square block, and each CU can be recursively split into four smaller CUs until a predetermined minimum size is reached. Each CU contains one or more prediction units (PUs).

[0004] Versatile Video Coding (VVC) is the latest international video coding standard developed by the Joint Video Experts Group (JVET) of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11. The input video signal is predicted from the reconstruction signal, which is derived from the coded picture area. The predicted residual signal is processed by block transform. The transform coefficients are quantized and entropy coded together with other side information in the bitstream. The reconstructed signal is generated by the prediction signal and the reconstructed residual signal after inverse transforming the dequantized transform coefficients. The reconstructed signal is further processed by in-loop filtering to remove coding artifacts. The decoded pictures are stored in the frame buffer and used to predict future pictures in the input video signal.

[0005] In VVC, a coded picture is divided into non-overlapping square block areas, represented by related coding tree units (CTUs). The leaf nodes of the coding tree correspond to coding units (CUs). A coded picture can be represented by a set of slices, each slice consisting of an integer number of CTUs. The CTUs in a slice are processed continuously by raster scanning. Bi-predictive (B) slices can be decoded using intra prediction or inter prediction, using up to two motion vectors and reference indices to predict the sample values ​​of each block. Predictive (P) slices are decoded using intra prediction or inter prediction, using up to one motion vector and reference index to predict the sample values ​​of each block. Intra (I) slices are decoded using only intra prediction.

[0006] A CTU can be partitioned into one or more non-overlapping coding units (CUs) using a quadtree (QT) with a nested multi-type tree (MTT) structure to accommodate various local motion and texture characteristics. A CU can be further partitioned into smaller CUs using one of five partition types: quadtree partitioning, vertical binary tree partitioning, horizontal binary tree partitioning, vertical center-side ternary tree partitioning, and horizontal center-side ternary tree partitioning.

[0007] Each CU contains one or more prediction units (PUs). The prediction unit, together with the associated CU syntax, works as a basic unit of signal predictor information. A specified prediction process is used to predict the values ​​of the associated pixel samples within the PU. Each CU may contain one or more transform units (TUs) for representing prediction residual blocks. A transform unit (TU) consists of a transform block (TB) of luma samples and two corresponding transform blocks of chroma samples, each TB corresponding to a residual block of samples of a color component. Integer transforms are applied to the transform blocks. The level values ​​of the quantized coefficients are entropy coded in the bitstream along with other side information. The terms coding tree block (CTB), coding block (CB), prediction block (PB), and transform block (TB) are defined to specify a two-dimensional array of samples of a monochrome component associated with a CTU, CU, PU, ​​and TU, respectively. Therefore, a CTU consists of one luma CTB, two chroma CTBs, and associated syntax elements. Similar relationships apply to CUs, PUs, and TUs.

[0008] For each inter-predicted CU, motion parameters consisting of motion vector, reference picture index, reference picture list usage index, and additional information are used for inter-predicted sample generation. The motion parameters can be signaled explicitly or implicitly. When a CU is coded in skip mode, the CU is associated with one PU and has no significant residual coefficients, no coded motion vector increments or reference picture indexes. A merge mode is specified, in which the motion parameters of the current CU are obtained from neighboring CUs, including spatial and temporal candidates and the additional timetable introduced in VVC. This merge mode can be applied to any inter-predicted CU. An alternative to the merge mode is the explicit transmission of motion parameters, in which the motion vector, the corresponding reference picture index for each reference picture list, the reference picture list usage flag, and other required information are explicitly signaled in each CU. Summary of the invention

[0009] The following summary is illustrative only and is not intended to be limiting in any way. That is, the following summary is provided to introduce the concepts, highlights, benefits, and advantages of the novel and non-obvious technologies described herein. Selected and not all embodiments are further described in the detailed description below. Therefore, the following summary is not intended to determine the essential characteristics of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.

[0010] Some embodiments of the present application provide a method for deriving and using a unified convolution model, which can be configured for different codec tools. A video codec receives a selection result of selecting a codec tool from a plurality of codec tools. The video codec specifies a regression data set based on the selection of the codec tool. The specified regression data set is selected from component samples within or adjacent to a reference block or a current block. The video codec specifies the configuration of the convolution model based on the selection of the coding tool. The convolution model is configurable to support each of the plurality of codec tools. The video codec derives the parameters of the convolution model by applying the regression data set according to the specified configuration. The video codec applies the convolution model with the generated parameters to a reference component sample set to obtain a predicted component sample set. The video codec encodes or decodes the current block by using the predicted component samples.

[0011] In some embodiments, the convolution model is a high-order model including a filter having more than two filter taps. The parameters of the convolution model are obtained by using a unified solver for solving the matrix equations of the plurality of codec tools; wherein the matrix equation is represented by Ax=b, A is an autocorrelation matrix, b is a cross-correlation vector, and x is the parameter to be solved. The unified solver can use an elimination-based method, a decomposition-based method, or an iteration-based method to determine the optimal vector of the convolution model as a solution to the parameters of the convolution model.

[0012] In some embodiments, the regression data set is selected from a unified range of a plurality of reference lines adjacent to the current block, the unified range being common to a plurality of codecs. In some embodiments, regression data of different codecs are selected from the same number of reference lines within the unified range. In some embodiments, the unified range of the plurality of reference lines is stored in a line buffer used by the plurality of codecs. The regression data set is selected from reference samples divided into a plurality of reference sample groups, the regression data of different codecs being selected from different reference sample groups.

[0013] In some embodiments, when the regression data set includes invalid samples, the video codec performs a unified action common to the plurality of codec tools. Invalid samples are samples that exceed a constraint boundary or are prohibited by at least one of the plurality of codec tools. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Conceptual illustration of the chrominance and luminance samples used to derive linear model parameters.

[0015] Figure 2 An example of classification of adjacent samples into two groups is marked.

[0016] Figure 3 Conceptual illustration of the spatial components of a convolution filter.

[0017] Figure 4 A reference region for deriving filter coefficients of a convolutional model for the current block is shown.

[0018] Figure 5 The reconstructed brightness sample and its neighboring samples are marked.

[0019] Figure 6 A convolutional model generation system 600 is conceptually illustrated that is configurable for different codecs.

[0020] Fig. 7A -B conceptually illustrates example sources of component samples that can be used to build high-order convolutional models.

[0021] Figure 8 An exemplary video encoder that may implement cross-component prediction is shown.

[0022] Fig. 9 Portions of a video encoder that implements a unified cross-component convolutional model are shown.

[0023] Fig.10 Conceptually illustrates the process for deriving and using a unified convolutional model that is configurable for different codecs.

[0024] Fig.11 An example of a video decoder that can implement cross-component prediction is shown.

[0025] Fig.12 Part of a video decoder that implements a unified cross-component convolutional model is shown.

[0026] Fig.13 Conceptually illustrates the process for deriving and using a unified convolutional model that is configurable for different codecs.

[0027] Fig.14 An electronic system for implementing some embodiments of the present application is conceptually described. DETAILED DESCRIPTION

[0028] It should be readily understood that the components of the present invention as generally described herein and illustrated in the accompanying drawings may be arranged and designed in a variety of different configurations. Therefore, as represented in the accompanying drawings, the following more detailed description of the embodiments of the systems and methods of the present invention is not intended to limit the scope of the claimed invention, but is merely representative of selected embodiments of the present invention. References throughout this specification to "one embodiment," "an embodiment," or similar language are intended to mean that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing throughout this specification do not necessarily all refer to the same embodiment.

[0029] Moreover, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. However, those skilled in the relevant art will recognize that the present invention may be practiced without one or more of these specific details or using other methods, components, etc. In other cases, well-known structures or operations are not shown or described in detail to avoid blurring the various aspects of the present invention. The illustrated embodiments of the present invention will be best understood by reference to the accompanying drawings, in which the same parts are designated by the same numbers throughout. The following description is by way of example only, and simply illustrates certain selected embodiments of the apparatus and method consistent with the invention claimed herein.

[0030] I. Cross-Component Linear Model (CCLM)

[0031] The cross-component linear model (CCLM) or linear model (LM) mode is a cross-component prediction mode in which the chrominance components of a block are predicted from the collocated reconstructed luminance (Luma) samples by a linear model. The parameters of the linear model (e.g., scale and offset) are derived from the already reconstructed luminance and chrominance (Chroma) samples adjacent to the block. For example, in VVC, the CCLM mode exploits the inter-channel dependencies to predict chrominance samples from reconstructed luminance samples. This prediction is performed using a linear model of the following form:

[0032] P(i,j)=α·rec′ L (i,j)+β (1)

[0033] P(i,j) in formula (1) represents the predicted chroma sample in the CU (or the predicted chroma sample of the current CU), rec′ L(i, j) represents the downsampled reconstructed luma samples of the same CU (or the corresponding reconstructed luma samples of the current CU).

[0034] The CCLM model parameters α (scaling parameter) and β (offset parameter) are derived from up to four adjacent chroma samples and their corresponding downsampled luma samples. In LM_A mode (also denoted as LM-T mode), only the upper or top adjacent templates are used to calculate the linear model coefficients. In LM_L mode (also denoted as LM-L mode), only the left template is used to calculate the linear model coefficients. In LM-LA mode (also denoted as LM-LT mode), both the left and upper templates are used to calculate the linear model coefficients.

[0035] Figure 1 The chrominance and luma samples used to derive linear model parameters are conceptually illustrated. The diagram shows a current block 100 having luma component samples and chroma component samples in a 4:2:0 format. The luma and chroma samples adjacent to the current block are reconstructed samples. These reconstructed samples are used to derive the linear model (parameters α and β) of the cross components. Since the current block is in a 4:2:0 format, the luma samples are downsampled before being used for linear model derivation. In this example, there are 16 pairs of reconstructed luma (downsampled) and chroma samples adjacent to the current block. These 16 pairs of luma and chroma values ​​are used to derive linear model parameters.

[0036] Assuming the current chroma block size is W×H, then W' and H' are set to:

[0037] - When LM-LT mode is applied, W'=W, H'=H;

[0038] - When LM-T mode is applied, W'=W+H;

[0039] - When LM-L mode is applied, H'=H+W.

[0040] The upper adjacent position is represented by S[0,-1]...S[W'-1,-1], and the left adjacent position is represented by S[-1,0]...S[-1,H'-1]. Then four samples are selected as:

[0041] - When LM mode is applied (both upper and left adjacent samples are available), S[W' / 4,-1], S[3*W' / 4,-1], S[-1,H' / 4], S[-1,3*H' / 4];

[0042] - When the LM-T mode is applied (only the upper adjacent samples are available), S[W' / 8,-1], S[3*W' / 8,-1], S[5*W' / 8,-1], S[7*W' / 8,-1];

[0043] - When LM-L mode is applied (only left neighboring samples are available), S[-1,H' / 8], S[-1,3*H' / 8], S[-1,5*H' / 8], S[-1,7*H' / 8];

[0044] The four adjacent luma samples at the selected position are downsampled and compared four times to find the two larger values: x0A and x1A, and the two smaller values: x0B and x1B. Their corresponding chroma sample values ​​are denoted as y0A, y1A, y0B, and y1B. Then, XA, XB, YA, and YB are derived as:

[0045] Xa = (x0A+ x1A +1)>>1; Xb = (x0B + x1B +1)>>1 (2)

[0046] Ya = (y0A + y1A +1)>>1; Yb = (y0B + y1B +1)>>1 (3)

[0047] The linear model parameters α and β are obtained according to the following formula:

[0048]

[0049] β=Y b -α·X b (5)

[0050] The operation of calculating the α and β parameters according to formulas (4) and (5) can be implemented by a lookup table. In some embodiments, in order to reduce the memory required to store the lookup table, the diff value (the difference between the maximum and minimum values) and the parameter α are represented by exponential notation. For example, diff is approximated by a 4-bit effective part and an exponent. Therefore, the table of 1 / diff is simplified to 16 elements for 16 effective values, as shown below:

[0051] DivTable [ ] = {0, 7, 6, 5, 5, 4, 4, 3, 3, 2, 2, 1, 1, 1, 1, 0} (6)

[0052] This reduces the complexity of the calculations as well as the memory size required to store the required tables.

[0053] In some embodiments, in order to obtain more samples for calculating CCLM model parameters α and β, for LM-T mode, the upper template is expanded to include (W+H) samples, and for LM-L mode, the left template is expanded to include (H+W) samples. For LM-LT mode, both the expanded left template and the expanded upper template are used to calculate the linear model coefficients.

[0054] To match the chroma sample positions of a 4:2:0 video sequence, two types of downsampling filters are applied to luma samples to achieve a 2 to 1 downsampling ratio in both the horizontal and vertical directions. The choice of downsampling filter is specified by a flag at the sequence parameter set (SPS) level. The two downsampling filters are as follows, corresponding to the contents of "type 0" and "type 2" ("type-0" and "type-2"), respectively.

[0055] recL'(i,j)=[recL(2i-1,2j-1)+2*recL(2i-1,2j-1)+recL(2i+1,2j-1)+recL(2i-1,2j)+2*recL(2i,2j) +recL(2i+1,2j)+4] >> 3 (7) recL'(i,j)=[recL(2i,2j-1)+recL(2i-1,2j)+4*recL(2i,2j)+recL(2i+1,2j)+recL(2i,2j+1)+4] >>3 (8)

[0058] In some embodiments, when the upper layer reference line is at a CTU boundary, only one luma line (normal line buffer in intra prediction) is used for down-sampled luma samples.

[0059] In some embodiments, the α and β parameter calculations are performed as part of the decoding process, rather than just as an encoder search operation. Therefore, no syntax is used to communicate the values ​​of α and β to the decoder.

[0060] For chroma intra mode codec, a total of 8 intra modes are allowed. These modes include five traditional intra modes and three cross-component linear modes (LM_LA, LM_A, and LM_L). Chroma intra mode codec can directly depend on the intra prediction mode of the corresponding luma block. The chroma intra mode signals and the corresponding luma intra prediction modes are shown in the following table:

[0061]

[0062] Since the block partitioning structure of independent luminance and chrominance components is enabled in I slices, a chrominance block may correspond to multiple luminance blocks. Therefore, for the chrominance derivative mode (DM) mode, the intra prediction mode of the corresponding luminance block covering the center position of the current chrominance block is directly inherited.

[0063] A unified binarization table (mapped to bin string) is used for chroma intra prediction mode according to the following table:

[0064]

[0065]

[0066] In this table, the first bin indicates whether it is normal mode (0) or LM mode (1). If it is LM mode, then the next bin indicates whether it is LM_CHROMA (0). If it is not LM_chroma, the next bin indicates whether it is LM_L (0) or LM_A (1). For this case, when sps_cclm_enabled_flag is 0, the first bin of the binarization table of the corresponding intra_chroma_pred_mode can be discarded before entropy encoding and decoding. Or, in other words, the first bin is inferred to be 0 and therefore will not be encoded and decoded. This single binarization table is used for the cases where sps_cclm_enabled_flag is equal to 0 and 1. The first two bins in the table are context encoded and decoded with their own context model, and the remaining bins are bypass encoded and decoded.

[0067] In addition, to reduce intra chroma latency in dual trees, when the 64x64 luma codec tree node is not split (and the ISP does not have CUs for 64x64) or is split with QT, the chroma CUs in the 32x32 / 32x16 chroma codec tree nodes are allowed to use CCLM as follows:

[0068] If a 32x32 chroma node is not split or is split with QT, all chroma CUs in the 32x32 node can use CCLM.

[0069] If a 32x32 chroma node is split using horizontal BT, and the 32x16 child node is not split or uses vertical BT, all chroma CUs in the 32x16 chroma node can use CCLM.

[0070] In all other luma and chroma codec tree partitioning conditions, CCLM is not allowed for chroma CUs. II. Multi-mode CCLM (MMLM)

[0071] The multi-model CCLM mode (MMLM) uses two models to predict chrominance samples from the luma samples of the entire CU. Similar to CCLM, three multi-model CCLM modes (MMLM_LA, MMLM_A, and MMLM_L) are used to indicate whether to use both the upper and left neighboring samples, only the upper neighboring samples, or only the left neighboring samples in the derivation of model parameters.

[0072] In MMLM, the adjacent luminance samples and adjacent chrominance samples of the current block are divided into two groups, each of which is used as a training set to derive a linear model (i.e., derive a specific α and β for a specific group). In addition, the samples of the current luminance block are also classified according to the same rules as the classification of the adjacent luminance samples.

[0073] Figure 2 An example of classifying neighboring samples into two groups is marked. The threshold is calculated as the average of the neighboring reconstructed luma samples. Neighboring samples at [x, y] are classified into the first group if Rec'L[x, y] <= Threshold, and neighboring samples at [x, y] are classified into the second group if Rec'L[x, y]> Threshold. Therefore, the multi-model CCLM prediction of chroma samples is:

[0074] Predc[x,y]=α1×Rec′L[x,y]+β1if Rec′L[x,y]≤Threshold

[0075] Predc[x,y]=α2×Rec′L[x,y]+β2if Rec'L[x,y]>Threshold III. Convolutional cross-component model

[0076] In some embodiments, a convolution cross component model (CCCM) is applied to improve cross component prediction performance. For some embodiments, the convolution model has a 7-tap filter with a 5-tap plus signed shape spatial component, a nonlinear term, and a bias term. The input of the spatial 5-tap component of the filter includes a center (C) luminance sample, which matches the chrominance sample to be predicted, and its upper / north (N), lower / south (S), left / west (W), and right / east (E) neighboring samples. Figure 3 Conceptual illustration of the spatial components of a convolution filter. The nonlinear term (denoted as P) is expressed as a power of two of the center luminance sample C, scaled by the sample value range of the content:

[0077] P = (C*C + midVal) >> bitDepth (9)

[0078] Therefore, for 10-bit content, the nonlinear term P is calculated as:

[0079] P = (C*C + 512) >> 10 (10)

[0080] The bias term (denoted as B) represents a scalar offset between the input and output (similar to the bias term in CCLM) and is set to the intermediate chroma value (512 for 10-bit content). The output of the filter is calculated as the convolution between the filter coefficients ci and the input value and is clipped to the valid chroma sample range:

[0081] predChromaVal = c0C + c1N + c2S + c3E + c4W + c5P + c6B (11)

[0082] The filter coefficients ci are calculated by minimizing the MSE between predicted and reconstructed chroma samples over a reference region. Figure 4 A reference region for deriving filter coefficients for the convolution model of the current block is shown. The reference region includes (reference) rows of (chrominance) samples above and to the left of the current block 400. (In this example, the current block 400 is a PU). The reference region extends one PU width to the right and one PU height below the PU boundary. The region is adjusted to include only available samples. The extended area of ​​the reference region is used to support "side samples" of the spatial filter with a signed shape, and is filled in when unavailable.

[0083] The minimization of MSE is achieved by calculating the autocorrelation matrix of the luminance input and the cross-correlation vector between the luminance input and the chrominance output. The autocorrelation matrix is ​​decomposed by LDL, and the final filter coefficients are calculated using the inverse permutation method. This process is similar to the calculation of the ALF filter coefficients in ECM, however, in some embodiments, LDL decomposition is selected instead of Cholesky decomposition to avoid the use of square root operations.

[0084] IV Multiple Linear Models

[0085] For some embodiments, the relationship between two sets of data X and Y (reference sample and current sample, respectively) can be represented by the following linear model:

[0086] y=c0 x+c1 (12)

[0087] More generally, multilinear models can be used to represent more complex relationships between data sets X and Y:

[0088] y=c0 x0+c1 x1+c2 x2+…+cN-1 xN-1+cN (13)

[0089] The multilinear model can be rewritten as:

[0090] y=c0 x0+c1 x1+c2 x2+…+cN-1xN-1+cN=cf T

[0091] c=[c0,c1,c2,…cN-1,cN]

[0092] f=[x0,x1,x2,…xN-1] (14)

[0093] The optimal parameter set c can be obtained by minimizing the SSD (sum of squared differences) of (y–cf T) for all regression data. The SSD minimization problem is equivalent to solving the matrix equation Ax=b, where A is the autocorrelation matrix of the eigenvector f and b is the cross-correlation vector of the eigenvector f and the target y. The matrix equation Ax=b can be solved using Gaussian elimination or Cholesky or LDL decomposition.

[0094] V. Unified Cross-Component Prediction

[0095] A. Unified High-level Model

[0096] Some embodiments of the present application provide methods for improving cross component prediction accuracy or codec performance. The video codec does not use a linear model, but uses a high-order model to perform cross component prediction. The high-order model includes a k-tap space term, a nonlinear term (denoted as P), and a bias term (denoted as B). The high-order model can be specified as:

[0097]

[0098] in is the downsampled reconstructed brightness sample at position (i, j), yes A neighboring sample of x , b, and c are model parameters. Figure 5 The reconstructed luma sample and its neighboring samples are marked. In some embodiments, the high-order model formula (15) is used to derive model parameters between color components, or between samples of the current frame and the reference frame. The model parameters can be adaptively configured according to a specific codec mode or tool. For example, the chrominance value of the CCCM predicted by formula (10) can be represented by the high-order model formula (15) by setting the following: recL'(i,j) = C; neiRecL'(1) = N; neiRecL'(2) = W; neiRecL'(3) = E; and neiRecL'(4) = S. In some embodiments, some different codec modes can share the same model and the same model parameter derivation method.

[0099] In some embodiments, for one coding mode, the equation used to generate the chrominance predictor is as follows: (This can be viewed as a special case of equation (15) by setting parameters a1 to ak-1 and b to zero)

[0100]

[0101] In some embodiments, for one codec mode, the equation for generating the chrominance predictor is as follows: (This can be viewed as a special case of equation (15) by setting parameters a1 to ak-1 to zero)

[0102]

[0103] In some embodiments, for one codec mode, the equation for generating the chrominance predictor is as follows: (This can be viewed as a special case of equation (15) by setting parameters a5 to ak-1 and b to zero)

[0104]

[0105] In some embodiments, for one codec mode, the equation for generating the chrominance predictor is as follows: (This can be viewed as a special case of equation (15) by setting parameters a5 to ak-1 and b to zero)

[0106]

[0107] In some embodiments, for one codec mode, the equation for generating the chrominance predictor is as follows: (This can be viewed as a special case of equation (15) by setting parameters a10 to ak-1 and b to zero)

[0108]

[0109] In some embodiments, for one codec mode, the equation used to generate the illumination compensation predictor is as follows: (This can be viewed as a special case of equation (15) by setting parameters a1 to ak-1 and b to zero)

[0110] pred LIC (i,j)=a0×rec (i,j) +c×B (21)

[0111] For some embodiments using a linear model or a high-order model (e.g., formula (15) or any special case of formula (15)), the optimal parameters can be obtained by solving the matrix equation Ax=b, where A is the autocorrelation matrix, b is the cross-correlation vector, and x is the optimal parameter to be solved. In some embodiments, for different codec tools, the same method or unified solver is used to solve the matrix equation Ax=b.

[0112] In some embodiments, an elimination-based method (e.g., Gaussian elimination) can be used to solve the equation Ax=b and find the optimal vector x for a linear model or a higher-order model. In some embodiments, a decomposition-based method (e.g., Cholesky or LDL decomposition) can be used to solve the equation Ax=b and find the optimal vector x, which is the optimal parameter for a linear model or a higher-order model. In some embodiments, an iteration-based method (e.g., Jacobi method) can be used to solve the equation Ax=b and find the optimal vector x, which is the optimal parameter for a linear model or a higher-order model.

[0113] For example, for CCCM, the predicted chromaticity value formula (10) can be rewritten as:

[0114] y = c0C + c1N + c2S + c3E + c4W + c5P + c6B (22)

[0115] A is the autocorrelation matrix of the eigenvectors f = [C, N, S, E, W, P, B]

[0116]

[0117] b is the cross-correlation vector between the feature vector f and the target y

[0118]

[0119] B. Uniform Selection of Regression Data

[0120] In some embodiments, the selection of regression data (including reference samples and current samples) when deriving model parameters can also be unified between different coding modes. The same input interface of the reference samples of the high-order model is used or shared between multiple different coding modes.

[0121] In some embodiments, different codec tools share the same method to define a unified data range that can be used for regression. For example, the reference / current samples can be divided into many groups, and the video codec executing a specific codec mode can input the required reference / current samples by selecting those reference / current samples belonging to the specified group as regression data, thereby deriving the corresponding model parameters.

[0122] In some embodiments, different codec tools may share the same method to handle invalid data. Some codec tools may only allow samples on the upper side or the left side to be regressed, while processing the reference data on the other side as invalid reference data, setting it to NULL, 0, or a predetermined inactive value. For example, assuming that the input interface requires reference samples on the upper side, upper right side, upper left side, and lower left side of the current block and / or the reference block (the reference block may be a luminance block corresponding to the current block or a reference block for motion compensation), if the current codec mode only uses reference samples on one side, the video codec may set the reference samples on the other side to NULL, 0, or a predefined value, and render those reference samples not used in the model derivation process as inactive. As another example, if the required reference data is outside the restricted boundary (slice boundary, sub-picture boundary, or CTU boundary), the video codec may process the reference data in the external area as invalid reference data by setting the reference data to NULL, 0, a predefined inactive value, or repeating it from the adjacent available data.

[0123] In some embodiments, the codec mode may specify that a high-pass filter or a low-pass filter is applied to a reference / current sample to obtain a gradient or smoothed / downsampled reference / current sample, and the video codec may input the required reference / current sample when executing the codec mode, and then apply a high-pass filter or a low-pass filter to the input reference / current sample.

[0124] In some embodiments, one reference line buffer is used by different codec modes that share the same high-order model. The number of reference lines can be 2, 3, 4, 5, or 6 for different codec modes. In some embodiments, regression data for different codecs are selected from the same number of reference lines within a uniform range. If a required reference line falls outside a restriction boundary (such as a slice boundary, a sub-picture boundary, or a CTU boundary), the content of the required reference line is repeated from one or more available reference lines. In some embodiments, reference lines that fall outside the restriction boundary are not used in the model parameter derivation process.

[0125] Figure 6 The convolutional model generation system 600 is conceptually illustrated and can be configured for different codecs. In some embodiments, the system 600 is implemented by a video codec (encoder or decoder). The convolutional model generation system 600 can be configured for one of several codecs. Each codec has a corresponding configuration set for the convolutional model generation system 600.

[0126] As shown in the diagram, the codec multiplexer 610 selects one of several configuration data sets based on the codec select signal. The selected configuration data set 605 is used to configure the data selector 620, the filter engine 630, and the regression engine 640.

[0127] The data selector 620 retrieves the reconstructed samples from the memory 660 (e.g., the reconstructed picture buffer 850 or the decoded picture buffer 1150). The filtering engine 630 may apply a filtering operation to the reconstructed samples. The reconstructed samples (possibly filtered) are then used as regression data by the regression engine 640 to generate the convolution model 650.

[0128] The data selector 620 can be configured to retrieve the reconstructed samples required by the selected codec tool. In some embodiments, the reconstructed samples associated with the current block are divided into several groups, and the selected configuration data set 605 specifies which group (several groups) of reconstructed samples are to be retrieved. In some embodiments, the selected configuration data set specifies which one or several reference lines are to be used for the selected codec tool. In some embodiments, the selected configuration data set specifies the processing of invalid reference data for the selected codec tool. For example, when the reference samples or reference lines required by the selected codec tool fall outside the specified boundaries, the data selector 620 can set the reference data to NULL, 0, a predefined inactive value, or repeat from the adjacent available data.

[0129] The selected configuration data set 605 also configures the filter engine 630 to apply a high pass or low pass filter on the reconstructed samples. This allows filtering to be applied on the regression data as required by the selected codec.

[0130] The regression engine 640 can be configured to perform data regression based on the reconstructed samples provided by the filter engine 630 and the data selector 620. The regression engine 640 can perform LDL decomposition or Cholesky decomposition to generate model parameters for the convolution model 650. The convolution model 650 is capable of supporting formula (15), but can be configured by the selected configuration data set 605 to support any of the formulas (16)-(22) by setting the specified terms of the higher-order model to zero. With the generated model parameters, the convolution model 650 can be used to generate prediction samples for encoding or decoding the current block.

[0131] C. General Prediction of Component Samples

[0132] The method of the high-order convolution model described in this section can be used for cross-component prediction, for example, using luma samples as reference data to predict chroma samples of the same block. In some embodiments, the high-order convolution model described in this section can be constructed based on any two groups of component samples as regression data (X and Y), and the constructed model can be applied to make predictions between another two groups of component samples with similar relationships. For example, the high-order convolution model can be constructed based on component samples of a reference block and component samples of a reference template adjacent to the reference block, and the constructed model can be used to perform predictions on component samples of the current block based on component samples of a current template adjacent to the current block.

[0133] Fig. 7A -B conceptually illustrates an example source of component samples that can be used to build a high-order convolution model. These figures show a current block and its reference block. The current block 710 is in the current picture 700. The reference block 711 is a predictor of the current block 710 based on inter-frame prediction or intra-frame prediction.

[0134] Fig. 7A It is shown that reference block 711 is a predictor under inter prediction for current block 710. As shown in the figure, motion vector (MV) 720 of current block 710 determines reference block 711 in reference picture 701 as a predictor for the current block for inter prediction.

[0135] Figure 7B Reference block 711 is shown to be a predictor for intra prediction of current block 710. As shown in the diagram, an intra prediction direction or mode 721 or block vector (BV) identifies or derives reference block 711 from samples of current picture 700 and is determined as a predictor for current block 710 for intra prediction.

[0136] like Fig. 7A As shown in FIG. 1-B, the current template area 730 includes reconstructed samples adjacent to the current block 710 in the current picture 700. The current template area 730 has a corresponding or juxtaposed reference template area 731 adjacent to the reference block 711 in the current picture 700 (for intra prediction) or the reference picture 701 (for inter prediction). The current sample can be collected from within or around the current block 710 and the current template area 730. The reference sample can be collected from within or around the reference block 711 and the reference template area 731. In some embodiments, the reference sample and the current sample are used to derive a convolution model.

[0137] The convolution model may be a cross-component model. For example, in some embodiments, the convolution model may be constructed using a reference and current sample of a first color component in a collocated region (e.g., between a Y sample in the reference template 731 and a Y sample in the current template 730), and used to predict a sample of a second color component in the collocated region (e.g., a fine prediction from Cr / Cb of the reference block 711 to Cr / Cb of the current block 710). For another example, in some embodiments, the convolution / cross-component model may be constructed using corresponding cross-component references and / or current samples of the first and second color components of the first region (e.g., Y and Cr / Cb in the reference block 711 or the current template 730), and used to perform cross-component prediction in the second region (e.g., a prediction from Y to Cr / Cb in the current block 710).

[0138] The convolution model can also be used in a non-cross-component manner. For example, a convolution model derived based on reference and current samples of the first color component in and around the first block (e.g., between the Y sample in the reference template 731 and the Y sample of the reference block 711) can be used to predict samples of the same first color component in the second block (e.g., prediction from the Y sample of the current template 730 to the Y sample of the current block 710).

[0139] Any of the methods proposed above can be implemented in an encoder and / or a decoder. For example, any of the methods proposed can be implemented in an inter / intra / prediction module of an encoder and / or an inter / intra / prediction module of a decoder. Alternatively, any of the methods proposed can be implemented as a circuit coupled to an inter / intra / prediction module of an encoder and / or an inter / intra / prediction module of a decoder to provide information required by the inter / intra / prediction module.

[0140] VI. Video Encoder Example

[0141] Figure 8An exemplary video encoder 800 that can implement cross-component prediction is shown. As shown, the video encoder 800 receives an input video signal from a video source 805 and encodes the signal into a bitstream 895. The video encoder 800 has a plurality of components or modules for encoding the signal from the video source 805, including at least a transform module 810, a quantization module 811, an inverse quantization module 814, an inverse transform module 815, an intra-picture estimation module 820, an intra-frame prediction module 825, a motion compensation module 830, a motion estimation module 835, an in-loop filter 845, a reconstructed picture buffer 850, an MV buffer 865, an MV prediction module 875, and some components in the entropy encoder 890. The motion compensation module 830 and the motion estimation module 835 are part of the inter-frame prediction module 840.

[0142] In some embodiments, modules 810-890 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device or electronic device. In some embodiments, modules 810-890 are modules of hardware circuits implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 810-890 are illustrated as separate modules, some modules may be combined into a single module.

[0143] The video source 805 provides a raw video signal representing pixel data for each video frame without compression. A subtractor 808 calculates the difference between the raw video pixel data of the video source 805 and the predicted pixel data 813 from the motion compensation module 830 or the intra prediction module 825 as a prediction residual 809. The transform module 810 converts the difference (or residual pixel data or residual signal) into transform coefficients 816 (e.g., by performing a discrete cosine transform, or DCT). The quantization module 811 quantizes the transform coefficients 816 into quantized data (or quantized coefficients) 812, which are encoded into a bitstream 895 by an entropy encoder 890.

[0144] The inverse quantization module 814 dequantizes the quantized data (or quantized coefficients) 812 to obtain transform coefficients 816, and the inverse transform module 815 inversely transforms the transform coefficients 816 to generate a reconstructed residual 819. The reconstructed residual 819 is added to the predicted pixel data 813 to generate reconstructed pixel data 817. In some embodiments, the reconstructed pixel data 817 is temporarily stored in a line buffer (not shown) for intra-picture prediction and spatial MV prediction. The reconstructed pixels are filtered by the in-loop filter 845 and stored in a reconstructed picture buffer 850. In some embodiments, the reconstructed picture buffer 850 is an external storage of the video encoder 800. In some embodiments, the reconstructed picture buffer 850 is an internal storage of the video encoder 800.

[0145] The intra estimation module 820 performs intra prediction based on the reconstructed pixel data 817 to generate intra prediction data. The intra prediction data is provided to the entropy encoder 890 to be encoded into a bitstream 895. The intra prediction data is also used by the intra prediction module 825 to generate predicted pixel data 813.

[0146] The motion estimation module 835 performs inter-frame prediction by generating MVs to refer to pixel data of a previously decoded frame stored in the reconstructed picture buffer 850. These MVs are provided to the motion compensation module 830 to generate predicted pixel data.

[0147] The video encoder 800 uses MV prediction to generate a predicted MV, and instead of encoding the complete actual MV in the bitstream, the difference between the MV used for motion compensation and the predicted MV is encoded as residual motion data and stored in the bitstream 895.

[0148] The MV prediction module 875 generates a predicted MV based on a reference MV generated for encoding a previous video frame, i.e., a motion compensation MV for performing motion compensation. The MV prediction module 875 retrieves the reference MV of the previous video frame from the MV buffer 865. The video encoder 800 stores the MV generated for the current video frame in the MV buffer 865 as a reference MV for generating the predicted MV.

[0149] The MV prediction module 875 uses the reference MV to create a predicted MV. The predicted MV can be calculated by spatial MV prediction or temporal MV prediction. The difference (residual motion data) between the predicted MV and the motion compensation MV (MC MV) of the current frame is encoded by the entropy encoder 890 to the bit stream 895.

[0150] The entropy encoder 890 encodes various parameters and data into a bitstream 895 by using an entropy coding technique, such as context adaptive binary arithmetic coding (CABAC) or Huffman coding. The entropy encoder 890 encodes various header elements, flags, and quantized transform coefficients 812 and residual motion data as syntax elements into the bitstream 895. The bitstream 895 is in turn stored in a storage device or transmitted to a decoder via a communication medium such as a network.

[0151] The in-loop filter 845 performs filtering or smoothing operations on the reconstructed pixel data 817 to reduce codec artifacts, particularly at block boundaries. In some embodiments, the filtering or smoothing operations performed by the in-loop filter 845 include a block unlock filter (DBF), a sample self-adjusting offset (SAO), and / or a self-adjusting loop filter (ALF).

[0152] Fig. 9Portions of a video encoder 800 implementing a unified cross-component convolution model are shown. When the current block is encoded or decoded by cross prediction, the motion estimation module 835 provides an MV, which is used by the motion compensation module 830 to identify a reference block in a reference picture. When the current block is encoded or decoded by intra prediction, the intra prediction estimation module 820 provides an intra mode or BV, which is used by the intra prediction module 825 to identify a reference block in the current picture. In some embodiments, the reference block is used as an initial predictor for the current block. The initial predictor may include component samples (luminance or chrominance) of a reference block for predicting component samples of the current block, or component samples of the current block for cross-component prediction.

[0153] To derive the convolution model 910, the regression data selection module 930 retrieves component samples of pixels in and / or around the current block and in and / or around the reference block from the reconstructed picture buffer 850 as regression data, which may include reference samples (X) and current samples (Y). The regression data selection module 930 selects as reference / current samples from different sample sets (e.g., different reference rows or different sample groups) according to the codec tool selected for the current block.

[0154] The model builder 905 uses the regression data to derive the parameters of the convolution model 910 using techniques such as elimination, iteration, or decomposition. The convolution model 910 is a high-order model with a plurality of filter taps, which can be configured to support different codecs, for example, by setting certain items or filter taps to zero according to the codec selected for the current block. Figure 6 The derivation of the convolutional model 910 for different codecs is described.

[0155] The convolution model 910 is applied to the initial predictor 915 to generate a refined predictor 925. Samples of the refined predictor 925 may be used as the predicted pixel data 813. In some embodiments, the refined intra-frame prediction or inter-frame prediction performed by the convolution model 910 may be explicitly or implicitly enabled or disabled (so that the samples of the initial predictor are used as the predicted pixel data 813), and the entropy encoder 890 may signal this to the syntax element. In some embodiments, the prediction generator 920 may combine the initial predictor 915 and the refined predictor 925 (e.g., as a weighted sum) as the predicted pixel data 813.

[0156] Fig.10The process 1000 for deriving and using a unified convolutional model that can be configured for different codec tools is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing the encoder 800 perform the process 1000 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing the encoder 800 performs the process 1000.

[0157] The encoder receives (at 1010) data to be encoded as a current pixel block in a current picture of a video. The encoder receives (at 1020) a selection result of selecting a codec from among a plurality of codecs.

[0158] The encoder specifies (at 1030) a regression data set according to the selection of the codec tool. The specified regression data set is selected from component samples in or adjacent to the reference block or current block. In some embodiments, the regression data set is selected from a plurality of reference lines adjacent to the current block, and regression data of different codec tools are selected from different reference lines. In some embodiments, the regression data set is selected from reference samples divided into a plurality of reference sample groups, and regression data of different codec tools are selected from different reference sample groups. In some embodiments, when the regression data set includes invalid samples that fall outside a restricted boundary (e.g., a slice boundary, a sub-picture boundary, or a CTU boundary), the encoder performs an action according to the selection of the codec tool (e.g., setting the invalid samples to NULL, 0, or a predetermined inactive value).

[0159] The encoder (at 1040) specifies the configuration of the convolution model based on the selection of the codec tool. The convolution model is configurable to support each codec tool. In some embodiments, the convolution model is a high-order model with a filter having more than two filter taps. In some embodiments, the convolution model includes a plurality of items, and the encoder configures the convolution model by setting one or more items to zero. The convolution model can have a plurality of taps corresponding to component samples around the center component sample. The center sample can be a luminance sample that matches the chrominance sample to be predicted. The convolution model can include (signed shape) multi-tap spatial components, nonlinear terms, and bias terms, and the input to the spatial 5-tap component includes the center (C) component sample and its north (N), south (S), west (W) and east (E) neighboring component samples.

[0160] The encoder (at 1050) derives parameters of the convolutional model according to the specified configuration by applying the regression data set. In some embodiments, the parameters of the convolutional model are derived by finding the best vector of the high-order model using, for example, an elimination-based method, a decomposition-based method, or an iteration-based method.

[0161] The encoder applies (at 1060) the convolution model with the generated parameters to the reference component sample set to obtain the prediction component sample set. The encoder encodes (at 1070) the current block by using the prediction component samples to generate a prediction residual and reconstruct the current block.

[0162] VII. Video Decoder Example

[0163] In some embodiments, the encoder may signal (or generate) one or more syntax elements in the bitstream so that the decoder may parse the one or more syntax elements from the bitstream.

[0164] Fig.11 An example of a video decoder 1100 that can implement cross-component prediction is shown. As shown in the figure, the video decoder 1100 is an image decoding or video decoding circuit that receives a bitstream 1195 and decodes the content of the bitstream into pixel data of a video frame for display. The video decoder 1100 has a plurality of components or modules for decoding the bitstream 1195, including some components selected from an inverse quantization module 1111, an inverse transform module 1110, an intra-frame prediction module 1125, a motion compensation module 1130, an in-loop filter 1145, a decoded picture buffer 1150, an MV buffer 1165, an MV prediction module 1175, and a parser 1190. The motion compensation module 1130 is a part of the inter-frame prediction module 1140.

[0165] In some embodiments, modules 1110-1190 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 1110-1190 are modules of hardware circuits implemented by one or more ICs of an electronic device. Although modules 1110-1190 are illustrated as separate modules, some of these modules may be combined into a single module.

[0166] The parser 1190 (or entropy decoder) receives the bitstream 1195 and performs initial parsing according to the syntax defined by the video codec or image codec standard. The parsed syntax elements include various header elements, flags, and quantization data (or quantization coefficients) 1112. The parser 1190 parses out the various syntax elements by using entropy coding techniques, such as context-adaptive binary arithmetic coding (CABAC) or Huffman coding.

[0167] The inverse quantization module 1111 dequantizes the quantized data (or quantized coefficients) 1112 to obtain transform coefficients, and the inverse transform module 1110 inversely transforms the transform coefficients 1116 to generate a reconstructed residual signal 1119. The reconstructed residual signal 1119 is added to the predicted pixel data 1113 from the intra prediction module 1125 or the motion compensation module 1130 to generate decoded pixel data 1117. The decoded pixel data is filtered by the in-loop filter 1145 and stored in the decoded picture buffer 1150. In some embodiments, the decoded picture buffer 1150 is an external storage of the video decoder 1100. In some embodiments, the decoded picture buffer 1150 is an internal storage of the video decoder 1100.

[0168] The intra prediction module 1125 receives intra prediction data from the bitstream 1195 and generates predicted pixel data 1113 from decoded pixel data 1117 stored in the decoded picture buffer 1150. In some embodiments, the decoded pixel data 1117 is also stored in a line buffer (not shown) for intra prediction and spatial MV prediction.

[0169] In some embodiments, the contents of the decoded picture buffer 1150 are used for display. The display device 1155 directly retrieves the contents of the decoded picture buffer 1150 for display, or retrieves the contents of the decoded picture buffer to a display buffer. In some embodiments, the display device receives pixel values ​​from the decoded picture buffer 1150 via pixel transfer.

[0170] The motion compensation module 1130 generates predicted pixel data 1113 from decoded pixel data 1117 stored in the decoded picture buffer 1150 according to motion compensated MVs (MC MVs). These motion compensated MVs are decoded by adding the residual motion data received from the bitstream 1195 to the predicted MVs received from the MV prediction module 1175.

[0171] The MV prediction module 1175 generates a predicted MV based on a reference MV generated for decoding a previous video frame, for example, a motion compensated MV for performing motion compensation. The MV prediction module 1175 retrieves the reference MV of the previous video frame from the MV buffer 1165. The video decoder 1100 stores the motion compensated MV generated for decoding the current video frame in the MV buffer 1165 as a reference MV for generating the predicted MV.

[0172] The in-loop filter 1145 performs filtering or smoothing operations on the decoded pixel data 1117 to reduce coding artifacts, particularly at block boundaries. In some embodiments, the filtering or smoothing operations performed by the in-loop filter 1145 include a block unlock filter (DBF), a sample adaptive offset (SAO), and / or an adaptive loop filter (ALF).

[0173] Fig.12 Portions of a video decoder 1100 implementing a unified cross-component convolution model are shown. When the current block is encoded by cross prediction, the entropy decoder module 1190 or the MV buffer 865 can provide an MV that is used by the motion compensation module 1130 to identify a reference block in a reference picture. When the current block is encoded and decoded by intra prediction, the entropy decoder module 1190 provides an intra mode or BV that is used by the intra prediction module 1125 to identify a reference block in the current picture. In some embodiments, the reference block is used as an initial predictor for the current block. The initial predictor may include component samples (luminance or chrominance) of a reference block for predicting component samples of the current block, or component samples of the current block for cross-component prediction.

[0174] To derive the convolution model 1210, the regression data selection module 1230 retrieves component samples of pixels in and / or around the current block and in and / or around the reference block from the reconstructed picture buffer 1150 as regression data, which may include reference samples (X) and current samples (Y). The regression data selection module 1230 selects as reference / current samples from different sample sets (e.g., different reference rows or different sample groups) according to the codec tool selected for the current block.

[0175] The model builder 1205 uses the regression data to derive the parameters of the convolution model 1210 using techniques such as elimination, iteration, or decomposition. The convolution model 1210 is a high-order model with a plurality of filter taps that can be configured to support different codecs, for example, by setting certain items or filter taps to zero depending on the codec selected for the current block. Figure 6 The derivation of the convolutional model 1210 for different codecs is described.

[0176] The convolution model 1210 is applied to the initial predictor 1215 to generate a refined predictor 1225. Samples of the refined predictor 1225 may be used as the predicted pixel data 1113. In some embodiments, the entropy decoder 1190 may receive syntax elements for explicitly enabling or disabling the refined intra-frame prediction or inter-frame prediction of the convolution model 1210 so that the samples of the initial predictor may be used as the predicted pixel data 1113. In some embodiments, the prediction generator 1220 may combine the initial predictor 1215 and the refined predictor 1225 (e.g., as a weighted sum) as the predicted pixel data 1113.

[0177] Fig.13The process 1300 for deriving and using a unified convolutional model that can be configured for different codec tools is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing the decoder 1100 perform the process 1300 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing the decoder 1100 performs the process 1300.

[0178] The decoder receives (at 1310) data to be decoded as a current pixel block in a current picture of a video. The decoder receives (at 1320) a selection result of selecting a codec tool among a plurality of codec tools.

[0179] The decoder (at 1330) specifies a regression data set according to the selection of the codec tool. The specified regression data set is selected from component samples within or adjacent to the reference block or current block. In some embodiments, the regression data set is selected from a plurality of reference lines adjacent to the current block, and regression data of different codec tools are selected from different reference lines. In some embodiments, the regression data set is selected from reference samples divided into a plurality of reference sample groups, and regression data of different codec tools are selected from different reference sample groups. In some embodiments, when the regression data set includes invalid samples that fall outside a restricted boundary (e.g., a slice boundary, a sub-picture boundary, or a CTU boundary), the decoder performs an action according to the selection of the codec tool (e.g., setting the invalid samples to NULL, 0, or a predetermined inactive value).

[0180] The decoder (at 1340) specifies the configuration of the convolution model based on the selection of the codec tool. The convolution model is configurable to support each codec tool. In some embodiments, the convolution model is a high-order model with a filter having more than two filter taps. In some embodiments, the convolution model includes a plurality of items, and the decoder configures the convolution model by setting one or more items to zero. The convolution model can have a plurality of taps corresponding to component samples around the center component sample. The center sample can be a luminance sample that matches the chrominance sample to be predicted. The convolution model can include (signed shape) multi-tap spatial components, nonlinear terms, and bias terms, and the input to the spatial 5-tap component includes the center (C) component sample and its north (N), south (S), west (W) and east (E) neighboring component samples.

[0181] The decoder (at 1350) derives the parameters of the convolutional model according to the specified configuration by applying the regression data set. In some embodiments, the parameters of the convolutional model are derived by finding the best vector of the high-order model using, for example, an elimination-based method, a decomposition-based method, or an iteration-based method.

[0182] The decoder (at 1360) applies the convolution model with the generated parameters to the reference component sample set to obtain the prediction component sample set. The decoder (at 1370) reconstructs the current block by using the prediction component samples. The decoder can then provide the reconstructed current block for display as part of the reconstructed current picture.

[0183] VIII. Electronic System Examples

[0184] Many of the features and applications described above are implemented as software processes that are specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more computing or processing units (e.g., one or more processors, processor cores, or other processing units), they cause the processing unit to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, random access memory (RAM) chips, hard disks, erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), etc. Computer-readable media do not include carrier waves and electronic signals transmitted wirelessly or through wired connections.

[0185] In this specification, the term "software" refers to firmware residing in read-only memory or application programs stored in magnetic memory, which can be read into memory for processing by a processor. In addition, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while maintaining different software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of independent programs that jointly implement the software inventions described herein are within the scope of this application. In some embodiments, the software program, when installed and run on one or more electronic systems, defines one or more specific machine implementations that execute and implement the operations of the software program.

[0186] Fig.14 The electronic system 1400 for implementing some embodiments of the present application is conceptually illustrated. The electronic system 1400 may be a computer (e.g., a desktop computer, a personal computer, a tablet computer, etc.), a phone, a PDA, or any other type of electronic device. Such an electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 1400 includes a bus 1405, a processing unit 1410, a graphics processing unit (GPU) 1415, a system memory 1420, a network 1425, a read-only memory 1430, a permanent storage device 1435, an input device 1440, and an output device 1445.

[0187] The bus 1405 is collectively referred to as all system, peripheral device, and chipset buses, which communicatively connect the numerous internal devices of the electronic system 1400. For example, the bus 1405 communicatively connects the processing unit 1410 with the GPU 1415, the read-only memory 1430, the system memory 1420, and the permanent storage device 1435.

[0188] From these various storage units, processing unit 1410 retrieves instructions to be executed and data to be processed to perform the processes of the present application. In different embodiments, the processing unit can be a single processor or a multi-core processor. Some instructions are passed to and executed by GPU 1415. GPU 1415 can offload various calculations or supplement the image processing provided by processing unit 1410.

[0189] Read-only memory (ROM) 1430 stores static data and instructions and is used by processing unit 1410 and other modules of the electronic system. On the other hand, permanent storage device 1435 is a read-write storage device. This device is a non-volatile storage unit that can store instructions and data even when electronic system 1400 is turned off. Some embodiments of the present application use a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) as permanent storage device 1435.

[0190] Other embodiments use removable storage devices (such as floppy disks, flash memory devices, etc., and their corresponding disk drives) as permanent storage devices. Like permanent storage device 1435, system memory 1420 is a read-write memory device. However, unlike storage device 1435, system memory 1420 is a volatile read-write memory, such as random access memory. System memory 1420 stores some instructions and data used by the processor at runtime. In some embodiments, processes according to the contents of the present application are stored in system memory 1420, permanent storage device 1435, and / or read-only memory 1430. For example, various memory units include instructions for processing multimedia clips according to some embodiments. From these different storage units, processing unit 1410 retrieves instructions to be executed and data to be processed in order to execute the processes of some embodiments.

[0191] Bus 1405 is also connected to input and output devices 1440 and 1445. Input device 1440 enables a user to communicate information and select commands to the electronic system. Input device 1440 includes an alphanumeric keyboard and pointing device (also known as a "cursor control"), a camera (such as a webcam), a microphone or similar device for receiving voice commands, etc. Output device 1445 displays images generated by the electronic system or outputs data in other ways. Output device 1445 includes a printer and display device, such as a cathode ray tube (CRT) or liquid crystal display (LCD), and a speaker or similar audio output device. Some embodiments include devices such as a touch screen that serve as both input and output devices.

[0192] Finally, if Fig.14 As shown, bus 1405 also couples electronic system 1400 to a network 1425 via a network adapter (not shown). In this manner, the computer can be part of a computer network, such as a local area network ("LAN"), a wide area network ("WAN"), or an intranet, or a network of networks, such as the Internet. Any or all components of electronic system 1400 may be used in conjunction with the present disclosure.

[0193] Some embodiments include electronic components, such as microprocessors, storage devices, and memory, which store computer program instructions on machine-readable or computer-readable media (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, compact disk-read only (CD-ROM), compact disk-recordable (CD-R), compact disk-rewritable (CD-RW), read-only digital versatile disks (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable CD-ROM, ultra-density CD-ROM, any other optical or magnetic media, and floppy disk. Computer-readable media can store a computer program that can be executed by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as produced by a compiler, and files including high-level code that is executed by a computer, electronic component, or microprocessor using an interpreter.

[0194] Although the above discussion refers primarily to microprocessors or multi-core processors executing software, many of the functions and applications described above are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored in the circuit itself. In addition, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM devices.

[0195] In this specification and any patent scope of this application, the terms "computer", "server", "processor", and "memory" refer to electronic or other technical devices. These terms do not include people or groups of people. In this specification, the terms "display" or "show" refer to displaying on an electronic device. In this specification and any patent scope of this application, the terms "computer-readable medium", "computer-readable media", and "machine-readable medium" are entirely limited to tangible, physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, and any other transient signals.

[0196] Although the present application has been described with reference to many specific details, a person skilled in the art will recognize that the present application can be embodied in other specific forms without departing from the spirit of the present application. Fig.10 and Fig.13 ) conceptually illustrates the process. The specific operations of these processes may not be performed in the exact order shown and described. The specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. In addition, the process can be implemented using several sub-processes or as part of a larger macro process. Therefore, it can be understood by those skilled in the art that the content of this application is not limited by the foregoing illustrative details, but is defined by the scope of the appended claims.

[0197] Additional Notes

[0198] The subject matter described herein sometimes shows different components contained in or connected to different other components.It should be understood that these depicted architectures are only examples, and many other architectures can actually be implemented to achieve the same function.Conceptually, the arrangement of any components to achieve the same function can be effectively regarded as "associated" to achieve the desired function.Therefore, any two components combined to achieve a specific function herein can be regarded as "associated" to achieve the desired function, regardless of the architecture or intermediate components.Similarly, any two components associated in this way can also be regarded as "operably connected" or "operably coupled" to achieve the desired function, and any two components that can be so associated can also be regarded as "operably coupled" to achieve the desired function.Specific operable coupling examples include but are not limited to physically compatible and / or physically interactive components and / or wirelessly interactive and / or wirelessly interactive components and / or logically interactive and / or logically interactive components.

[0199] Furthermore, with respect to the use of virtually any plural and / or singular terms herein, one having technical skill can translate from the plural to the singular and / or from the singular to the plural as appropriate to the context and / or application. For the sake of clarity, the various singular / plural permutations may be expressly listed herein.

[0200] Furthermore, the skilled artisan will understand that, in general, the terms used herein, especially in the appended claims, such as the bodies of the appended claims, are generally intended to be "open" terms, e.g., the word "comprising" should be interpreted as "including but not limited to," the word "having" should be interpreted as "having at least," the word "including" should be interpreted as "including but not limited to," etc. The skilled artisan will also understand that if a specific number of an introduced claim recitation is intended, such intent will be explicitly stated in the claim, and in the absence of such a statement, no such intent is present. For example, to facilitate understanding, the following appended claims may contain the use of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, the use of these phrases should not be interpreted as implying that a claim recitation introduced by the indefinite article "a" or "an" limits any claim containing such an introduction to only one such recitation, even if the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an," e.g., "a" and / or "an" should be interpreted as "at least one" or "one or more"; the same applies to definite articles used to introduce claim recitations. Furthermore, even if a specific number of an introduced claim statement is explicitly stated, a skilled person will recognize that such a statement should be interpreted as at least the stated number, e.g., simply stating "two statements" without other modifiers means at least two statements, or two or more statements. Furthermore, where a convention similar to "at least one of A, B, and C, etc. is used, such a construction is generally understood by a skilled person, e.g., "a system having at least one of A, B, and C" will include, but is not limited to, a system having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc. Where a convention similar to "at least one of A, B, or C, etc. is used, such a construction is generally understood by a skilled person, e.g., "a system having at least one of A, B, or C" will include, but is not limited to, a system having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc. A skilled person will also understand that almost any disjunctive word and / or term presenting two or more alternative terms, whether in the detailed description of the embodiments, the claims, or the drawings, should be understood to include one term, either term, or both terms. For example, "A or B" will be understood to include the possibilities of "A" or "B" or "A and B."

[0201] As can be seen from the foregoing, various embodiments of the present disclosure are described herein for illustrative purposes, and various modifications may be made without departing from the scope and spirit of the present disclosure. Therefore, the various embodiments disclosed herein are not intended to be limiting, and the true scope and spirit are indicated by the following claims.

Claims

1. A video encoding and decoding method, the method comprising: Receiving data of a pixel block to be encoded or decoded as a current block of a current picture of a video; receiving a selection result of selecting a codec tool from a plurality of codec tools; According to the selection of the encoding and decoding tool, a regression data set is specified; wherein the specified regression data set is selected from component samples in or adjacent to a reference block or a current block; Specifying a configuration of a convolutional model based on the selection of the codec; wherein the convolutional model is configurable to support each of the plurality of codecs; According to the specified configuration, generating parameters of the convolutional model by applying the regression dataset; applying the convolutional model with the generated parameters to a reference component sample set to obtain a predicted component sample set; and The current block is encoded or decoded by using the prediction component samples.

2. The method according to claim 1, wherein: The convolution model is a high-order model including a filter having more than two filter taps.

3. The method according to claim 2, wherein: The parameters of the convolution model are obtained by using a unified solver, which is used to solve the matrix equation of the multiple codec tools; wherein the matrix equation is represented by Ax=b, A is the autocorrelation matrix, b is the cross-correlation vector, and x is the parameter to be solved.

4. The method according to claim 3, wherein: The unified solver uses an elimination-based approach to determine the optimal vector of the convolutional model as a solution to the parameters of the convolutional model.

5. The method according to claim 3, wherein: The unified solver uses a decomposition-based approach to determine the optimal vector of the convolutional model as a solution to the parameters of the convolutional model.

6. The method according to claim 3, wherein: The unified solver uses an iteration-based approach to determine the optimal vector of the convolutional model as a solution to the parameters of the convolutional model.

7. The method according to claim 3, wherein: The regression data set is selected from a unified range of a plurality of reference lines adjacent to the current block, and the unified range is common to a plurality of codec tools.

8. The method according to claim 7, wherein: The regression data for different codecs are selected from the same number of reference rows within this uniform range.

9. The method according to claim 7, wherein: The unified range of the plurality of reference lines is stored in a line buffer used by the plurality of codec tools.

10. The method according to claim 7, wherein: The regression data set is selected from reference samples divided into a plurality of reference sample groups, and regression data of different encoding and decoding tools are selected from different reference sample groups.

11. The method according to claim 7, wherein: Further including: When the regression data set includes invalid samples, a unified action common to the plurality of codec tools is performed.

12. The method according to claim 11, wherein: The invalid sample refers to a sample that exceeds a limit boundary or is prohibited by at least one of the plurality of codecs.

13. An electronic device, comprising: The video encoding circuit is configured to perform: Receiving data of a pixel block to be encoded or decoded as a current block of a current picture of a video; receiving a selection result of selecting a codec tool from a plurality of codec tools; According to the selection of the encoding and decoding tool, a regression data set is specified; wherein the specified regression data set is selected from component samples in or adjacent to a reference block or a current block; Specifying a configuration of a convolutional model based on the selection of the codec; wherein the convolutional model is configurable to support each of the plurality of codecs; According to the specified configuration, by applying the regression data set, the parameters of the convolutional model are obtained; applying the convolutional model with the generated parameters to a reference component sample set to obtain a predicted component sample set; and The current block is encoded or decoded by using the prediction component samples.

14. A method for video decoding, comprising: Receiving data of a pixel block to be decoded as a current block of a current picture of a video; receiving a selection result of selecting a codec tool from a plurality of codec tools; According to the selection of the encoding and decoding tool, a regression data set is specified; wherein the specified regression data set is selected from component samples in or adjacent to a reference block or a current block; Specifying a configuration of a convolutional model based on the selection of the codec; wherein the convolutional model is configurable to support each of the plurality of codecs; According to the specified configuration, by applying the regression data set, the parameters of the convolutional model are obtained; applying the convolutional model with the generated parameters to a reference component sample set to obtain a predicted component sample set; and The current block is reconstructed by using the prediction component sample set.

Images