Method, apparatus, and computer program product for encoding and decoding digital media content - Patents.com

Abstract

An embodiment relates to a method for encoding / decoding, the method including obtaining a set of input samples (310), obtaining a triangular matrix, a cross-correlation vector and a scaling vector using the set of input samples (320), determining an intermediate vector by a first back-substitution using the triangular matrix and the cross-correlation vector (330), scaling the intermediate vector by a first back-substitution using the scaling vector (340), determining a filter coefficient vector by a second back-substitution using the triangular matrix and the scaled intermediate vector (350), and determining predicted sample values ​​of a block of samples based on the filter coefficient vector (360). An embodiment also relates to an apparatus and a computer program product for implementing the method.

Description

[Technical field]

[0001] The present solution generally relates to encoding and decoding digital media content, such as video or still image data. [Background technology]

[0002] This section is intended to describe a background or context for the invention as described in the claims. The description herein may include concepts that could be pursued, but not necessarily concepts that have been previously conceived or pursued. Thus, unless otherwise indicated herein, the subject matter described in this section is not prior art to the description and claims in this application, and is not admitted to be prior art by inclusion in this section.

[0003] A video coding system may consist of an encoder that converts an input video into a compressed representation suitable for storage / transmission, and a decoder that can restore the compressed video representation to a viewable form again. The encoder may discard some information in the original video sequence in order to represent the video in a more compact form, e.g., to allow the video information to be stored / transmitted at a lower bit rate than would otherwise be required. Summary of the Invention

[0004] The scope of protection sought for the various embodiments of the invention is defined by the independent claims. To the extent there are embodiments and features described herein that do not fall within the scope of the independent claims, they should be interpreted as examples useful for understanding the various embodiments of the invention.

[0005] Various aspects include a method, an apparatus and a computer readable medium having a computer program stored thereon as characterized by what is stated in the independent claims. Various embodiments are disclosed in the dependent claims.

[0006] According to a first aspect, an apparatus is provided, comprising: means for obtaining a set of input samples; means for obtaining a triangular matrix, a cross-correlation vector and a scaling vector utilizing the set of input samples; means for determining an intermediate vector by a first back-substitution using the triangular matrix and the cross-correlation vector; means for scaling the intermediate vector using the scaling vector; means for determining a filter coefficient vector by a second back-substitution using the triangular matrix and the scaled intermediate vector; and means for determining predicted sample values ​​of a block of samples based on the filter coefficient vector.

[0007] According to a second aspect, a method is provided that includes obtaining a set of input samples, obtaining a triangular matrix, a cross-correlation vector and a scaling vector utilizing the set of input samples, determining an intermediate vector by a first back-substitution using the triangular matrix and the cross-correlation vector, scaling the intermediate vector using the scaling vector, determining a filter coefficient vector by a second back-substitution using the triangular matrix and the scaled intermediate vector, and determining a predicted sample value for a block of samples based on the filter coefficient vector.

[0008] According to a third aspect, there is provided an apparatus comprising at least one processor and a memory including computer program code, the memory and the computer program code configured by the at least one processor to cause the apparatus to at least perform the following: obtaining a set of input samples; obtaining a triangular matrix, a cross-correlation vector and a scaling vector utilizing the set of input samples; determining an intermediate vector by a first back-substitution using the triangular matrix and the cross-correlation vector; scaling the intermediate vector using the scaling vector; determining a filter coefficient vector by a second back-substitution using the triangular matrix and the scaled intermediate vector; and determining a predicted sample value for a block of samples based on the filter coefficient vector.

[0009] According to a fourth aspect, there is provided a computer program product including computer program code that, when executed on at least one processor, is configured to cause an apparatus or system to obtain a set of input samples; obtain a triangular matrix, a cross-correlation vector and a scaling vector utilizing the set of input samples; determine an intermediate vector by a first back-substitution using the triangular matrix and the cross-correlation vector; scale the intermediate vector using the scaling vector; determine a filter coefficient vector by a second back-substitution using the triangular matrix and the scaled intermediate vector; and determine predicted sample values ​​for a block of samples based on the filter coefficient vector.

[0010] According to one embodiment, the triangular matrix and the scaling vector are determined by decomposing the autocorrelation matrix into at least a triangular matrix and a scaling vector, the autocorrelation matrix being calculated using a set of reference samples in a defined neighborhood of the block of samples.

[0011] According to one embodiment, the autocorrelation matrix and cross-correlation vectors are determined by subtracting the average luma value from the set of luma values ​​and subtracting the average chroma value from the set of chroma values.

[0012] According to one embodiment, the autocorrelation matrix and the cross-correlation vectors are scaled based on the bit depth of the block of samples.

[0013] According to one embodiment, the process of computing the autocorrelation matrix and the cross-correlation vectors involves bit-wise shifting operations.

[0014] According to one embodiment, the set of samples includes one or more of luminance sample values, functions of luminance sample values, and bias parameters.

[0015] According to an embodiment, the apparatus comprises means for encoding a set of samples and / or means for decoding the encoded set of samples.

[0016] According to one embodiment, a computer program product is embodied on a non-transitory computer readable medium.

[0017] Various embodiments are described in further detail below with reference to the accompanying drawings. [Brief description of the drawings]

[0018] [Figure 1] FIG. 13 is a diagram illustrating an example of an encoding process. [Diagram 2] FIG. 11 is a diagram illustrating an example of a decoding process. [Diagram 3] 1 is a flow chart illustrating a method according to one embodiment. [Figure 4] FIG. 1 illustrates an apparatus according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] In the following, some embodiments are described in the context of one video coding configuration. However, it should be noted that the embodiments of the present invention are not necessarily limited to this particular configuration. The embodiments discussed herein relate to intra-prediction in video or still image coding using sparse linear cross-component regression.

[0020] The following description and drawings are illustrative and should not be construed as unnecessarily limiting. Specific details are provided for a thorough understanding of the present disclosure. However, in some cases, well-known or conventional details are not described in order to avoid obscuring the description. References to one embodiment or an embodiment in the present disclosure may refer to the same embodiment, but do not necessarily refer to at least one of the embodiments.

[0021] References herein to "one embodiment" or "one embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the present disclosure.

[0022] A video codec consists of an encoder and a decoder. The encoder is configured to convert the input video into a compressed representation suitable for storage / transmission. The decoder can restore the compressed video representation back to a viewable form. The encoder may discard some information in the original video sequence in order to represent the video in a more compact form, such as at a lower bit rate.

[0023] In most cases, the basic unit of input to an encoder and output of a decoder is a picture. A picture provided as input to an encoder may be called a source picture, and a picture decoded by a decoder may be called a decoded picture or a reconstructed picture.

[0024] The source picture and the decoded picture each consist of one or more sample arrays, for example one of the following sets of sample arrays: - Luma (Y) only (monochrome) - Luma and two chromas (YCbCr or YCgCo) - Green, Blue, and Red (GBR, also known as RGB) - Arrays representing other unspecified monochrome or tristimulus samplings (e.g., YZX, also known as XYZ)

[0025] A picture may be defined as either a frame or a field. A frame consists of a matrix of luma samples and possibly corresponding chroma samples. A field is a set of alternating sample rows of a frame, and may be used as an encoder input if the source signal is interlaced. The chroma sample array may not be present (so monochrome sampling may be used) or the chroma sample array may be subsampled compared to the luma sample array.

[0026] A bitstream may be defined as a sequence of bits forming a representation of coded pictures and associated data forming one or more coded video sequences, which in some coding formats or standards may be in the form of a network abstraction layer (NAL) unit stream or byte stream. A first bitstream may be followed by a second bitstream in the same logical channel, such as in the same file or in the same connection of a communication protocol. An elementary stream (in the context of video coding) may be defined as a sequence of one or more bitstreams. In some coding formats or standards, the end of a first bitstream may be indicated by a specific NAL unit, which may also be called the end of bitstream (EOB) NAL unit, which is the last NAL unit of the bitstream.

[0027] The phrase "along the bitstream" (e.g., phrases indicating along the bitstream) or along an encoded unit of the bitstream (e.g., phrases indicating along the encoded tiles) may be used in the claims and described embodiments to refer to transmission, signaling, or storage in a manner such that "out-of-band" data is associated with, but not included in, the bitstream or encoded unit. Phrases such as decoding along the bitstream or decoding along an encoded unit of the bitstream may refer to decoding referenced out-of-band data associated with the bitstream or encoded unit (which may be obtained from out-of-band transmission, signaling, or storage). For example, the phrase along the bitstream may be used when the bitstream is contained in a container file, such as a file conforming to the ISO Base Media File Format, and certain file metadata is stored in the file in a manner that associates the metadata with the bitstream, such as, for example, a box in a sample entry of a track that includes the bitstream, a sample group of a track that includes the bitstream, or a timed metadata track associated with a track that includes the bitstream.

[0028] Hybrid video codecs, such as ITU-T H.263 and H.264, may code video information in two phases. First, pixel values ​​in a certain picture region (or "block") are predicted, for example, by motion compensation means (finding and indicating a region in one of the already coded video frames that closely corresponds to the block being coded) or spatial means (using pixel values ​​around the block to be coded in a specified manner). In the first phase, predictive coding may be applied, for example, as so-called sample prediction and / or so-called syntax prediction. In sample prediction, pixel values ​​or sample values ​​in a certain picture region or "block" are predicted. These pixel values ​​or sample values ​​may be predicted, for example, using one or more of motion compensation or intra prediction mechanisms. Second, the prediction error, i.e. the difference between the predicted pixel block and the original pixel block, is coded. This may be performed by transforming the pixel value differences with a specified transform (e.g., the Discrete Cosine Transform (DCT) or a variant thereof), quantizing the coefficients, and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (image quality) and the size of the resulting encoded video representation (file size or transmission bit rate).

[0029] An example of the encoding process is shown in Figure 1. n ), the predicted representation of the image block (P' n ), prediction error signal (D n ), the reconstructed prediction error signal (D' n ), the preliminary reconstructed image (I' n ), the final reconstructed image (R' n ), transformation (T) and inverse transformation (T -1 ), quantization (Q) and dequantization (Q -1 ), entropy coding (E), reference frame memory (RFM), inter prediction (Pinter ), intra prediction (P intra ), mode selection (MS), and filtering (F).

[0030] In some video codecs, such as H.265 / HEVC, a video picture is divided into coding units (CUs) that cover an area of ​​the picture. A CU includes one or more prediction units (PUs) that define a prediction operation for samples in the CU, and one or more transform units (TUs) that define a prediction error coding operation for samples in the CU. A CU may be composed of square sample blocks with a size selectable from a predefined set of possible CU sizes. A CU with a maximum allowed size may be referred to as a largest coding unit (LCU) or coding tree unit (CTU), and a video picture may be divided into non-overlapping CTUs. A CTU may be further divided into a combination of smaller CUs, for example by recursively dividing the CTU and the resulting CU. Each resulting CU may have at least one PU and at least one TU associated with it. Each PU and TU may be further divided into smaller PUs and TUs to increase the granularity of the prediction and prediction error coding operations, respectively. Each PU is associated with prediction information, which defines what prediction is applied to the pixels in that PU (e.g., motion vector information for an inter-predicted PU, intra prediction direction information for an intra-predicted PU). Similarly, each TU is associated with information describing the prediction error decoding process for samples in that TU (e.g., including DCT coefficient information). Whether prediction error coding is applied to each CU may be signaled at the CU level. If there is no prediction error residual associated with a CU, it can be considered that there is no Tu for that CU. The division of an image into CUs and the division of CUs into PUs and TUs may be signaled in the bitstream, allowing the decoder to reproduce the intended structure of these units.

[0031] The decoder may reconstruct the output video by applying a prediction means similar to the encoder to form a predictive representation of the pixel block (using motion or spatial information created by the encoder and stored in the compressed representation) and applying prediction error decoding (the inverse operation of prediction error encoding that restores the quantized prediction error signal to the spatial pixel domain). After applying the prediction means and prediction error decoding means, the decoder is configured to sum the prediction signal and the prediction error signal (pixel values) to form the output video frame. The decoder (and the encoder) may also apply additional filtering means to improve the quality of the output video before passing the output video for display and / or storing it as a prediction reference for future frames in the video sequence. An example of the decoding process is shown in Figure 2. Figure 2 shows a predictive representation of an image block (P' n ), the reconstructed prediction error signal (D' n ), the preliminary reconstructed image (I' n ), the final reconstructed image (R' n ), inverse transformation (T -1 ), inverse quantization (Q -1 ), Entropy Decoding (E -1 ), Reference Frame Memory (RFM), Prediction (Inter or Intra) (P), and Filtering (F).

[0032] Color palette-based coding can be used instead of or in addition to the approach of using sample value prediction and transform coding to indicate the coded sample values. Palette-based coding refers to a group of approaches in which a palette, i.e., a set of colors and associated indexes, is defined, and the value of each sample in a coding unit is represented by indicating an index in the palette. Palette-based coding can achieve good coding efficiency for coding units with a relatively small number of colors (e.g., image regions representing computer screen contents such as text or simple graphics). To improve the coding efficiency of palette coding, various kinds of palette index prediction approaches can be used, or the palette index can be run-length coded to allow larger homogeneous regions to be efficiently represented. Also, escape coding can be used when a CU contains sample values ​​that do not occur repeatedly within the CU. Escape-coded samples are transmitted without reference to a palette index. Instead, a value can be indicated for each escape-coded sample individually.

[0033] When a CU is coded in palette mode, various prediction strategies are used to exploit the correlation between pixels within the CU. For example, mode information can be signaled for each row or pixel, indicating one of the following: The mode can be horizontal mode, meaning that a single palette index is signaled and the entire pixel line shares this index; The mode can be vertical mode, where the entire pixel line is the same as the line above and no further information is signaled; The mode can be normal mode, where for each pixel position, a flag is signaled indicating whether it is the same as the pixel to the left and one of the other pixels, and if not, the color index itself is transmitted separately.

[0034] In a video codec, the motion information may be indicated by means of a motion vector associated with each motion-compensated image block. Each of these motion vectors may represent the displacement of an image block in a picture to be coded (encoder side) or a picture to be decoded (decoder side) and a prediction source block in one of the coded or decoded pictures. To efficiently represent the motion vectors, they may be differentially coded with respect to a block-wise predicted motion vector. In a video codec, the predicted motion vector may be created in a predefined manner, such as by calculating the median value of the coded or decoded motion vectors of neighboring blocks. Another way to create a motion vector prediction is to generate a list of prediction candidates from neighboring blocks and / or co-located blocks in a temporal reference picture, and signal the selected candidate as a motion vector predictor. In addition to predicting the value of the motion vector, the reference index of the coded / decoded picture may also be predicted. The reference index may be predicted from neighboring blocks and / or co-located blocks in a temporal reference picture. Furthermore, highly efficient video codecs may employ an additional motion information encoding / decoding mechanism, often called merging / merge mode, in which all motion field information, including motion vectors and corresponding reference picture indexes in each available reference picture list, is predicted and used without change / modification. Similarly, prediction of motion field information may be performed using motion field information of neighboring blocks and / or co-located blocks in temporal reference pictures, and the motion field information to be used may be signaled among a list of motion field candidate lists filled with motion field information of available neighboring / co-located blocks.

[0035] The video codec may support motion-compensated prediction from one source image (unidirectional prediction) and two sources (bidirectional prediction). In the case of unidirectional prediction, a single motion vector may be applied, and in the case of bidirectional prediction, two motion vectors may be signaled and the motion-compensated predictions from the two sources may be averaged to create the final sample prediction. In the case of weighted prediction, the relative weights of the two predictions may be adjusted or a signaled offset may be added to the prediction signal.

[0036] In addition to applying motion compensation for inter-picture prediction, a similar approach can be applied to intra-picture prediction, where a displacement vector indicates where blocks of samples can be copied from the same picture to form a prediction of the block to be coded or decoded. This kind of intra-block copying method can significantly improve coding efficiency when there are repeating structures such as text or other graphics in the frame.

[0037] In video codecs, the prediction residual after motion compensation or intra prediction may first be transformed with a transform kernel (such as the DCT "Discrete Cosine Transform") and then coded, since there is often still some correlation present in the residual, and the transform can often reduce this correlation and provide more efficient coding.

[0038] A video encoder may utilize a Lagrangian cost function to find an optimal coding mode, such as a desired macroblock mode and associated motion vectors. This type of cost function uses a weighting factor λ to combine the (exact or estimated) image distortion caused by a lossy coding method with the (exact or estimated) amount of information needed to represent pixel values ​​in an image region. C=D+λR (formula 1) where C is the Lagrangian cost to be minimized, D is the image distortion (e.g., mean squared error) for the mode and motion vector under consideration, and R is the number of bits required to represent the data needed to reconstruct the image block at the decoder (including the amount of data representing the candidate motion vectors).

[0039] Scalable video coding refers to a coding structure in which one bitstream can contain multiple representations of content at different bitrates, resolutions, or frame rates. In these cases, the receiver can extract the desired representation depending on its characteristics (e.g., the resolution that best fits the display device). Alternatively, a server or network element can extract the portion of the bitstream that is sent to the receiver depending on, for example, the network characteristics or the processing capabilities of the receiver. A scalable bitstream may consist of a "base layer" that provides the lowest quality video available, and one or more enhancement layers that improve the video quality when received and decoded together with the lower layer. To improve the coding efficiency of the enhancement layer, the coded representation of the enhancement layer may depend on the lower layer. For example, the motion and mode information of the enhancement layer can be predicted from the lower layer. Similarly, the pixel data of the lower layer can also be used to create the prediction of the enhancement layer.

[0040] A scalable video codec for quality scalability (also known as signal-to-noise ratio or SNR) and / or spatial scalability may be implemented as follows: A conventional non-scalable video encoder and decoder are used for the base layer. The reconstructed / decoded picture of the base layer is included in a reference picture buffer for the enhancement layer. In H.264 / AVC, HEVC, and similar codecs that use reference picture lists for inter-prediction, the decoded picture of the base layer may be inserted into a reference picture list for encoding / decoding the enhancement layer picture, as well as the decoded reference picture of the enhancement layer. As a result, the encoder may select the base layer reference picture as the inter-prediction reference and indicate its use with a reference picture index in the encoded bitstream. The decoder decodes from the bitstream, e.g., from a reference picture index, that the base layer picture is used as an inter-prediction reference for the enhancement layer. When a decoded base layer picture is used as a predictive reference for an enhancement layer, it is called an inter-layer reference picture.

[0041] In addition to quality scalability, the following scalability modes exist: - Spatial scalability: The base layer pictures are coded at a lower resolution than the enhancement layer pictures. - Bit depth scalability: Base layer pictures are coded at a lower bit depth (eg, 8 bits) than enhancement layer pictures (eg, 10 bits or 12 bits). - Chroma format scalability: The enhancement layer pictures provide higher chroma fidelity (eg, encoded in 4:4:4 chroma format) than the base layer pictures (eg, 4:2:0 format).

[0042] In the aforementioned scalability case, the additional bitrate overhead can be minimized by encoding the enhancement layer using base layer information.

[0043] Scalability can be achieved in two ways: a) by introducing new coding modes to perform prediction of pixel values ​​or syntax from lower layers of the scalable representation, or b) by putting pictures of lower layers into a reference picture buffer (decoded picture buffer, DPB) of the higher layer. Approach a) is more flexible and therefore can provide better coding efficiency in most cases. However, approach b), i.e. reference frame-based scalability, can be implemented very efficiently with minimal changes to a single layer codec while still realizing most of the coding efficiency gains obtained. Reference frame-based scalability codecs can be implemented using the same hardware or software implementation for all layers and only handling DPB management by external means.

[0044] To take advantage of parallel processing, an image can be divided into image segments (slices or tiles) that can be encoded and decoded independently. A slice may refer to an image segment consisting of a certain number of basic coding units that are processed in a default encoding or decoding order, and a tile may refer to an image segment defined as a rectangular image region that is processed at least to some extent as an individual frame.

[0045] Video may be encoded in YUV or YCbCr color spaces, which are believed to reflect the characteristics of the human visual system and allow the use of lower quality representations in the Cb and Cr channels because human perception is less sensitive to the fidelity of the chrominance that these channels represent.

[0046] In some video codecs, such as the VVC / H.266 video codec, various cross-component prediction approaches using linear models are used. Such approaches generate a linear model that can be used to map luma sample values ​​to chroma sample values. The parameters of the linear model are constructed using available reconstructed luma and chroma reference samples that are outside the boundaries of the prediction block. Once the parameters are constructed, the linear model prescribed by those parameters is used to predict the chroma sample values ​​in the prediction block.

[0047] In the case where there is a strong linear correlation between the luma and chroma channels, applying cross-component linear model prediction to predict chroma samples may be more efficient than intra-component spatial prediction, since the texture of the reconstructed luma block may produce a very good predictor of the chroma texture. The coding efficiency of such a model can be further improved by including additional parameters or coefficients in the model. However, this makes the process for determining the optimal values ​​of those parameters relatively complex and computationally intensive.

[0048] Cross-component linear model prediction is used, for example, in the VVC / H.266 video codec. In its variants, there are three chroma prediction modes that use cross-component linear model prediction. One of them can be selected by the encoder as the prediction mode for a chroma prediction block and signaled to the decoder in the bitstream. The difference between the three modes is the set of reference samples used to generate the parameters of the linear model: one of the modes uses only samples above the prediction block, one of the models uses only samples to the left of the prediction block, and one of the modes uses samples both above and to the left of the prediction block. To keep the complexity of the parameter generation low, the parameters are calculated using only a subset of the reference samples available at the block boundaries.

[0049] One possibility to calculate the parameters of such a linear model is to use linear regression, for example as shown in the JVET (Joint Video Experts Team) contribution document JVET-D0110. In this embodiment, there can be two linear models operating in different ranges of the luma spectrum. Another possibility is to derive the parameters of a conventional cross-component model using various types of linear regression tools, for example ordinary least squares estimation, orthogonal matching pursuit, optimized orthogonal matching pursuit, ridge regression, or least absolute shrinkage and selection operator.

[0050] An embodiment of the present invention relates to a solution in which a convolutional cross-component filter is used to generate an autocorrelation matrix and a cross-correlation vector from a set of reference data, such as luma and chroma sample values ​​in the vicinity of a sample block to be predicted. Different variants may have different (types of) cross-correlation vectors, but often only one of them is active at a time. The reference data may also be called "training data" because the filter coefficients of the filter are calculated or "trained" using that data. The reference data consists of two sets of input data and target data (or output data). The purpose of the generated filter is to provide an accurate mapping from the input data to the target data. In the case of a convolutional cross-component filter, the input data may be configured to include luma samples in the vicinity of the sample block, and the target data may be configured to include chroma data in the vicinity of the sample block. After scaling the autocorrelation matrix and the cross-correlation vector based on the bit depth of the video content, the autocorrelation matrix may be decomposed and a series of back substitution processes may be used to determine the system parameters. The determined system parameters represent the filter coefficients of a convolutional cross-component filter that can be applied to a chroma sample block to predict chroma values ​​from luma values. The system is configured to perform intermediate scaling between the two back substitutions using the scaling vectors generated during the matrix decomposition. Furthermore, embodiments of the present invention define ways to reduce the computational requirements of the process and the dimensionality of the system by various means, such as, for example, removing the bias term from the linear equation system and estimating it separately. With respect to embodiments of the present invention, the term "encoder" refers to both video encoders and still image encoders. Similarly, the term "decoder" refers to both video decoders and still image decoders.

[0051] For simplicity, various data structures that contain multiple scalar values ​​are represented herein in the form of matrices and vectors. Of course, they may be represented or embodied in other forms as well. For example, they may be represented or embodied as arrays, lists, collections of other scalar values, or sets of individual scalar values.

[0052] To perform a method according to an embodiment of the present invention, parameters of a filter, such as a convolutional cross-component filter, are calculated, which may be represented by a vector x and convolved with an input vector z to generate predicted sample values ​​to be used, for example, for video or image coding purposes. The input vector z may, for example, contain luma samples or downsampled luma samples in a predetermined position relative to the predicted chroma samples calculated as the output of the filtering operation. The vector x may consist of n parameters and may be expressed as follows: x=[x 0 ...x n-1 ] T

[0053] The input vector z can also consist of n input values ​​and can be expressed as: z=[z 0 ...z n-1 ] T

[0054] The predicted sample value p can then be calculated by convolving or multiplying the input z with the filter parameter vector x as follows:

number

[0055] The input vector z may be configured to include, for example, luminance sample values, functions of luminance sample values, or constants, or a combination thereof. Including a constant in the input vector z is equivalent to adding a constant to the output of the filter p. Such a constant may be called a bias term or bias parameter, and may be used to represent an offset between the input and output values.

[0056] The filter parameters x can generally be calculated by finding the solution to a set of equations that can be expressed in matrix form as follows: Ax=y where A represents the autocorrelation matrix of the determined input reference or "training" samples used in the process, and y represents the cross-correlation vector between the input training samples and the corresponding target training samples. The entries in the n×n matrix A and the vector y having n values ​​can be calculated, for example, as follows:

[0057]

number

[0058] Since the resulting matrix A is symmetric (i.e., A[c][d] is equal to A[d][c]), it is sufficient to compute either the upper or lower triangular part of the matrix, which can be done, for example, by starting a loop involving parameter d with the current value of parameter c as follows: A = zeroMatrixOfSize(n, n) for c = 0, …, n-1 for d = c, …, n-1 for i = 0, …, N-1 A[c][d] += R[i][c] * R[i][d]

[0059] To allow vector x to be calculated using fixed-point arithmetic, the values ​​of matrix A and vector y are scaled based on the bit depth of the video or image content, which can be done by including bitwise shift operations in the process of calculating matrix A and vector y, for example: A = zeroMatrixOfSize(n, n) for c = 0, …, n-1 for d = c, …, n-1 for i = 0, …, N-1 A[c][d] += R[i][c] * R[i][d] << shift y = zeroVectorOfSize(n) for c = 0, …, n-1 for i = 0, …, N-1 y[c] += R[i][c] * s[i] << shift Or it can be performed after the summation: A = zeroMatrixOfSize(n, n) for c = 0, …, n-1 for d = c, …, n-1 for i = 0, …, N-1 A[c][d] += R[i][c] * R[i][d] A[c][d] = A[c][d] << shift y = zeroVectorOfSize(n) for c = 0, …, n-1 for i = 0, …, N-1 y[c] += R[i][c] * s[i] y[c] = y[c] << shift Or it can be done the other way around, for example by generating the intermediate matrix A and vector y without shifting, then scanning them again and performing the shift during the second scan.

[0060] In addition to or instead of performing the shift based on the bit depth of the content, other aspects are also contemplated. For example, the values ​​of matrix A and vector y can be shifted based on the number of training vectors or samples included in the process that generates A and y, or an approximation of such a number. The shift can be signaled in the video bitstream or derived from a syntax element or parameter signaled in the video bitstream.

[0061] For example, the shift can be calculated as follows: shift=efg Here, e may represent a bit shift related to the maximum dynamic range allowed for the values ​​of matrix A and vector y. It may be set, for example, to 30, 28, 26, 24, or 16, or any other value known to be suitable for the purpose. The value f may represent a shift related to the bit depth of the video or image content being processed. Since both matrix A and vector y are typically generated by multiplying values ​​having similar or identical precision to the video or image content with other values ​​having the same characteristics, it is advantageous to use a value of twice the bit depth of the content as the value of f. Thus, for example, for a 10-bit video or image content, the value of f may be set to 20. The value g may represent a shift related to the number of training samples of the vectors included in the construction of matrix A and vector y. For example, the logarithm of 2 or a multiple of the logarithm of 2 of the number of training vectors or samples may be used as the value of g. Alternatively, the value of g may be set to a constant and combined with the value of e to simplify the determination of the shift value. If the determined shift value is positive, a bitwise left shift may be performed, and if negative, a bitwise right shift may be performed. Also, to further simplify the shift process, the shift can be specified to have only positive or negative values, and if determining the shift value would violate specified constraints, a value of zero can be used for the shift.

[0062] As a further example, the shift parameter can be calculated using any of the following notations: shift=B-2*bitDepth-log2(N), or shift=B-2*bitDepth where B is a constant known to both the video encoder and decoder, and bitDepth is a measure of the bit depth of the video content, which may be a function of the bit depth of the luma channel, the chroma channel, or a combination thereof. N in the above equation may represent the number of training or reference samples used in the matrix generation process, or an estimate thereof. The log2(N) operator may refer to the logarithm of 2 of the value N, and may include rounding up, rounding down, or rounding to the nearest integer value. Other selections may also be made. For example, instead of or in addition to the number of reference samples, the number of samples predicted by the filter may be selected as one of the bases for calculating the shift parameters.

[0063] Filter coefficients x of vector x in equation Ax=y i To solve for , an approach based on matrix decomposition is used. Different decompositions can be selected for the process. For example, LDL, Cholesky, or QR decompositions can be used. For example, the implementation described in the following pseudocode can be used, with matrix A as input and upper triangular matrix U and vector d as output: for i = 0, …, n-1 { d[i] = A[i][i] for k = i - 1, …, 0 { t = MULT(U[k][i], U[k][i]) d[i] -= MULT(t, d[k]); } if d[i] <= singularityThreshold { return false; } for j = i + 1, …, n { s = A[i][j] for k = i - 1, …, 0 { t = MULT(U[k][j], U[k][i]) s -= MULT(t, d[k]) } U[i][j] = DIV(s, d[i]) } }

[0064] By performing this decomposition, the upper triangular output matrix U corresponds to the transpose of the lower triangular matrix L of the LDL decomposition, and the output vector d contains the values ​​of the diagonal elements of the diagonal matrix D of the LDL decomposition. The values ​​of the output vector d (i.e., the scaling vector) may be called scaling values ​​because they are used to scale the values ​​of the triangular matrix and also to scale intermediate values ​​when solving the decomposed system. The values ​​of the scaling vector d may be represented, for example, as vectors, arrays, lists, or diagonal elements of a matrix. To save memory in software or hardware implementations, the values ​​of the vector d may be stored as diagonal elements of the output matrix U, whose values ​​are not specified in the implementation according to the pseudocode example above.

[0065] As another example, we can generate a lower triangular matrix L. As a result, the equation Ax=y Then, the filter coefficient vector x is LDL T x=y Then replace A by its LDL decomposition and perform the backsubstitution and scaling operations, as in, U T DUx=y This can be found by using the matrix U calculated in the pseudo code example as follows:

[0066] The filter coefficient vector x can then be found in three steps: In the first step, the terms of DUx can be expressed as an intermediate vector z, which can be found by backsubstitution. DUx=z U T z=y

[0067] In the second step, D can be eliminated by dividing or scaling the elements of the intermediate vector z by the elements of vector d.

number

[0068] In the third step, since the above equations are again in the form of an upper / lower triangular matrix multiplied by a vector x equal to another vector, back-substitution can be used to directly determine the vector x. Thus, the overall process of determining the filter coefficients can be structured to have three stages: a first back-substitution process, a scaling process, and a second back-substitution process. The scaling process between the two back-substitutions is advantageously performed using the vector generated as the product of the matrix decomposition.

[0069] The multiplication operation MULT and the division operation DIV can be implemented in various ways. For example, floating-point or fixed-point implementations can be used. In general, it may be beneficial to use fixed-point arithmetic, since a fixed-point implementation may provide faster execution in some computing architectures. To advantageously balance the numerical stability of the computing process with the precision of the fixed-point representation, the MULT operation can be performed by rounding to the nearest integer, and the DIV operation can be performed by rounding to zero. For example, these functions can be defined in pseudocode as follows: MULT(x, y) { return ( x * y + (1 << ( DECIM_BITS - 1) ) >> DECIM_BITS } DIV(x, y) { return ( x << DECIM_BITS ) / y}

[0070] Alternatively, the DIV operation may be implemented as a combination of, for example, a table lookup operation and a bitwise shift operation. Rounding of terms may also be included, as in the MULT function in the example above. The DECIM_BITS parameter determines the number of decimal bits in the fixed-point representation and may be set to various values ​​depending on the desired precision of the operation.

[0071] Backsubstitution can be performed in a variety of ways. For example, the following pseudocode can be used: z[0] = y[0]; for i = 1, …, numEq - 1 { sum = 0 for j = 0, …, i-1 { sum += MULT(z[j], U[j][i]) } z[i] = y[i] - sum }

[0072] The dependencies between constructing the matrix decomposition and finding the filter coefficients from the system of equations can be jointly optimized. For example, the division operations in the matrix decomposition and between or during the backsubstitution process have the same denominator d i (corresponding to the diagonal elements of matrix D or the elements of diagonal vector d in the above example, and the d[i] term in the pseudocode example). In these cases, d i Instead of dividing by , we change the operation to the reciprocal of that number, 1 / d i It may be advantageous to convert this to a multiplication by x. In this way, the number of division operations is reduced to one per output coefficient, since only one division operation is required to compute the reciprocal of each diagonal element.

[0073] Determining the filter coefficients is advantageously performed by two back-substitution operations and a scaling operation between them, the scaling operation scaling the output value of the first back-substitution operation using the value generated as the output of the matrix decomposition operation, the scaling can include, for example, a multiplication, division or bit-shift operation, or a combination thereof.

[0074] In some cases, the autocorrelation matrix A may be singular. The values ​​of the diagonal vector d can be used as a measure of singularity. In the above pseudocode example, each element of d (d[i]) is compared to a threshold, singularityThreshold, to determine singularity. The value of singularityThreshold can be determined in various ways. For example, it can be set to a fixed value, such as 0 or other integer or floating-point number. It can also be calculated based on the bit depth of the video content or the characteristics of matrix A.

[0075] If A is determined to be singular, various measures can be taken. For example, a regularization term can be added to the diagonal elements of matrix A, and the decomposition can be attempted again. However, this type of approach can nearly double the worst-case complexity of the decomposition operation, since the decomposition is performed at least partially twice. Matrix A is advantageously regularized by adding a positive term to each of its diagonal elements before attempting to decompose the matrix. The value of the positive term can be determined separately. For example, a constant value of 1 or other constant can be used as the regularization term. The value of the term can also be determined based on the bit depth of the video content or characteristics of matrix A. If matrix A is still found to be singular during the first decomposition attempt, it is advantageous to set the filter coefficient vector x to a vector independent of matrix A, and further attempts to decompose matrix A can be omitted. This approach limits the number of matrix decomposition attempts to one, thereby nearly halving the worst-case complexity of the approach to regularizing matrix A after a failed decomposition. Various approaches can be used to determine the filter coefficient vector x independently of matrix A. For example, all elements of x can be set to zero, or the elements associated with the bias term of the filter can be set to a value derived from the bit depth of the video or from the average luma and chroma values ​​of the reference samples, and the remaining elements can be set to zero.

[0076] Filter coefficient x i There are various ways to reduce the computational complexity associated with computing z. For example, when using a convolution filter to predict chroma values ​​based on input luma values, it may be desirable to include a so-called bias term in the filtering process. A bias term can be thought of as a constant that is added to the filter output. One way to implement such a bias is to reduce the computational complexity associated with computing z b The trick is to make x a constant (e.g., a value corresponding to an intermediate luma value, e.g., the value 512 for 10-bit video content, or the value 128 for 8-bit video content). Such a choice results in the corresponding filter coefficient xb determines the amount of bias added to the filter output. The bias term x b can be calculated as above as one of the filter coefficients of vector x. However, it can also be estimated in other ways. For example, if the input to the filter is a luma value and we output a predicted luma, and the filter is generated using a set of reference luma and chroma samples, then the average reference luma value y mean and the average reference chroma sample value c mean When calculating the filter coefficients, y can be calculated either before or when generating the autocorrelation matrix A and the cross-correlation vector y. mean can be subtracted from the reference luma value, c mean can be calculated from the reference chroma samples. Similarly, when performing the convolution operation to calculate the filter output, y can be calculated from the input luma samples. mean Then, the bias between the luma and chroma samples can be calculated by subtracting the average reference chroma value c mean can be restored by adding to the output of the filter.

[0077]

number

[0078] The encoder or decoder may be configured to compute a subset of the filter coefficients from the decomposed autocorrelation and cross-correlation vectors, which may be advantageously done by performing a backsubstitution operation only on those elements from the system of equations that correspond to the set of coefficients of interest.

[0079] In one embodiment, the set of filter coefficients is determined using a first backsubstitution process, a scaling process, and a second backsubstitution process.

[0080] In one embodiment, the set of filter coefficients is determined based on an autocorrelation matrix decomposed into at least a triangular matrix and a set of scaling values.

[0081] In one embodiment, the autocorrelation matrix is ​​decomposed into at least a triangular matrix and a set of scaling values, where the scaling values ​​are represented as vectors, arrays, lists, or diagonal elements of the matrix.

[0082] In one embodiment, the set of filter coefficients is determined using a first backsubstitution, scaling, and a second backsubstitution, where the scaling is performed using values ​​generated by decomposing the matrix.

[0083] In one embodiment, the set of filter coefficients is determined using a first backsubstitution, scaling, and a second backsubstitution, where the scaling is performed using values ​​generated by decomposing the autocorrelation matrix.

[0084] In one embodiment, the autocorrelation matrix is ​​decomposed into at least a triangular matrix and a set of scaling values ​​using floating point arithmetic.

[0085] In one embodiment, the autocorrelation matrix is ​​decomposed into at least a triangular matrix and a set of scaling values ​​using fixed-point arithmetic.

[0086] In one embodiment, the autocorrelation matrix is ​​decomposed into at least a triangular matrix and a set of scaling values ​​using fixed-point arithmetic, where the fixed-point multiplication is configured to include rounding to the nearest fixed-point value and the fixed-point division operation is configured to include rounding towards zero.

[0087] In one embodiment, the determined set of scaling factors is applied between the two backsubstitution processes.

[0088] In one embodiment, the elements of the autocorrelation matrix and the cross-correlation vector are shifted using a bit-shifting operation where the amount of bit-shifting is determined based on the bit-depth of the video content.

[0089] In one embodiment, the elements of the autocorrelation matrix and the cross-correlation vector are shifted using a bit-shifting operation in which the amount of bit-shifting is determined based on the number of reference samples or a measure based on the number of reference samples.

[0090] In one embodiment, the elements of the autocorrelation matrix and the cross-correlation vector are shifted using a bit-shifting operation in which the amount of bit-shifting is determined based on the number of samples to be predicted or a measure based on the number of samples to be predicted.

[0091] In one embodiment, the elements of the autocorrelation matrix and the cross-correlation vector are shifted using a bit-shifting operation in which the bit-shift amount is determined based on the bit-depth of the video content and the number of reference samples used to generate the autocorrelation matrix and the cross-correlation vector.

[0092] In one embodiment, the singularity of the autocorrelation matrix is ​​determined during the decomposition by comparing the output scaling values ​​to an integer-valued threshold.

[0093] In one embodiment, if the autocorrelation matrix is ​​determined to be singular, the filter coefficients are set to zero.

[0094] In one embodiment, if the autocorrelation matrix is ​​determined to be singular, all filter coefficients except for the bias term are set to zero.

[0095] In one embodiment, if the autocorrelation matrix is ​​determined to be singular, the bias term of the filter coefficients is set to a value that represents the average of the reference chroma values.

[0096] In one embodiment, if the autocorrelation matrix is ​​determined to be singular, the bias term of the filter coefficients is set to a value representing 512 for 10-bit video and is set to a value of 128 for 8-bit video.

[0097] In one embodiment, positive values ​​are added to the diagonal elements of the autocorrelation matrix before the decomposition.

[0098] In one embodiment, when determining the autocorrelation matrix and the cross-correlation vector, the average luma value is subtracted from the luma value and the average chroma value is subtracted from the chroma value, and the average luma value and the average chroma value are calculated using a set of reference samples.

[0099] In one embodiment, before applying the convolution filtering operation, an average luma value is subtracted from the input luma value and an average chroma value is added to the filter output, and the average luma value and average chroma value are calculated using a set of reference samples.

[0100] In one embodiment, a backsubstitution process is applied to a selected subset of elements in the triangular matrix and the target vector.

[0101] In one embodiment, the determined set of filter coefficients is used to predict chroma values ​​from luma values.

[0102] A method according to one embodiment is shown in Figure 3. The method generally includes obtaining a set of input samples 310, utilizing the set of input samples to obtain a triangular matrix, a cross-correlation vector and a scaling vector 320, determining an intermediate vector by a first back-substitution using the triangular matrix and the cross-correlation vector 330, scaling the intermediate vector by a second back-substitution using the scaling vector 340, determining a filter coefficient vector by a second back-substitution using the triangular matrix and the scaled intermediate vector 350, and determining a predicted sample value of a block of samples based on the filter coefficient vector 360. Each step can be implemented by a respective module of a computer system.

[0103] An apparatus according to an embodiment includes a means for obtaining a set of input samples, a means for obtaining a triangular matrix, a cross-correlation vector and a scaling vector using the set of input samples, a means for determining an intermediate vector by a first back-substitution using the triangular matrix and the cross-correlation vector, a means for scaling the intermediate vector by a second back-substitution using the triangular matrix and the scaled intermediate vector, and a means for determining a predicted sample value of a block of samples based on the filter coefficient vector. The means includes at least one processor and a memory including computer program code, and the processor may further include a processor circuit. The memory and the computer program code are configured by the at least one processor to cause the apparatus to perform the method of FIG. 3 according to various embodiments.

[0104] The various embodiments may be implemented with the aid of computer program code residing in a memory and causing an associated device to perform the method. For example, a device may include circuitry and electronics for processing, receiving, and transmitting data, computer program code in a memory, and a processor that, when executing the computer program code, causes the device to perform features of an embodiment. Still further, a network device, such as a server, may include circuitry and electronics for processing, receiving, and transmitting data, computer program code in a memory, and a processor that, when executing the computer program code, causes the network device to perform features of the various embodiments.

[0105] If desired, the various functions discussed herein may be performed in different orders and / or concurrently with one another. Further, if desired, one or more of the functions and embodiments described above may be optional or combined.

[0106] Although various aspects of the embodiments are set out in the independent claims, other aspects include other combinations of features of the described embodiments and / or dependent claims with features of the independent claims, and not only the combinations explicitly set out in the claims.

[0107] It should also be noted that although exemplary embodiments have been described herein above, these descriptions should not be construed in a limiting sense, rather, there are several variations and modifications that can be made without departing from the scope of the present disclosure as defined in the appended claims.

Claims

1. means for obtaining a set of input samples; means for obtaining a triangular matrix, a cross-correlation vector, and a scaling vector utilizing the set of input samples; means for determining an intermediate vector by a first backsubstitution using the triangular matrix and the cross-correlation vector; means for scaling the intermediate vector using the scaling vector; means for determining a filter coefficient vector by a second back-substitution using the triangular matrix and the scaled intermediate vector; means for determining predicted sample values ​​for a block of samples based on said filter coefficient vector; An apparatus comprising:

2. 2. The apparatus of claim 1, further comprising: means for determining the triangular matrix and the scaling vector by decomposing an autocorrelation matrix into at least the triangular matrix and the scaling vector, the autocorrelation matrix being calculated using a set of reference samples in a defined neighborhood of the block of samples.

3. The apparatus of claim 2 , further comprising means for subtracting an average luma value from a set of luma values ​​and subtracting an average chroma value from a set of chroma values ​​to determine the autocorrelation matrix and the cross-correlation vector.

4. The apparatus of claim 1 , 2 or 3 further comprising means for scaling the autocorrelation matrix and the cross-correlation vector based on a bit depth of the block of samples.

5. The apparatus of claim 4 , further comprising: means for including a bitwise shift operation in the process of computing the autocorrelation matrix and the cross-correlation vector.

6. The apparatus of any preceding claim, wherein the set of samples comprises one or more of the following: luminance sample values, functions of luminance sample values, and bias parameters.

7. Apparatus according to any one of claims 1 to 6, further comprising means for encoding the set of samples and / or means for decoding the encoded set of samples.

8. obtaining a set of input samples; obtaining a triangular matrix, a cross-correlation vector, and a scaling vector utilizing the set of input samples; determining an intermediate vector by a first backsubstitution using the triangular matrix and the cross-correlation vector; scaling the intermediate vector using the scaling vector; determining a filter coefficient vector by a second back-substitution using the triangular matrix and the scaled intermediate vector; determining predicted sample values ​​for a block of samples based on the filter coefficient vector; A method comprising:

9. 9. The method of claim 8, further comprising determining the triangular matrix and the scaling vector by decomposing an autocorrelation matrix into at least the triangular matrix and the scaling vector, the autocorrelation matrix being calculated using a set of reference samples in a defined neighborhood of the block of samples.

10. 10. The method of claim 9, further comprising: subtracting an average luma value from a set of luma values ​​and subtracting an average chroma value from a set of chroma values ​​to determine the autocorrelation matrix and the cross-correlation vector.

11. The method of claim 8, 9 or 10, further comprising scaling the autocorrelation matrix and the cross-correlation vector based on a bit depth of the block of samples.

12. The method of claim 11 , further comprising including a bitwise shift operation in the process of calculating the autocorrelation matrix and the cross-correlation vector.

13. The method of any one of claims 8 to 12, wherein the set of samples comprises one or more of the following: luminance sample values, functions of luminance sample values, and bias parameters.

14. The method of any one of claims 8 to 13, further comprising encoding the set of samples and / or decoding the encoded set of samples.

15. 1. An apparatus comprising at least one processor and a memory containing computer program code, the memory and the computer program code being transmitted by the at least one processor to the apparatus to at least: obtaining a set of input samples; obtaining a triangular matrix, a cross-correlation vector, and a scaling vector utilizing the set of input samples; determining an intermediate vector by a first backsubstitution using the triangular matrix and the cross-correlation vector; scaling the intermediate vector using the scaling vector; determining a filter coefficient vector by a second back-substitution using the triangular matrix and the scaled intermediate vector; determining predicted sample values ​​for a block of samples based on the filter coefficient vector; An apparatus configured to cause the

Images