Method, apparatus and computer program product for encoding and decoding digital media content

Abstract

Embodiments relate to a method for encoding / decoding, the method comprising obtaining (310) a set of input samples; obtaining (320) a triangular matrix, a cross-correlation vector, and a scaling vector by utilizing the input sample set; determining (330) an intermediate vector with a first back substitution using the triangular matrix and the cross-correlation vector; scaling (340) the intermediate vector using the scaling vector; determining (350) a filter coefficient vector with a second back substitution using a triangular matrix and the scaled intermediate vector; and determining (360) a predicted sample value for the sample block based on the filter coefficient vector. Embodiments also relate to an apparatus and a computer program product for implementing the method.

Description

Technical Field

[0001] The present solution generally relates to encoding and decoding of digital media content, such as video or still image data. Background Art

[0002] This section is intended to provide a background or context to the invention described in the claims. The descriptions herein may include concepts that may be pursued, but not necessarily concepts previously conceived or pursued. Therefore, unless otherwise indicated herein, what is described in this section is not prior art to the description and claims in this application and should not be considered prior art by inclusion in this section.

[0003] A video coding system may include an encoder that transforms an input video into a compressed representation suitable for storage / transmission, and a decoder that can decompress the compressed video representation back into a visual form. The encoder may discard some information in the original video sequence in order to represent the video in a more compact form, for example, to enable the video information to be stored / transmitted at a lower bit rate than might otherwise be required. Summary of the invention

[0004] The independent claims set out the scope of protection sought by various embodiments of the invention. Embodiments and features described in this specification that do not fall within the scope of the independent claims, if any, are to be construed as examples that aid in understanding the various embodiments of the invention.

[0005] Various aspects include a method, an apparatus, and a computer readable medium comprising a computer program stored therein, which are characterized by what is stated in the independent claims. Various embodiments are disclosed in the dependent claims.

[0006] According to a first aspect, an apparatus is provided, the apparatus comprising means for obtaining an input sample set; means for obtaining a triangular matrix, a cross-correlation vector, and a scaling vector by utilizing the input sample set; means for determining an intermediate vector using a first back iteration using the triangular matrix and the cross-correlation vector; means for scaling the intermediate vector using the scaling vector; means for determining a filter coefficient vector using a second back iteration using the triangular matrix and the scaled intermediate vector; and means for determining a predicted sample value for a sample block based on the filter coefficient vector.

[0007] According to a second aspect, a method is provided, the method comprising: obtaining an input sample set; obtaining a triangular matrix, a cross-correlation vector, and a scaling vector by utilizing the input sample set; determining an intermediate vector using a first back iteration using the triangular matrix and the cross-correlation vector; scaling the intermediate vector using the scaling vector; determining a filter coefficient vector using a second back iteration using the triangular matrix and the scaled intermediate vector; and determining a predicted sample value for a sample block based on the filter coefficient vector.

[0008] According to a third aspect, a device is provided, the device comprising at least one processor, a memory comprising computer program code, the memory and the computer program code being configured to, together with the at least one processor, cause the device to at least perform the following operations: obtain an input sample set; obtain a triangular matrix, a cross-correlation vector, and a scaling vector by utilizing the input sample set; determine an intermediate vector using a first back iteration using the triangular matrix and the cross-correlation vector; scale the intermediate vector using the scaling vector; determine a filter coefficient vector using a second back iteration using the triangular matrix and the scaled intermediate vector; and determine a predicted sample value for a sample block based on the filter coefficient vector.

[0009] According to a fourth aspect, a computer program product is provided, the computer program product comprising computer program code, which is configured to, when executed on at least one processor, cause an apparatus or system to: obtain an input sample set; obtain a triangular matrix, a cross-correlation vector, and a scaling vector by utilizing the input sample set; determine an intermediate vector using a first back iteration using the triangular matrix and the cross-correlation vector; scale the intermediate vector using the scaling vector; determine a filter coefficient vector using a second back iteration using the triangular matrix and the scaled intermediate vector; and determine a predicted sample value for a sample block based on the filter coefficient vector.

[0010] According to one embodiment, the triangular matrix and the scaling vector are determined by decomposing an autocorrelation matrix into at least a triangular matrix and a scaling vector, wherein the autocorrelation matrix is ​​calculated using a set of reference samples in a determined neighborhood of the block of samples.

[0011] According to one embodiment, the autocorrelation matrix and the cross-correlation vector are determined by subtracting the average luma value from the set of luma values, and subtracting the average chroma value from the set of chroma values.

[0012] According to one embodiment, the autocorrelation matrix and the cross-correlation vector are scaled based on the bit depth of the sample set.

[0013] According to one embodiment, a bit-wise shift operation is included in the process of calculating the autocorrelation matrix and the cross-correlation vector.

[0014] According to one embodiment, the sample set includes one or more of the following: luma sample values; functions of luma sample values; bias parameters.

[0015] According to an embodiment, the apparatus comprises means for encoding a set of samples and / or decoding an encoded set of samples.

[0016] According to one embodiment, a computer program product is embodied on a non-transitory computer readable medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Hereinafter, various embodiments will be described in more detail with reference to the accompanying drawings, in which

[0018] Figure 1 An example of the encoding process is shown;

[0019] Figure 2 An example of the decoding process is shown;

[0020] Figure 3 is a flow chart illustrating a method according to an embodiment; and

[0021] Figure 4 An apparatus according to an embodiment is shown. DETAILED DESCRIPTION

[0022] In the following, several embodiments will be described in the context of a video coding arrangement. However, it should be noted that the present embodiments are not necessarily limited to this particular arrangement. The embodiments discussed in this specification relate to intra prediction in video or still image coding using sparse linear cross component regression.

[0023] The following description and drawings are illustrative and should not be construed as unnecessary limitations. Specific details are provided for a thorough understanding of the present disclosure. However, in some cases, in order to avoid obscuring the description, well-known or conventional details are not described. In the present disclosure, references to an embodiment or embodiments may be, but are not necessarily, references to the same embodiment, and such references represent at least one embodiment.

[0024] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment of the invention is included in at least one embodiment of the present disclosure.

[0025] A video codec includes an encoder and a decoder. The encoder is configured to transform the input video into a compressed representation suitable for storage / transmission. The decoder is able to decompress the compressed video representation back into a visual form. The encoder may discard some information in the original video sequence in order to represent the video in a more compact form, such as at a lower bit rate.

[0026] In most cases, the basic unit for the input of the encoder and the output of the decoder, respectively, is a picture. A picture as input to the encoder may also be referred to as a source picture, and a picture decoded by the decoder may be referred to as a decoded picture or a reconstructed picture.

[0027] Both the source picture and the decoded picture consist of one or more sample arrays, such as one of the following sets of sample arrays:

[0028] - Luminance (Y) only (monochrome);

[0029] - Luma and dual chroma (YCbCr or YCgCo);

[0030] - Green, Blue and Red (GBR, also known as RGB);

[0031] - arrays representing other unspecified monochromatic or tristimulus color sampling (e.g., YZX,

[0032] Also known as XYZ).

[0033] A picture can be defined as a frame or a field. A frame consists of a matrix of luma samples and possibly corresponding chroma samples. A field is a collection of alternating sample rows of a frame, and can be used as encoder input when the source signal is interlaced. The chroma sample array may not be present (and therefore monochrome sampling may be used), or the chroma sample array may be subsampled when compared to the luma sample array.

[0034] A bitstream can be defined as a sequence of bits, which in some coding formats or standards may be in the form of a network abstraction layer (NAL) unit stream or a byte stream, which forms a representation of coded pictures and associated data, which form one or more coded video sequences. The first bitstream may be followed by a second bitstream in the same logical channel, such as in the same file or the same connection of a communication protocol. An elementary stream (in the context of video coding) may be defined as a sequence of one or more bitstreams. In some coding formats or standards, the end of the first bitstream may be indicated by a specific NAL unit, which may be called an end-of-bitstream (EOB) NAL unit and is the last NAL unit of the bitstream.

[0035] The phrase "along the bitstream" (e.g., indicating along the bitstream) or along the coding units of the bitstream (e.g., indicating along the coding blocks) may be used in the claims and described embodiments to refer to transmission, signaling, or storage in a manner that "out-of-band" data is associated with the bitstream or coding units, respectively, but is not included therein. The phrases decoding along the bitstream or along the coding units of the bitstream, etc. may refer to decoding the referenced out-of-band data associated with the bitstream or coding units, respectively, which may be obtained from the out-of-band transmission, signaling, or storage. For example, the phrase along the bitstream may be used when the bitstream is contained in a container file (such as a file conforming to the ISO base media file format) and certain file metadata is stored in the file in a manner that associates the metadata with the bitstream, such as a box in a sample entry of a track containing the bitstream, a sample group of a track containing the bitstream, or a timed metadata track associated with a track containing the bitstream.

[0036] Hybrid video codecs (e.g., ITU-T H.263 and H.264) can encode video information in two stages. First, the pixel values ​​in a certain picture area (or "block") are predicted, for example, by motion compensation means (finding and indicating the area in one of the previously encoded video frames that closely corresponds to the block being encoded) or by spatial means (using the pixel values ​​around the block to be encoded in a specified manner). In the first stage, predictive coding can be applied, for example, as so-called sample prediction and / or so-called syntax prediction. In sample prediction, pixels of sample values ​​in a certain picture area or "block" are predicted. For example, these pixels or sample values ​​can be predicted using one or more of motion compensation or intra-frame prediction mechanisms. Second, the prediction error (i.e., the difference between the predicted pixel block and the original pixel block) is encoded. This can be achieved by transforming the difference in pixel values ​​into a specified transform (e.g., discrete cosine transform (DCT) or its variant), quantizing coefficients, and entropy encoding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and the size of the final encoded video representation (file size or transmission bit rate).

[0037] An example of the encoding process is Figure 1 shown. Figure 1 The image to be encoded (I n ); The predicted representation of the image block (P' n ); prediction error signal (D n ); reconstructed prediction error signal (D' n ); Preliminary reconstructed image (I' n ); Final reconstructed image (R' n ); Transform (T) and Inverse Transform (T -1 ), quantization (Q) and inverse quantization (Q-1 ); Entropy coding (E); Reference frame memory (RFM); Inter-frame prediction (P inter ); intra prediction (P intra ); mode selection (MS) and filtering (F).

[0038] In some video codecs such as H.265 / HEVC, a video picture is divided into coding units (CUs) covering an area of ​​the picture. A CU includes one or more prediction units (PUs) defining a prediction process for samples within the CU and one or more transform units (TUs) defining a prediction error coding process for samples in the above CU. A CU may include a square block of samples, the size of which may be selected from a predefined set of possible CU sizes. A CU with the maximum allowed size may be named an LCU (largest coding unit) or a CTU (coding tree unit), and a video picture may be divided into non-overlapping CTUs. A CTU may be further split into a combination of smaller CUs, for example, by recursively splitting a CTU and the resulting CUs. Each resulting CU may have at least one PU and at least one TU associated with it. Each PU and TU may be further split into smaller PUs and TUs to increase the granularity of the prediction and prediction error coding processes, respectively. Each PU has prediction information associated with it to define which prediction is applied to the pixels within the PU (e.g., motion vector information for an inter-prediction PU and intra-prediction directionality information for an intra-prediction PU). Similarly, each TU is associated with information describing the prediction error decoding process for the samples within the TU (including, for example, DCT coefficient information). Whether or not prediction error coding is applied to each CU can be signaled at the CU level. If no prediction error residual is associated with a CU, it can be considered that there is no TU for the CU. The division of an image into CUs and the division of a CU into PUs and TUs can be signaled in the bitstream to enable a decoder to reproduce the expected structure of these units.

[0039] The decoder can reconstruct the output video by applying a prediction component similar to the encoder to form a predicted representation of the pixel block (using the motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (the inverse operation of prediction error encoding, recovering the quantized prediction error signal in the spatial pixel domain). After applying the prediction and prediction error decoding components, the decoder is configured to add the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) may also apply additional filtering components to improve the quality of the output video before passing the output video for display and / or storing the output video as a prediction reference for upcoming frames in the video sequence. An example of the decoding process is as follows: Figure 2 shown. Figure 2 The predicted representation of the image block (P' n); reconstructed prediction error signal (D' n ); Preliminary reconstructed image (I' n ); Final reconstructed image (R' n ); inverse transform (T -1 ); inverse quantization (Q -1 ); Entropy decoding (E -1 ); reference frame memory (RFM); prediction (inter or intra) (P); and filtering (F).

[0040] As an alternative or supplement to the method of indicating the coded sample value using sample value prediction and transform coding, palette-based coding can also be used. Palette-based coding refers to a series of methods in which a palette, i.e., a set of colors and related indexes, is defined, and the value of each sample in the coding unit is represented by indicating its index in the palette. Palette-based coding can achieve good coding efficiency in coding units with a relatively small number of colors (such as image areas representing computer screen content, such as text or simple graphics). In order to improve the coding efficiency of palette coding, different types of palette index prediction methods can be used, or the palette index can be run-length encoded so that larger uniform areas can be efficiently represented. In addition, if the CU contains sample values ​​that are not repeated within the CU, escape coding can be used. The transmission of escape-coded samples does not refer to any palette index. Instead, their values ​​can be indicated individually for each escape-coded sample.

[0041] When a CU is encoded in palette mode, various prediction strategies are used to exploit the correlation between pixels within the CU. For example, mode information can be signaled for each row or pixel to indicate one of the following: the mode can be horizontal mode, which means that a single palette index is signaled and the entire pixel row shares the index; the mode can be vertical mode, in which the entire pixel row is the same as the above row, and no additional information is signaled; the mode can be normal mode, in which a flag is signaled for each pixel position to indicate whether it is the same as one of the left and other pixels, and if not, the color index itself is sent separately.

[0042] In a video codec, motion information can be indicated by a motion vector associated with each motion compensated image block. Each of these motion vectors can represent the displacement of an image block in a picture to be encoded (on the encoder side) or decoded (on the decoder side), and the displacement of a prediction source block in one of the previously encoded or decoded pictures. In order to efficiently represent motion vectors, these motion vectors can be differentially encoded relative to block-specific prediction motion vectors. In a video codec, prediction motion vectors can be created in a predefined manner, such as calculating decoded motion vectors of media or adjacent blocks of an encoder. Another way to create motion vector predictions is to generate a candidate prediction list from adjacent blocks and / or collocated blocks in a temporal reference picture, and signal the selected candidate as a motion vector predictor. In addition to predicting motion vector values, reference indexes of previously encoded / decoded pictures can also be predicted. Reference indexes can be predicted based on adjacent blocks and / or collocated blocks in a temporal reference picture. In addition, the high efficiency video codec may employ an additional motion information encoding / decoding mechanism, commonly referred to as a merging / merging mode, in which all motion field information, including motion vectors and corresponding reference picture indices for each available reference picture list, is predicted and used without any modification / correction. Similarly, motion field information prediction may be performed using motion field information of neighboring blocks and / or collocated blocks in a temporal reference picture, and the used motion field information may be signaled in a list of motion field candidate lists populated with motion field information of available neighboring / collocated blocks.

[0043] Video codecs may support motion compensated prediction from one source image (uni-prediction) and from two sources (bi-prediction). In the case of uni-prediction, a single motion vector may be applied, while in the case of bi-prediction, two motion vectors may be signaled and the motion compensated predictions from the two sources may be averaged to create the final sample prediction. In the case of weighted prediction, the relative weights of the two predictions may be adjusted or a signaled offset may be added to the prediction signal.

[0044] In addition to applying motion compensation to inter-picture prediction, a similar approach can also be applied to intra-picture prediction. In this case, the displacement vector indicates where a block of samples can be copied from the same picture to form a prediction of the block to be encoded or decoded. This intra-block copying approach can significantly improve coding efficiency in the presence of repetitive structures within a frame, such as text or other graphics.

[0045] In a video codec, the prediction residuals after motion compensation or intra prediction can be first transformed with a transform kernel (such as DCT "Discrete Cosine Transform") and then encoded. The reason for this is that there is usually still some correlation between the residuals, and in many cases, the transform can help reduce this correlation and provide more efficient coding.

[0046] Video encoders can use Lagrangian cost functions to find the best coding mode, e.g., the desired macroblock mode and the associated motion vectors. This cost function uses a weighting factor λ to relate the (exact or estimated) image distortion caused by the lossy coding method to the (exact or estimated) amount of information needed to represent the pixel values ​​in the image region:

[0047] C = D + λR (Equation 1) where C is the Lagrangian cost to be minimized, D is the image distortion (e.g., mean square error) taking into account mode and motion vectors, and R is the number of bits required to represent the data needed to reconstruct the image block in the decoder (including the amount of data used to represent the candidate motion vectors).

[0048] Scalable video coding refers to a coding structure in which one bitstream can contain multiple representations of content at different bitrates, resolutions, or frame rates. In these cases, the receiver can extract the desired representation based on its characteristics (e.g., the resolution that best matches the display device). Alternatively, a server or network element can extract parts of the bitstream to be sent to the receiver based on, for example, the network characteristics or processing capabilities of the receiver. A scalable bitstream can include a "base layer" that provides the lowest quality video available and one or more enhancement layers that can improve the video quality when received and decoded together with the lower layers. In order to improve the coding efficiency of the enhancement layer, the coded representation of the layer can depend on the lower layers. For example, motion and pattern information of the enhancement layer can be predicted from the lower layers. Similarly, pixel data of the lower layers can be used to create predictions of the enhancement layer.

[0049] A scalable video codec for quality scalability (also known as signal-to-noise ratio or SNR) and / or spatial scalability can be implemented as follows. For the base layer, a conventional non-scalable video encoder and decoder are used. The reconstructed / decoded pictures of the base layer are included in the reference picture buffer for the enhancement layer. In H.264 / AVC, HEVC, and similar codecs that use (multiple) reference picture lists for inter-frame prediction, the base layer decoded pictures can be inserted into (multiple) reference picture lists for encoding / decoding the enhancement layer pictures similarly to the decoded reference pictures of the enhancement layer. Therefore, the encoder can select the base layer reference picture as an inter-frame prediction reference and can indicate its use in the encoded bitstream with a reference picture index. The decoder decodes the base layer picture from the bitstream (e.g., from the reference picture index) to be used as an inter-frame prediction reference for the enhancement layer. When the decoded base layer picture is used as a prediction reference for the enhancement layer, it is called an inter-layer reference picture.

[0050] In addition to quality scalability, the following scalability modes exist:

[0051] - Spatial scalability: Base layer pictures are coded at a lower resolution than enhancement layer pictures.

[0052] - Bit depth scalability: The coding bit depth of base layer pictures is lower (e.g., 8 bits) than that of enhancement layer pictures (e.g., 10 or 12 bits).

[0053] - Chroma format scalability: Enhancement layer pictures provide higher chroma fidelity (eg, encoded in 4:4:4 chroma format) than base layer pictures (eg, 4:2:0 format).

[0054] In the above scalability case, the base layer information can be used to encode the enhancement layer to minimize the additional bit rate overhead.

[0055] Scalability can be achieved in two ways: a) by introducing new coding modes for performing prediction of pixel values ​​or syntax from the lower layers of the scalable representation; or b) by placing lower layer pictures into the reference picture buffer (decoded picture buffer DPB) of the higher layer. Method a) is more flexible and can therefore provide better coding efficiency in most cases. However, method b), i.e. reference frame based scalability, can be implemented very efficiently with minimal changes to the single layer codec while still achieving most of the available coding efficiency gains. Reference frame based scalability codecs can be implemented using the same hardware or software implementation for all layers, with only DPB management required by external means.

[0056] In order to be able to take advantage of parallel processing, images can be split into independently encodable and decodable image segments (slices or tiles). A slice may refer to an image segment consisting of a certain number of basic coding units that are processed in a default encoding or decoding order, while a tile may refer to an image segment defined as a rectangular image area that is processed, at least to some extent, as an individual frame.

[0057] Video can be encoded in YUV or YCbCr color space, which reflects some characteristics of the human visual system and allows the use of lower quality Cb and Cr channel representations because human perception is less sensitive to the chrominance fidelity of these channel representations.

[0058] In some video codecs (such as the VVC / H.266 video codec), a different cross-component prediction method using a linear model has been used. This method generates a linear model that can be used to map luma sample values ​​to chroma sample values. The parameters of the linear model are constructed using the available reconstructed luma and chroma reference samples outside the prediction block boundaries. After the parameters are constructed, the linear model specified by these parameters is used to predict the chroma sample values ​​within the prediction block.

[0059] Applying cross-component linear model prediction to predict chroma samples can be efficient compared to intra-component spatial prediction, because the texture of the reconstructed luma block can sometimes make a very good predictor for the chroma texture if there is a strong linear correlation between the luma and chroma channels. The coding efficiency of such a model can be further improved by adding additional parameters or coefficients to the model. However, this results in a relatively complex and computationally demanding process for determining the optimized values ​​of these parameters.

[0060] For example, cross-component linear model prediction is used in the VVC / H.266 video codec. In this variant, there are three chroma prediction modes that use cross-component linear model prediction. One of these modes can be selected by the encoder as the prediction mode for the chroma prediction block and signaled to the decoder in the bitstream. The difference between the three modes is the set of reference samples used to generate the parameters of the linear model. One of these modes uses only samples above the prediction block; one of these models uses only samples to the left of the prediction block; and one of these modes uses samples above and to the left of the prediction block. In order to keep the complexity of parameter generation low, only a subset of the reference samples available on the block boundary is used to calculate the parameters.

[0061] One possibility to calculate the parameters for such a linear model is to use linear regression, such as that given in the JVET (Joint Video Experts Group) contribution JVET-D0110. In this implementation, there may be two linear models operating in different luminance spectrum ranges. Another possibility is to derive the parameters of the traditional cross-component model by using different types of linear regression tools (such as ordinary least squares estimation, orthogonal matching pursuit, optimized orthogonal matching pursuit, ridge regression or least absolute shrinkage and selection operators).

[0062] The present embodiment relates to a solution for generating an autocorrelation matrix and a cross-correlation vector from a reference data set, such as luminance and chrominance sample values ​​near a sample block to be predicted using a convolution cross-component filter. There may be different (kinds) of cross-correlation vectors in different variants, but usually only one is active at a time. The reference data may also be referred to as "training data" because the filter coefficients of the filter are calculated or "trained" using the data. The reference data consists of two sets: input data and target data (or output data). The purpose of the generated filter is to provide an accurate mapping from the input data to the target data. In the case of a convolution cross-component filter, the input data may be configured to include luminance samples near the sample block, and the target data may be set to include chrominance data near the sample block. Before decomposing the autocorrelation matrix and solving the system parameters using a back-substitution process set, the autocorrelation matrix and the cross-correlation vector may be optionally scaled based on the bit depth of the video content. The solution parameters of the system represent the filter coefficients of the convolution cross-component filter, which can be used to predict chrominance values ​​based on luminance values ​​for chrominance sample blocks. The system is configured to perform intermediate scaling between two back-substitutions using the scaling vector generated during the matrix decomposition. In addition, the present embodiment defines a method of how to reduce the computational requirements of the process and system dimensions in different ways, for example, by excluding the bias term from the linear system of equations and estimating it separately. With respect to the present embodiment, the term "encoder" refers to both video and still image encoders. Likewise, the term "decoder" refers to both video and still image decoders.

[0063] For simplicity, different data structures containing multiple scalar values ​​are represented here in the form of matrices and vectors. Of course, these can also be represented or implemented in other forms. For example, these can be represented or implemented as arrays, lists, other sets of scalar values, or sets of individual scalar values.

[0064] In order to perform the method according to the present embodiment, the parameters of a filter such as a convolution cross-component filter are calculated, which parameters can be represented by a vector x and can be convolved with an input vector z to produce, for example, predicted sample values ​​to be used for video or image encoding purposes. The input vector z may contain, for example, luma samples or downsampled luma samples at predetermined positions relative to predicted chroma samples calculated as an output of the filtering operation. The vector x may consist of n parameters and may be given as follows:

[0065] x=[x0 … x n-1 ] T

[0066] The input vector z can also consist of n input values ​​and can be represented as:

[0067] z=[z0 … z n-1 ] T

[0068] The predicted sample value p can then be calculated by convolving or multiplying the input z with the filter parameter vector x as follows:

[0069]

[0070] The input vector z can be configured to include, for example, a function of a brightness sample value, a brightness sample value, or a constant, or a combination thereof. Including a constant in the input vector z corresponds to adding a constant to the output of the filter p. Such a constant can be referred to as a bias term or bias parameter and can be used to represent an offset between an input value and an output value.

[0071] The filter parameters x can usually be calculated by finding solutions to a set of equations which can be expressed in matrix form as:

[0072] Ax=y

[0073] Where A represents the autocorrelation matrix of the determined input reference samples or "training" samples used in the process, and y represents the cross-correlation vector between the input training samples and the corresponding target training samples. The entries in the n×n matrix A and the vector y with n values ​​can be calculated as follows:

[0074]

[0075]

[0076] Where N is the number of training vectors included in the process, the R matrix contains the input training vectors as its rows, and s represents the vector with the output training samples. This can be given in pseudocode as follows:

[0077]

[0078] where zeroMatrixOfSize(n,n) returns a matrix of size n×n with all values ​​zero, and zeroVectorOfSize(n) returns a vector of size n with all values ​​zero.

[0079] Since the resulting A matrix is ​​symmetric (i.e., A[c][d] is equal to A[d][c]), it is sufficient to compute either the upper or lower triangular part of the matrix. For example, a loop involving parameter d can be started by using the current value of parameter c as follows:

[0080]

[0081] To allow the use of fixed-point arithmetic to compute vector x, the values ​​of matrix A and vector y are scaled based on the bit depth of the video or image content. For example, this can be accomplished by including bitwise shift operations in the computation of matrix A and vector y, as shown below:

[0082]

[0083] Or it can be done after the sum operation, like this:

[0084]

[0085] Or in another way. For example, the generation of the intermediate matrix A and the vector y is done without shifting and scanning them again, and the shifting is performed during the second scan.

[0086] In addition to or instead of performing a shift based on the content bit depth, other aspects may also be considered. For example, the values ​​in the matrix A and the vector y may be shifted based on the number of training vectors or samples included in the generation of A and y, or some approximation of such a number. The shift may also be signaled in the video bitstream or derived from a syntax element or parameter signaled in the video bitstream.

[0087] For example, the shift can be calculated as

[0088] Shift = efg

[0089] Wherein e can represent the bit shift associated with the maximum dynamic range allowed by the values ​​in the matrix A and the vector y. It can be set to, for example, 30, 28, 26, 24 or 16 or other values ​​suitable for the purpose. The value f can represent the shift associated with the bit depth of the video or image content being processed. Since the matrix A and the vector y are usually generated by multiplying a value with similar or the same precision as the video or image content by another value with the same characteristics, it is advantageous to use twice the bit depth value of the content as the value of f. Therefore, for example, for 10-bit video or image content, the value of f can be set to 20. The value g can represent the shift associated with the number of training samples of the vector included in the construction of the matrix A and the vector y. For example, the logarithm of 2 or the logarithm multiple of 2 of the number of training vectors or samples can be used as the value of g. Alternatively, the value of g can be set to a constant and combined with the value of e to simplify the determination of the shift value. If the determined shift value is positive, a bitwise left shift can be performed, and if it is negative, a bitwise right shift can be performed. Likewise, to further simplify the shifting process, a shift may be specified to have only positive or negative values, and if determination of the shift value violates the specified restrictions, a shift value of zero may be used.

[0090] As another example, the shift parameter may be calculated using one of the following notations:

[0091] Shift = B-2*bitDepth-log2(N), or

[0092] Shift = B-2*bitDepth

[0093] Where B is a constant known to both the video encoder and decoder, bitDepth is a measure of the bit depth of the video content, and can be a function of the bit depth of the luminance channel, the chrominance channel, or a combination thereof. N in the above equation can represent the number of training or reference samples used in the matrix generation process, or an estimate thereof. The log2(N) operator can refer to the logarithm of 2 of the value N, and can include rounding up, down, or to the nearest integer value. In addition, other selections can be made. For example, as an alternative or in addition to the number of reference samples, the number of samples to be predicted by the filter can be selected as one of the bases for calculating the shift parameters.

[0094] To solve the equation Ax=y for the filter coefficients x in the vector x i , using a matrix decomposition based method. Different decompositions can be selected for this process. For example, LDL decomposition, Cholesky decomposition or QR decomposition can be used. For example, an implementation described by the following pseudo code can be used, with matrix A as input and upper triangular matrix U and vector d as output:

[0095]

[0096]

[0097] By performing this decomposition, the upper triangular output matrix U corresponds to the transpose of the lower triangular L matrix of the LDL decomposition, and the output vector d contains the values ​​of the diagonal elements of the diagonal matrix D of the LDL decomposition. The values ​​of the output vector d (i.e., the scaling vector) can be called scaling values ​​because these values ​​are used to scale the values ​​of the triangular matrix and are also used to scale intermediate values ​​when solving the decomposition system. The values ​​of the scaling vector d can be represented as vectors, arrays, lists or diagonal elements of matrices, for example. In order to save memory in software or hardware implementations, the values ​​of the vector d can be stored as diagonal elements of the output matrix U, otherwise in the implementation according to the above pseudocode example, these elements have unspecified values.

[0098] As an alternative example, a lower triangular matrix L may be generated. Thus, the filter coefficient vector of Eq. x

[0099] Ax=y

[0100] Now we can solve it by replacing the back-substitution and scaling operations after A with the LDL decomposition:

[0101] LDL T x=y

[0102] Or solve using the U matrix calculated using the pseudocode example:

[0103] U T DUx=y

[0104] Now, the filter coefficient vector x can be solved in three steps. In the first step, the DUx term can be labeled as an intermediate vector z, which can be solved by back substitution:

[0105] DUx=z

[0106] U T z=y

[0107] In a second step, D can be removed by dividing or scaling the elements of the intermediate vector z by the elements of the vector d:

[0108]

[0109] In the third step, since the above equation is again in the form of an upper / lower triangular matrix multiplied by a vector x equal to another vector, back-substitution can be used to directly solve for vector x. Therefore, the entire process of solving the filter coefficients can be configured to have three stages: a first back-substitution process, a scaling process, and a second back-substitution process. Advantageously, the scaling process between the two back-substitutions is performed using a vector generated as a product of the matrix decomposition.

[0110] The multiplication operation MULT and the division operation DIV can be implemented in different ways. For example, a floating point or fixed point implementation can be used. Since a fixed point implementation can provide faster execution speeds on certain computing architectures, it can often be beneficial to use fixed point arithmetic. In order to achieve a favorable balance between the numerical stability of the computation process and the accuracy of the fixed point representation, the MULT operation can be rounded to the nearest integer, while the DIV operation can be rounded to zero.

[0111] For example, a function could be defined in pseudocode as follows:

[0112]

[0113] Alternatively, the DIV operation can be implemented as a combination of, for example, a table lookup operation and a bitwise shift operation. It can also include rounding terms, just like the MULT function in the example above. The DECIM_BITS parameter determines the number of decimal digits in the fixed-point representation and can be set to different values ​​depending on the desired precision of the operation.

[0114] Backsubstitution can be performed in different ways. For example, the following pseudocode can be used:

[0115]

[0116] The dependencies between the process of constructing the matrix factorization and the process of solving the filter coefficients from the system of equations can be jointly optimized. For example, the division operation in the matrix factorization and the division operation between or during the back-substitution process can be configured to use the same denominator d i (corresponding to the diagonal elements of matrix D or the elements of diagonal vector d in the above examples and the D[i] entries in the pseudocode examples.) In these cases, the operation can be advantageously converted to correspond to the number 1 / d i Multiply by the reciprocal of d instead of dividing by d i In this way, only one division operation is required to compute the inverse of each diagonal element, reducing the number of division operations to once per output coefficient.

[0117] Advantageously, the filter coefficients are solved by two back-substitution processes, and a scaling operation is performed between the two processes. The scaling operation uses the values ​​generated as the output of the matrix decomposition operation to scale the output values ​​of the first back-substitution process. The scaling may include, for example, multiplication, division or bit shift operations or a combination of these operations.

[0118] In some cases, the autocorrelation matrix A can be singular. The value of the diagonal vector d can be used as a measure of singularity. In the above example pseudocode, each element of d(d[i]) is compared with a threshold singularityThreshold to determine singularity. The value of singularityThreshold can be determined in different ways. For example, it can be set to a certain value, such as 0 or other integers or floating point numbers. It can also be calculated based on the bit depth of the video content or the characteristics of the A matrix.

[0119] In the case where matrix A is determined to be singular, different actions can be taken. For example, a regularization term can be added to the diagonal elements of matrix A, and then decomposition can be attempted again. However, this method can roughly double the worst-case complexity of the decomposition operation because the decomposition can be at least partially completed twice. Advantageously, before attempting to decompose the matrix, matrix A is regularized by adding a positive term to each diagonal element of matrix A. The value of the positive term can be determined differently. For example, a constant value of 1 or other constants can be used as a regularization term. The value of this term can also be determined based on the bit depth of the video content or the characteristics of matrix A. If matrix A is still found to be singular when trying to decompose matrix A for the first time, the filter coefficient vector x is advantageously set to a vector independent of matrix A, and additional decomposition attempts to matrix A can be omitted. This method limits the number of matrix decomposition attempts to once, so as to roughly halve the worst-case complexity of the method for regularizing matrix A after decomposition failure. Different methods can be used to determine the filter coefficient vector x independent of matrix A. For example, all elements of x may be set to zero, or elements associated with the bias term of a filter may be set to values ​​derived from the bit depth of the video or the average luma and chroma values ​​of a reference sample, and the remaining elements may be set to zero.

[0120] There are different ways to reduce and calculate the filter coefficients x i The associated computational complexity. For example, where a convolution filter is used to predict chrominance values ​​based on input luma values, it may be desirable to include a so-called bias term in the filtering process. The bias term can be thought of as a constant to be added to the filter output. One way to implement this bias is to make the input z b One of the filter coefficients x is kept constant (e.g., a value corresponding to an intermediate brightness value, such as a value of 512 for 10-bit video content, or a value of 128 for 8-bit video content). b Determines the bias value to be added to the filter output. The bias term x bcan be calculated as one of the filter coefficients in the vector x as described above. However, it can also be estimated in other ways. For example, if the input to the filter is luma values ​​and the output is predicted luma, and the filter is generated using a set of reference luma and chroma samples, then the average reference luma value y can be calculated mean and the average reference chrominance sample value c mean When calculating the filter coefficients, y can be subtracted from the reference luminance value before or when generating the autocorrelation matrix A and the cross-correlation vector y. mean And c can be calculated from the reference chrominance samples mean Similarly, when performing a convolution operation to compute the filter output, y can be subtracted from the input luma samples. mean Then, by taking the average reference chromaticity value c mean Adding to the output of the filter, the bias between luma and chroma samples can be restored as follows:

[0121]

[0122] The encoder or decoder may be configured to calculate a subset of filter coefficients from the decomposed autocorrelation matrix and cross-correlation vector. This may be advantageously performed by performing a back-substitution operation on only those elements corresponding to the set of coefficients to be solved in the system of equations.

[0123] In one embodiment, the set of filter coefficients is determined using a first back-alternation process, a scaling process, and a second back-alternation process.

[0124] In one embodiment, the set of filter coefficients is determined based on an autocorrelation matrix decomposed into at least a triangular matrix and a set of scaling values.

[0125] In one embodiment, the autocorrelation matrix is ​​decomposed into at least a triangular matrix of sets of scaled values, wherein the scaled values ​​are represented as vectors, arrays, lists or diagonal elements of the matrix.

[0126] In one embodiment, a set of filter coefficients is determined using a first back-alteration, scaling, and a second back-alteration, wherein the scaling is performed with values ​​generated by decomposing the matrix.

[0127] In one embodiment, a set of filter coefficients is determined using a first back-alteration, scaling, and a second back-alteration, wherein the scaling is performed with values ​​generated by decomposing the autocorrelation matrix.

[0128] In one embodiment, the autocorrelation matrix is ​​decomposed into at least a triangular matrix of sets of scaled values ​​using floating point operations.

[0129] In one embodiment, the autocorrelation matrix is ​​decomposed into at least a triangular matrix of sets of scaled values ​​using fixed point arithmetic.

[0130] In one embodiment, the autocorrelation matrix of the set of scaled values ​​is decomposed into at least a triangular matrix using fixed point operations, wherein fixed point multiplication is configured to include rounding to the nearest fixed point value and fixed point division operations are configured to include rounding to zero.

[0131] In one embodiment, a set of determined scaling factors is applied between two back-iteration processes.

[0132] In one embodiment, a bit shift operation is used to shift the elements of the autocorrelation matrix and the cross-correlation vector, wherein the amount of the bit shift is determined based on the bit depth of the video content.

[0133] In one embodiment, a bit shift operation is used to shift the elements of the autocorrelation matrix and the cross-correlation vector, wherein the amount of the bit shift is determined based on the number of reference samples or a measure based on the number of reference samples.

[0134] In one embodiment, a bit shift operation is used to shift the elements of the autocorrelation matrix and the cross-correlation vector, wherein the amount of the bit shift is determined based on the number of samples to be predicted or based on a measure of the number of samples to be predicted.

[0135] In one embodiment, a bit shift operation is used to shift elements of the autocorrelation matrix and the cross-correlation vector, wherein the amount of the bit shift is determined based on the bit depth of the video content and the number of reference samples used to generate the autocorrelation matrix and the cross-correlation vector.

[0136] In one embodiment, the singularity of the autocorrelation matrix is ​​determined during the decomposition process by comparing the output scaled values ​​to an integer valued threshold.

[0137] In one embodiment, if the autocorrelation matrix is ​​determined to be singular, the filter coefficients are set to zero.

[0138] In one embodiment, if the autocorrelation matrix is ​​determined to be singular, all filter coefficients except the bias term are set to zero.

[0139] In one embodiment, if the autocorrelation matrix is ​​determined to be singular, the bias term of the filter coefficients is set to a value representing the mean of the reference chrominance values.

[0140] In one embodiment, if the autocorrelation matrix is ​​determined to be singular, the bias term for the filter coefficients is set to a value of 512 for 10-bit video and a value of 128 for 8-bit video.

[0141] In one embodiment, positive values ​​are added to the diagonal elements of the autocorrelation matrix prior to decomposition.

[0142] In one embodiment, when determining the autocorrelation matrix and the cross-correlation vector, a mean luma value is subtracted from the luma value and a mean chroma value is subtracted from the chroma value, wherein the mean luma and chroma values ​​are calculated using a reference sample set.

[0143] In one embodiment, before applying the convolution filtering operation, an average luma value is subtracted from the input luma value and an average chroma value is added to the filter output, wherein the average luma and chroma values ​​are calculated using a reference sample set.

[0144] In one embodiment, a back-substitution process is applied to a selected subset of elements in the triangular matrix and the target vector.

[0145] In one embodiment, the determined set of filter coefficients is used to predict chrominance values ​​from luma values.

[0146] According to the method of the embodiment Figure 3 As shown. The method generally includes obtaining 310 an input sample set; obtaining 320 a triangular matrix, a cross-correlation vector, and a scaling vector by using the input sample set; determining 330 an intermediate vector using the triangular matrix and the cross-correlation vector using a first back iteration; scaling 340 the intermediate vector using the scaling vector; determining 350 a filter coefficient vector using the triangular matrix and the scaled intermediate vector using a second back iteration; and determining 360 a predicted sample value for a sample block based on the filter coefficient vector. Each step can be implemented by a corresponding module of a computer system.

[0147] An apparatus according to an embodiment includes a component for obtaining an input sample set; a component for obtaining a triangular matrix, a cross-correlation vector, and a scaling vector by using the input sample set; a component for determining an intermediate vector using a first back iteration using the triangular matrix and the cross-correlation vector; a component for scaling the intermediate vector using the scaling vector; a component for determining a filter coefficient vector using a second back iteration using the triangular matrix and the scaled intermediate vector; and a component for determining a predicted sample value for a sample block based on the filter coefficient vector. The apparatus includes at least one processor and a memory including computer program code, wherein the processor may also include a processor circuit system. The memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to perform the operations according to various embodiments. Figure 3 method.

[0148] Various embodiments may be implemented with the aid of a computer program code that resides in a memory and causes an associated device to perform the method. For example, a device may include circuitry and electronics for processing, receiving, and sending data, computer program code in a memory, and a processor that, when running the computer program code, causes the device to perform features of the embodiments. In addition, a network device such as a server may include circuitry and electronics for processing, receiving, and sending data, computer program code in a memory, and a processor that, when running the computer program code, causes the network device to perform features of various embodiments.

[0149] If desired, the different functions discussed herein may be performed in different orders and / or concurrently with other functions. In addition, if desired, one or more of the above functions and embodiments may be optional or may be combined.

[0150] Although various aspects of the embodiment are listed in the independent claim, further aspects include other combinations of features of the described embodiment and / or dependent claims with features of the independent claim and not just the combinations explicitly listed in the claim.

[0151] It is also noted herein that while the above describes example embodiments, these descriptions should not be viewed in a limiting sense. Rather, several variations and modifications may be made without departing from the scope of the present disclosure as defined in the appended claims.

Claims

1. A device comprising: - means for obtaining a set of input samples; - means for obtaining a triangular matrix, a cross-correlation vector, and a scaling vector by utilizing said set of input samples; - means for determining an intermediate vector using said triangular matrix and said cross-correlation vector by a first back iteration; - means for scaling said intermediate vector using said scaling vector; - means for determining a filter coefficient vector using said triangular matrix and said scaled intermediate vector using a second back iteration; and - means for determining a predicted sample value for a block of samples based on said filter coefficient vector.

2. The apparatus according to claim 1, further comprising: Means for determining the triangular matrix and the scaling vector by decomposing an autocorrelation matrix into at least the triangular matrix and the scaling vector, wherein the autocorrelation matrix is ​​calculated using a set of reference samples in the determined neighborhood of the block of samples.

3. The apparatus according to claim 2, further comprising: Means for determining the autocorrelation matrix and the cross-correlation vector by subtracting an average luma value from a set of luma values ​​and subtracting an average chroma value from a set of chroma values.

4. The device according to claim 1, 2 or 3, further comprising: Means for scaling the autocorrelation matrix and the cross-correlation vector based on a bit depth of the block of samples.

5. The apparatus according to claim 4, further comprising: Means for including a bit-wise shift operation in the process of calculating said autocorrelation matrix and said cross-correlation vector.

6. The apparatus according to any one of claims 1 to 5, wherein the sample set comprises one or more of: luma sample values; functions of luma sample values; bias parameters.

7. The device according to any one of claims 1 to 6, further comprising: Means for encoding the sample set, and / or means for decoding the encoded sample set.

8. A method comprising: - obtain an input sample set; - obtaining a triangular matrix, a cross-correlation vector, and a scaling vector by utilizing the input sample set; - determining an intermediate vector using the triangular matrix and the cross-correlation vector using a first back iteration; - scaling the intermediate vector using the scaling vector; - determining a filter coefficient vector using the triangular matrix and the scaled intermediate vector using a second back iteration; as well as - determining predicted sample values ​​for a block of samples based on said filter coefficient vector.

9. The method according to claim 8, further comprising: The triangular matrix and the scaling vector are determined by decomposing an autocorrelation matrix into at least the triangular matrix and the scaling vector, wherein the autocorrelation matrix is ​​calculated using a set of reference samples in a determined neighborhood of the block of samples.

10. The method according to claim 9, further comprising: The autocorrelation matrix and the cross-correlation vector are determined by subtracting an average luma value from a set of luma values ​​and subtracting an average chroma value from a set of chroma values.

11. The method according to claim 8, 9 or 10, further comprising: The autocorrelation matrix and the cross-correlation vector are scaled based on a bit depth of the block of samples.

12. The method according to claim 11, further comprising: The process of calculating the autocorrelation matrix and the cross-correlation vector includes a bit-by-bit shift operation.

13. The method according to any one of claims 8 to 12, wherein the sample set comprises one or more of: luma sample values; a function of luma sample values; a bias parameter.

14. The method according to any one of claims 8 to 13, further comprising: The sample set is encoded and / or the encoded sample set is decoded.

15. An apparatus comprising at least one processor, a memory comprising computer program code, the memory and the computer program code being configured to, together with the at least one processor, cause the apparatus to at least perform the following operations: - obtain an input sample set; - obtaining a triangular matrix, a cross-correlation vector, and a scaling vector by utilizing the input sample set; - determining an intermediate vector using the triangular matrix and the cross-correlation vector using a first back iteration; - scaling the intermediate vector using the scaling vector; - determining a filter coefficient vector using a second back iteration using the triangular matrix and the scaled intermediate vector; and - determining predicted sample values ​​for a block of samples based on said filter coefficient vector.

Images