Method and apparatus for video encoding and decoding using matrix-based intra-frame prediction

Abstract

Different implementations are described, in particular, an implementation of video encoding and decoding based on linear weighted intra prediction (also called matrix-based intra prediction) is proposed. Thus, for a block encoded or decoded in linear weighted intra prediction, intra prediction samples are obtained from at least two matrix-vector products between at least two selected reduced-size weight matrices and a set of neighboring reference samples. Advantageously, this arrangement allows reducing the amount of memory used to store data and allows reducing the complexity of the intra prediction sample calculations.

Description

Technical Field

[0001] At least one of the embodiments of the present invention generally relates to (for example) a method or device for video encoding or decoding, and more specifically, to a method or device comprising obtaining intra-frame prediction samples of a block from at least two matrix-vector products between at least two weight matrices and a set of neighboring reference samples. Background Art

[0002] The technical field of one or more implementation domains generally relates to video compression. At least some embodiments relate to improving compression efficiency compared to existing video compression systems such as HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265, and MPEG-H Part 2 described in "ITU-T H.265 Telecommunication Standardization Sector (10 / 2014) Series H: Audiovisual and Multimedia Systems, Infrastructure for Audiovisual Services - Coding for Mobile Video, High Efficiency Video Coding, Recommendation ITU-T H.265") or compared to video compression systems under development such as VVC (Versatile Video Coding, a new standard developed by the Joint Video Experts Group JVET).

[0003] To achieve high compression efficiency, image and video coding schemes typically employ prediction (including motion vector prediction) and transforms to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlations, and then the difference between the original image and the predicted image, which is typically represented as a prediction error or prediction residual, is transformed, quantized, and entropy decoded. To reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy decoding, quantization, transform, and prediction.

[0004] A recent addition to high compression techniques includes intra prediction using a combination of downsampling and matrix multiplication of reference samples. A disadvantage of this intra prediction is that a large amount of data is required to store the weighting parameters used in the various filtering processes. Therefore, it is desirable to optimize the storage of the intra prediction matrix for memory efficiency and to optimize the calculation of the intra prediction samples at decoding time for complexity. Summary of the invention

[0005] The object of the present invention is to overcome at least one disadvantage of the prior art. To this end, according to a general aspect of at least one embodiment, a method for video coding is proposed, which comprises coding a block of a picture of a video in an intra prediction mode using linear weighted prediction. According to a specific characteristic, the coding comprises obtaining intra prediction samples of the block from: at least two matrix-vector products between at least two selected weight matrices among a set of weight matrices and associated bias vectors and an associated bias vector, and a set of neighboring reference samples.

[0006] Advantageously, at least two selected weight matrices are constructed to involve fewer multiplications per sample to be predicted, and fewer coefficients to be stored in memory. Thus, low inference complexity enables the ability of matrix-based intra prediction to be implemented in resource-constrained devices that typically implement decoders.

[0007] According to another general aspect of at least one embodiment, a method for video decoding is proposed, comprising decoding a block of a picture of a video, the block being coded in intra prediction mode using linear weighted prediction. For the encoding method, according to a particular characteristic, decoding further comprises obtaining intra prediction samples of the block from: at least two matrix-vector products between at least two selected weight matrices from a set of weight matrices and associated bias vectors and associated biases, and a set of neighboring reference samples.

[0008] According to another general aspect of at least one embodiment, an apparatus for video encoding is proposed, comprising means for implementing any one of the embodiments of the encoding method.

[0009] According to another general aspect of at least one embodiment, an apparatus for video decoding is proposed, comprising means for implementing any one of the embodiments of the decoding method.

[0010] According to another general aspect of at least one embodiment, there is provided an apparatus for video encoding, comprising one or more processors and at least one memory. The one or more processors are configured to implement any one of the embodiments of the encoding method.

[0011] According to another general aspect of at least one embodiment, there is provided an apparatus for video decoding, comprising one or more processors and at least one memory. The one or more processors are configured to implement any one of the embodiments of the decoding method.

[0012] According to another general aspect of at least one embodiment, encoding or decoding a block using linear weighted prediction further comprises: determining a set x of n neighboring reference samples; selecting at least two weight matrices G and H of sizes m×r and r×n from a set of weight matrices and associated bias vectors based on a linear weighted intra prediction mode and a block shape; T and an associated bias vector b, the first selected weight matrix G has a size of m×r, and the second selected weight matrix G has a size of r×n, where m, n, r are integers, and r is less than m and r is less than n; and from the first matrix product H T x, the second matrix product G(H T x) and add the associated bias b to obtain the intra-frame prediction sample of the block The result is

[0013] According to another general aspect of at least one embodiment, the first and second weight matrices G and H T is obtained from a low-rank processing of a weight matrix M of size m×n, and where the estimate of the weight matrix M is are the first and second weight matrices G and H T The product of

[0014] According to another general aspect of at least one embodiment, r is determined to reduce matrix product complexity.

[0015] According to another general aspect of at least one embodiment, r is determined in response to a singular value decomposition (SVD) of the weight matrix M.

[0016] According to another general aspect of at least one embodiment, for a block of size width*height, the set x of neighboring reference samples includes all reconstructed samples from a neighboring top line of size width and from a neighboring left line of size height.

[0017] According to another general aspect of at least one embodiment, for a block of size width*height, the set x of neighboring reference samples includes all reconstructed samples from the neighboring top line of size 2*width and from the neighboring left line of size 2*height.

[0018] According to another general aspect of at least one embodiment, at least one high-level syntax element representing a set of weight matrices and associated biases is signaled in a picture parameter set (PPS) such that all blocks in a frame using linear weighted prediction use the signaled set of weight matrices. According to another general aspect of at least one embodiment, at least one high-level syntax element representing a set of weight matrices and associated biases is signaled in a sequence parameter set (SPS) such that all blocks in a sequence using linear weighted prediction use the signaled set of weight matrices.

[0019] According to another general aspect of at least one embodiment, the bit depth of two matrix-vector products of matrices is increased relative to the bit depth of intra-prediction samples.

[0020] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided, which contains data content generated according to any of the methods or apparatus of the foregoing description.

[0021] According to another general aspect of at least one embodiment, a signal is provided, the signal comprising video data generated according to any of the methods or apparatus described above.

[0022] One or more of the embodiments of the present invention also provides a computer-readable storage medium on which instructions for encoding or decoding video data according to any of the above methods are stored. An embodiment of the present invention also provides a computer-readable storage medium on which a bit stream generated according to the above method is stored. An embodiment of the present invention also provides a method and apparatus for transmitting a bit stream generated according to the above method. An embodiment of the present invention also provides a computer program product including instructions for executing any of the described methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Examples depicting splitting patterns for binary and ternary trees;

[0024] Figure 2 depicts splitting of a coding tree block (CTB) into coding blocks (CBs) according to various splitting modes, such as binary and ternary tree splitting modes;

[0025] Figure 3 The process of sample reference generation from adjacent reference samples is shown;

[0026] Figure 4 Represents various prediction directions as used in intra prediction;

[0027] Figure 5A , Figure 5B , Figure 5C , Figure 5D represents affine linear weighted intra prediction (ALWIP), also known as matrix intra prediction (MIP) for various block sizes, according to a particular embodiment;

[0028] Figure 6 An example encoding or decoding method including intra-frame prediction according to a general aspect of at least one embodiment is shown;

[0029] Figure 7 A block diagram illustrating an embodiment of a video encoder in which various aspects of the embodiments may be implemented;

[0030] Figure 8 A block diagram illustrating an embodiment of a video encoder in which various aspects of the embodiments may be implemented;

[0031] Fig. 9 A block diagram is shown of an example apparatus in which various aspects of the embodiments may be implemented. DETAILED DESCRIPTION

[0032] It should be understood that the accompanying drawings and descriptions have been simplified to illustrate the elements relevant to a clear understanding of the principles of the present invention, and for clarity, many other elements found in typical coding and / or decoding devices have been removed. It should be understood that although the terms first and second can be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element.

[0033] Various embodiments are described for encoding / decoding of pictures. They may be applied to encoding / decoding a portion of a picture, such as a slice or tile, or an entire sequence of pictures.

[0034] Various methods are described above, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for the correct operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0035] At least one embodiment effectively combines the matrix-based intra prediction tools and matrix structure decomposition used in VVC to simultaneously reduce the memory footprint of these matrices and facilitate decoder-side calculations. According to a non-limiting example, the matrix structure decomposition includes matrix factorization, such as a low-rank representation and a low-displacement rank representation of a weight matrix.

[0036] In Section 1, some limitations on matrix-based intra prediction are disclosed.

[0037] In section 2, several embodiments of a method for encoding or decoding using linear weighted intra prediction are disclosed.

[0038] In Section 3, additional information and general embodiments for implementing the disclosed methods are disclosed.

[0039] 1. Intra-frame prediction: principles and limitations

[0040] In HEVC coding, pictures are partitioned into square-shaped CTUs with a configurable size, typically 64×64. In other video coding standards, the size can be 128×128 or 256×256. A CTU is the root of a quadtree that is partitioned into 4 equally sized square coding units (CUs), i.e., the size is half the size of the parent block in width and height. A quadtree is a tree in which a parent node can be split into four child nodes, each of which can become a parent node and then split into four child nodes. In HEVC, a coding block (CB) is partitioned into one or more prediction blocks (PBs) and forms the root of a quadtree that is partitioned into transform blocks (TBs). Corresponding to the coding blocks, prediction blocks, and transform blocks, a coding unit (CU) includes a tree structure set of prediction units (PUs) and transform units (TUs), the PU includes prediction information for all color components, and the TU includes a residual decoding syntax structure for each color component. The sizes of the CB, PB, and TB of the luminance component apply to the corresponding CU, PU, ​​and TU.

[0041] In more recent coding systems, a CTU is the root of a coding tree that is partitioned into coding units (CUs). A coding tree is a tree in which a parent node (usually corresponding to a block) can be split into child nodes (e.g., split into 2, 3, or 4 child nodes), each of which can become a parent node and then split into child nodes. In addition to the quadtree split mode, new split modes (binary tree symmetric split mode, binary tree asymmetric split mode, and ternary tree split mode) are defined, which increases the total number of possible split modes. Figure 1 An example of such a splitting mode is depicted. The two splitting modes on the left are classic binary splitting modes, which split the block vertically (SPLIT_BT_VER) or horizontally (SPLIT_BT_HOR) into two identical sub-blocks. The two splitting modes on the right are examples of three-tree splitting modes. The SPLIT_TT_VER mode splits the block vertically into three sub-blocks with widths N / 4, N / 2, and N / 4, respectively. The SPLIT_TT_HOR mode splits the block horizontally into three sub-blocks with heights N / 4, N / 2, and N / 4, respectively. The decoding tree has a unique root node, such as a CTU. The leaves of the decoding tree are the terminal nodes of the tree. Each node of the decoding tree represents a block that can be further split into smaller blocks, which are also called sub-blocks or more generally sub-blocks. Once the partitioning of the CTU to the CU is determined, the CU corresponding to the leaf of the decoding tree is encoded. The partitioning of the CTU into CUs and the decoding parameters for encoding each CU (corresponding to the leaf of the decoding tree) can be determined on the encoder side by a rate-distortion optimization process. Figure 2 Depicts the splitting of a CTU into CUs according to various splitting modes. More precisely, it represents the coding tree of a CU with nested quadtree (QT), binary tree (BT), and ternary tree (TT) structures.

[0042] In this application, the term "block" or "picture block" may be used to refer to any of CTU, CU, PU, ​​TU, CB, PB, and TB. In addition, the term "block" or "picture block" may be used to refer to macroblocks, partitions, and sub-blocks as specified in H.264 / AVC or other video coding standards, and more generally to a series of samples of varying sizes.

[0043] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image", "picture" and "frame" are used interchangeably. Usually, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0044] Intra prediction in video compression refers to the spatial prediction of a pixel block using information from causal neighboring blocks (i.e., neighboring blocks that have been decoded in the same frame). This is a powerful coding tool because it allows high compression efficiency in INTRA frames and in INTER frames when there is no better temporal prediction. Therefore, intra prediction has been included as a core coding tool in all video compression standards including H.264 / AVC, HEVC, etc.

[0045] Intra prediction exploits the spatial correlation between pixels that may belong to the same object, background, or region, etc. In this context, intra prediction in video coding standards such as H.264 / AVC, H.265 / HEVC, etc. has been designed to capture the directionality of object orientation and the slowly changing intensity of regions or textures. In HEVC, for example, intra prediction includes 35 prediction modes, including a DC prediction mode, a plane (PLANAR) prediction mode, and 33 angle prediction modes. The angle mode is designed to model the directional structure of the object, while the DC and plane modes provide predictions for regions with slow and gradual intensity changes, and also provide predictions for regions with changing textures. In the general video decoding test model (VTM) aimed at designing the future standard H.266, the number of prediction modes has been increased to 67 to accommodate other directions that are particularly useful for large block sizes. In the case of the directional prediction mode, the filtered pixel values ​​from the adjacent left and upper neighbors are repeated in a predefined direction. In order to smoothly predict along block boundaries, a position-dependent intra prediction combination (PDPC) is introduced. Its purpose is to use additional reference samples to smooth the prediction at the target block boundary. For some prediction directions, some reference samples are not available, so padding of reference samples is used.

[0046] In HEVC, the encoding of frames of video sequences is based on a quadtree (QT) block structure. The frame is divided into square coding tree units (CTUs), all of which undergo quadtree-based splitting according to rate-distortion criteria to become multiple coding units (CUs). Each CU contains at least one prediction unit (PU), which is a base block for prediction tools. In intra-frame prediction, the PU is spatially predicted from causal adjacent PUs (i.e., the top and left PUs). For this purpose, HEVC uses a simple spatial model called prediction mode. Based on the decoded pixel values ​​in the top and left PUs called reference pixels, the encoder constructs different predictions for the target block and selects the prediction that results in the best RD performance. Of the 35 defined modes, one is a plane mode (indexed as mode 0), one is a DC mode (indexed as mode 1), and the remaining 33 (indexed as modes 2-34) are angle modes. The angle mode is intended to model the directional structure of objects in the frame. Therefore, the decoded pixel values ​​in the top and left CUs are simply repeated along the defined directions to fill the prediction block. Because this process can result in discontinuities along the top and left reference boundaries for some modes, those prediction modes include subsequent post-filtering to smooth pixel values ​​along those boundaries.

[0047] The above prediction model works very well as long as the intensity values ​​do not vary too much. However, in natural imaging, the intensity values ​​on an object often experience variations for several reasons. As a color property of the object itself, or due to lighting, depth, motion, etc., the intensity values ​​on a PU may experience variations that cannot be adequately modeled using pixel repetition. This is especially true when the PU size is large. In VTM, it has been proposed to use CTU sizes up to 128. Therefore, it makes more sense to consider other prediction models, such as interpolation, which will model the intensity variations more effectively.

[0048] Intra prediction in HEVC

[0049] The intra prediction process in HEVC consists of three steps: (1) reference sample generation; (2) intra sample prediction; and (3) post-processing of the predicted samples. Figure 3 As shown. For a PU of size N×N, a top row of 2N decoded samples is formed from the top and right top pixels of the previously reconstructed current PU. Similarly, a column of 2N samples on the left is formed from the reconstructed left and bottom left pixels. The corner pixels at the upper left position are also used to fill the gap between the top row and left column references. If some of the top or left samples are not available, for example, because the corresponding CU is not in the same slice, or the current CU is at the frame boundary, a method called reference sample replacement is performed, in which the missing samples are copied from the available samples in a clockwise direction. Next, depending on the current CU size and prediction mode, the reference samples are filtered using a specified filter.

[0050] The next step, intra sample prediction, consists of predicting the pixels of the target CU based on the reference samples. As mentioned before, in order to effectively predict different kinds of content, HEVC supports a range of prediction models. Planar and DC prediction modes are used to predict smooth and gradually changing areas, while angular prediction modes are used to capture different directional structures. HEVC supports 33 directional prediction modes indexed from 2 to 34. These prediction modes correspond to different prediction directions, such as Figure 4 shown.

[0051] like Figure 4 As shown, the defined angular direction results in a sample accuracy of 1 / 32. That is, between any two adjacent reference samples, there are 32 possible directions. The defined direction can be distinguished as vertical or horizontal. The prediction in the horizontal direction uses only the left reference sample or some left and some top reference samples. Similarly, the prediction in the vertical direction uses only the top reference sample or some top and some left reference samples. The direction using only the left reference sample or only the top reference sample is defined as the positive direction. Therefore, we have a horizontal positive direction from H0 to H+32, which uses only the left reference sample. Similarly, we have a vertical positive direction from V0 to V+32, which uses only the top reference sample. Other horizontal and vertical directions (H-2 to H-26 and V-2 to V-32) are defined as negative directions, and they use reference samples from both the left and top. The following table shows the relationship between the prediction mode and the angle parameter A specified by HEVC:

[0052] Table 1 Mapping of mode index to angle parameter A

[0053]

[0054]

[0055] In the HEVC reference software, the reference array is first constructed using the top and left reference samples. For vertical prediction, the reference array is horizontal, while for horizontal prediction, the reference array is vertical. For modes with a positive angle parameter A (modes 2 to 10 and 26 to 34), the reference array is simply the top reference samples topRef[] or the left reference samples leftRef[], depending on the direction:

[0056] topRef[x]=P[x-1][-1], 0≤x≤2N, for vertical prediction

[0057] leftRef[y]=P[-1][y-1], 0≤y≤2N, for horizontal prediction

[0058] Where N is the CU size, and P[x][y] is Figure 3The reconstructed samples of the neighboring CU depicted above. The sample coordinates at the top left pixel of the target CU are conventionally initialized to (0, 0). Therefore, the top reference sample will have its y coordinate as -1, while the left reference sample will have its x coordinate as -1.

[0059] For modes with negative angle parameter A (modes 11 to 25), the reference array requires pixels from both the top and left references. In this case, the reference array will be extended to negative indices beyond 0. As described above, sample values ​​on the reference array with positive indices are obtained depending on the vertical or horizontal prediction. Those pixels on the reference array with negative indices are obtained by projecting the left (for vertical prediction) or top reference pixels (for horizontal prediction) onto the reference array along the prediction direction.

[0060] Once the reference array is constructed, the predicted sample value Pred[x][y] at any pixel position (x, y) within the target CU is obtained by first projecting the pixel position (x, y) to the reference array along the selected direction to obtain the reference array position index i (integer index) and f (fractional index). Then, the predicted sample value Pred[x][y] is calculated at a sample resolution of (1 / 32) by interpolating between two adjacent reference samples, as shown below:

[0061] Pred[x][y]=((32-f)*topRef[x+i+1]+f*topRef[x+i+2]+16)>>5), 0≤x, y<N, for vertical prediction

[0062] Pred[x][y]=((32-f)*leftRef[y+i+1]+f*leftRef[y+i+2]+16)>>5), 0≤x, y<N, for horizontal prediction,

[0063] where i and f represent the integer and fractional parts of the projected displacement from the pixel location (x, y).

[0064] If Δ represents the projection displacement, then:

[0065] Δ = (x + 1) * A, for horizontal prediction, and

[0066] Δ=(y+1)*A, for vertical prediction.

[0067] i=Δ>>5, where >> is an arithmetic right shift.

[0068] f=Δ&31, where & is the bitwise AND operator.

[0069] In the case of f=0, ie no fractional part, the prediction is equal to the reference array sample in the prediction direction.

[0070] As we can observe from the above expressions, the vertical prediction is independent of the y-coordinate, while the horizontal prediction is independent of the x-coordinate. This means that for vertical prediction, the predicted values ​​are repeated in the direction predicted from the reference array on the top. Similarly, for horizontal prediction, the predicted values ​​are repeated in the direction predicted from the reference array on the left. Therefore, if the coordinates of two or more pixels have the same projection point on the reference array, they have the same predicted value.

[0071] VTM extension for intra prediction (VTM Universal Video Decoding Test Model )

[0072] Another prediction approach involving learned matrices for intra prediction, described in "CE3: Affine Linear Weighted Intra Prediction (CE3-4.1, CE3-4.2)" (JVET-N0217, 14th meeting: Geneva, CH, March 19-27, 2019), was recently adopted in the reference model of H.266 / VVC. This tool is also called matrix-based intra prediction (MIP) or linear weighted intra prediction. Several weight matrices are learned offline and then applied at the encoder and decoder in order to map the causally reconstructed samples to the current block to be predicted. Given the rate / distortion criterion for a coded block, one of the matrices is selected at the encoder side depending on the size and shape of the block and its performance in predicting the block.

[0073] Like almost all conventional intra prediction tools, to predict a rectangular block of width W and height H, affine linear weighted intra prediction (ALWIP) takes as input a row of H reconstructed neighboring boundary samples to the left of the block and a row of W reconstructed neighboring boundary samples above the block. Similar to the conventional intra prediction process of HEVC, affine linear weighted intra prediction has three main steps: (1) reference sample reduction; (2) intra sample prediction; and (3) interpolation of lost prediction samples.

[0074] First, neighboring reference samples are grouped and averaged, resulting in a vector of 4 values ​​in the case of W=H=4, and 8 values ​​for larger blocks. The method then operates as a fully connected layer of a neural network, where the input is the above vector and the output includes a vector representing a set of samples, which represents a sub-sampled set of samples in the block to be predicted. Finally, the full prediction is generated by interpolating the samples obtained from the sub-sampled set by linear interpolation.

[0075] Three sets of matrices and vectors S0, S1, S2 are available for the encoder to choose from. For blocks of size 4×4, set S0 contains 18 matrices of size 16×4 and 16 and bias vectors, respectively. For blocks of size 4×8, 8×4, and 8×8, S1 consists of 10 matrices of size 16×8 and bias vectors of size 16. Finally, for all other block shapes, set S2 consists of 6 matrices of size 64×8 and 6 bias vectors of size 64.

[0076] Depending on the mode and block shape, the input vector (which is called bdry for the "reduced boundary") red ) is rearranged as follows:

[0077]

[0078] in and Represent the average boundary samples on the top and left, respectively. Figure 5A shown.

[0079] Then, an output reduced set of samples pred for the prediction block is generated red as follows:

[0080] pred red =A.bdry red +b

[0081] Depending on the mode and block shape, the weight matrix A and the offset vector b are selected from the set S0, S1, S2, such as in

[0082]

[0083] as well as

[0084]

[0085] exist Figure 5B as well as Figure 5C The following table shows the pred in the case of 8×8 and 8×4 blocks. red The interpolation process performed.

[0086] For larger blocks, the reference samples are grouped and averaged so that the input vector is 8 samples long, as Figure 5D The above is shown for a 16×16 block.

[0087] Therefore, the method needs to store the sets S0, S1, S2 corresponding to the following:

[0088] - 18 matrices of size 16×4 and 18 offsets of size 16

[0089] - 10 matrices of size 16x8 and 10 offsets of size 16

[0090] - 6 matrices of size 64×8 and 6 offsets of size 64,

[0091] This represents 6336 coefficients to be decoded, which, when stored as 10-bit values, corresponds to 7.92 kilobytes of data.

[0092] Therefore, it is desirable to reduce the size of the matrices stored as well as to reduce the number of operations to calculate for each prediction sample.

[0093] 2. At least one embodiment of encoding or decoding using linear weighted intra-frame prediction

[0094] A general aspect of at least one embodiment is directed to reducing the amount of data required for affine linear weighted intra prediction (ALWIP), also known as matrix-based intra prediction (MIP). Advantageously, instead of applying the intra Pred (pred red =A.bdry red +b) of the section VTM extension, at least one embodiment proposes a matrix structure that requires fewer coefficients and involves fewer multiplications to derive a set of samples for intra-frame prediction before optional linear interpolation.

[0095] Figure 6 An example of an encoding or decoding method including obtaining intra-prediction samples according to a general aspect of at least one embodiment is shown.

[0096] Those skilled in the art will appreciate that once the intra prediction mode is known, linear weighted intra prediction is applied in the same way for encoding or decoding. Therefore, the principles of the present invention are disclosed for an encoding or decoding method. In this step 11, it is assumed that the block is encoded or decoded in intra prediction using linear weighted prediction. The block (usually a CU) belongs to a spatial partition of a picture, typically a CTU in a previous segment. The encoding or decoding method 10 includes, for a block encoded / decoded in intra prediction mode using linear weighted prediction, obtaining intra prediction samples of the block from: at least two matrix products between at least two selected weight matrices from a set of weight matrices and associated bias vectors and associated biases, and a set of adjacent reference samples. The sub-step of obtaining intra prediction samples of the block is now described in detail. In step 12, a set of n adjacent reference samples is determined, where n is an integer representing the number of reference samples. Advantageously, in a first variant, the reference samples as previously disclosed for the "reduced boundary" are obtained by averaging depending on the block shape (i.e., according to its width and height), and n is equal to 4 or 8, whereas, according to a second variant, for a block of size width*height, the set of neighboring reference samples x includes all reconstructed samples from the neighboring top line of size width and from the neighboring left line of size height. In other words, no downsampling by averaging the input samples is applied to the neighboring samples, so the number of reference samples is increased compared to the current VVC linear intra prediction. This is affordable because the structure of the matrix according to the principles of the invention involves fewer multiplications to obtain the prediction samples. Therefore, the complexity of the larger number of reference samples is balanced with a smaller matrix-vector product. Therefore, according to a third variant, as defined for the classical angular intra prediction, the reference samples are even extended to twice the block length or width plus the upper left pixel. For a block of size width*height, in addition to the upper left pixel, the set of neighboring reference samples also includes all reconstructed samples from the neighboring top line of size 2*width and from the neighboring left line of size 2*height. In step 13, at least two weight matrices are selected. The structures of the two matrices are described in the low rank section and the low displacement rank section below. The present invention principle involves using at least two weight matrices G and H of size m×r and r×n from a set of weight matrices and associated bias vectors based on a linear weighted intra prediction mode and a block shape as explained previously for weighted intra prediction. T And the associated bias vector b. However, according to the present principles, the first selected weight matrix G has a size of m×r, and the second selected weight matrix H Thas a size r×n, where m, n, and r are integers, r is less than m, and r is less than n. n is the number of neighboring reference samples of the block previously determined according to any variant. m is the reduced number of intra prediction samples obtained from linear weighted prediction. As previously disclosed for linear weighted prediction, mode k and block shape allow the selection of a pair of matrices (G, H T ). As shown below, the first and second weight matrices G and H T (For example) is obtained from a low-rank processing of the weight matrix M in the set S0, S1, S2. However, at least one embodiment is not limited to low-rank decomposition and any matrix factorization is compatible with at least one embodiment. To illustrate the reduction in the number of coefficients, the matrix 16×4 of the set S0 is considered in the following first example. According to the prior art, 64 coefficients are stored for the matrix, and according to at least one embodiment, r=2, 16×2+2×4=38 coefficients are stored. Using the above calculations, for the matrix 64×8 of the set S2, only 288 coefficients are required for the rank r=4, instead of 512 coefficients. Therefore, the number of operations to obtain the predicted samples is also reduced. In this embodiment, the first and second weight matrices are derived from the matrices in the previously disclosed matrix set, and the estimate of the weight matrix M are the first and second weight matrices G and H T The product of However, the inventive concept is not limited to obtaining matrices by processing a set of disclosed weight matrices. According to an alternative variant, with respect to the disclosed matrix M, a pair of matrices (G, H) of sizes m×r and r×n T ) is obtained from deep learning-based training. Therefore, the additional constraint is the rank r, defining G and H T The shape of , and is used as an additional input to the training phase. According to a first variant, r is determined to reduce the complexity of the matrix product. In a first non-limiting example, r is predetermined in the range [2; min(n, m)]. In another non-limiting example, r is determined in response to a single value decomposition SVD of a weight matrix M in the disclosed set of weight matrices. Then, in step 14, the first matrix product H is processed T x, where x is the reference sample set, also denoted as pred as described above red Then, in step 15, the second matrix product G(H T x). The processing of the multiplication is incorporated into the processor (GPU or CPU) as a combination of hardware and software known to those skilled in the art. Thus, a massively parallel implementation is compatible with the principles of the present invention. Then, in step 16, the bias vector is added, and the result is in is the reduced set of intra prediction samples. In a final optional step 17, in the case where the number of intra prediction samples is smaller than the number of samples in the block, linear interpolation allows obtaining a linearly weighted intra prediction for each sample of the block being coded or decoded.

[0097] According to at least one embodiment, the trade-off between performance and complexity can be rebalanced. Since fewer parameters can be stored by considering low rank, their bit depth can be increased, thereby improving performance while maintaining a smaller number of multiplications. According to another variant, the bit depth of the product of the two matrices is increased relative to the bit depth of the intra-frame prediction samples, and then the clipping is applied (before or after adding the bias). Advantageously, this process leads to higher precision. According to another variant, since the number of coefficients to be stored is reduced, the bit depth of the coefficients of the two matrices themselves is increased.

[0098] According to another embodiment, since the number of coefficients is reduced, the weight matrix can be adapted when encoding or decoding. Therefore, an indication is transmitted to the decoder in the bitstream so that the decoder applies the same weight matrix process as used at the encoder. According to at least one embodiment, the matrix can be transmitted in part or in whole in the bitstream. In a non-limiting example, at least one high-level syntax element representing a set of weight matrices and associated biases is included in a picture parameter set (PPS) to indicate that all blocks in a frame using linear weighted prediction use the set of weight matrices sent using a signal. In another non-limiting example, at least one high-level syntax element representing a set of weight matrices and associated biases is signaled in a sequence parameter set (SPS), and all blocks in a sequence using linear weighted prediction use the set of weight matrices sent using the signal.

[0099] Low rank

[0100] According to the first embodiment, the tradeoff of mapping complexity versus performance can be optimized by keeping weight singular vectors that are more relevant to generating intra prediction using a low-rank version of the original structure of the disclosed matrix set S0, S1, S2.

[0101] The low-rank factorization can be obtained using the well-known singular value decomposition (SVD) method, which states that for any matrix There exists a singular value decomposition M = U∑V T . U and V can be decomposed into U = [U1U2] and V = [V1V2], and Σ is a diagonal matrix containing the real non-negative singular values ​​of A in descending order. Therefore, Σ can be decomposed into U1, and ∑1 have sizes m×r, r×n, and r×r, respectively.

[0102] r is a parameter chosen. If it is equal to the actual rank of A, then there is equality; otherwise, the resulting operator exists in the approximation of M.

[0103] storage The common way to write where G and H are of sizes m×r and r×n T Corresponding to U1, Multiply by the square root of the diagonal value of ∑1.

[0104] consider Figure 5D The case described in , where the tool first generates a set of 64 samples from 8 input samples. This involves 8×64=512 coefficients. In this case, applied to an input vector of size 8×1, the corresponding process requires 512 multiplications. In general, the complexity of matrix-vector multiplication of size m×n is O(mn).

[0105] Consider the rank 4 approximation, then G and H T The size is 64×4 and 4×8, which results in 256+32=288 parameters. Applied to an input vector of size 8, the corresponding process needs to be used to calculate H T x and G(H T x). Thus, a total of 256+32=288 multiplications are required instead of 512. In general, the complexity of the product of rank r is reduced to O((m+n)r).

[0106] In current VVC weighted intra prediction, the number of both input and output samples is reduced to meet the complexity criteria. Both reductions can be considered manual, as they downsample the input reference samples by averaging them, and compute a subset of prediction samples, which are interpolated to produce the final prediction block. The proposed structure enables training a weight structure that optimizes the use of all reference samples while ensuring a reasonable number of weights, thereby requiring fewer operations.

[0107] In addition, for existing methods, the range of predicted pixels needs to fit the range of pixels of the video, for example (0, 1023) in the case of 10-bit output video. This should be handled by normalizing the weights and biases to ensure it or by clipping the output signal.

[0108] Low displacement rank

[0109] Alternatively, according to a second embodiment, the weight matrix can be approximated by a low displacement rank matrix (LDR). The LDR representation will contain fewer weights to store and load in memory, which will enable the inference of the weight matrix to be performed on a decoder with very limited resources.

[0110] The calculation of the LDR matrix is ​​described in Pan, Victor Y, and Xinmao Wang, "Inversion of the Displacement Operator", SIAM Journal on Matrix Analysis and Applications, Vol. 24.3, pp. 660-677, 2003.

[0111] Assume that the matrix M has low rank r k <<min{m,n}, then this means:

[0112] L A，B (M) = M-AMB = GH T

[0113] Where A and B are m×m and n×n shape matrices, respectively, G is an m×r matrix, H is an n×r matrix, and m and n are the number of rows and columns of the original weight matrix M. Here, the displacement r is a selected parameter. For the case of low-rank approximation, a small r means higher compression and computational efficiency.

[0114] Pan, Victor Y et al. proved that the above equation can be expressed as:

[0115]

[0116] where r is the selected displacement rank parameter, Z f =(I n-1 f ) is the f-circular matrix, g j and h j is the r vector of the above G and H. The operator Z f，n，m The definition is as follows:

[0117] For a vector v=(v1, ..., v m ) T , have:

[0118]

[0119] J n represents an n×n reflection matrix, i.e., all anti-diagonal entries are 1, as shown below:

[0120]

[0121] As demonstrated by Pan, Victor Y et al., using the LDR approximation is particularly advantageous when dealing with large matrices, since due to the structure of the chosen operator, the matrix multiplication can be replaced by a Fast Fourier Transform operation.

[0122] This may become convenient when extending the current tool to larger initial matrices.

[0123] 3. Additional Examples and Information

[0124] The present application describes a number of aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described as having specificity, and at least in order to illustrate individual characteristics, are usually described in a manner that may sound limited. However, this is for the purpose of describing clearly, and does not limit the application or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide other aspects. In addition, these aspects can also be combined and interchanged with the aspects described in the documents submitted earlier.

[0125] The aspects described and contemplated in this application can be implemented in many different forms. Figure 7 , Figure 8 and Fig. 9 Some embodiments are provided, but other embodiments are contemplated and are not intended to be construed in detail. Figure 7 , Figure 8 and Fig. 9 The discussion does not limit the breadth of implementation. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the methods described, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the methods described.

[0126] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image", "picture" and "frame" are used interchangeably. Usually, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0127] Various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0128] Various methods and other aspects described in this application can be used to modify modules, such as Figure 7 and Figure 8The intra prediction modules (160, 260) of the video encoder 100 and decoder 200 are shown. Furthermore, aspects of the present invention are not limited to VVC or HEVC, and may be applied to, for example, other standards and recommendations (whether previously existing or developed in the future), as well as extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, the aspects described in this application may be used alone or in combination.

[0129] By way of example, various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes, and the described aspects are not limited to these specific values.

[0130] Figure 7 A video encoder 100 is shown. Variations of the encoder 100 are contemplated, but for the sake of clarity, the encoder 100 is described below without describing all contemplated variations.

[0131] Before being encoded, the video sequence may undergo pre-encoding processing (101), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0132] In encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is divided (102) and processed in units such as CUs. Each unit is encoded using, for example, intra or inter mode. When the unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and motion compensation (170) are performed. The encoder decides (105) whether to encode the unit using one of intra mode or inter mode, and indicates the intra / inter decision by, for example, a prediction mode flag. The prediction residual is calculated, for example, by subtracting (110) the predicted block from the original image block.

[0133] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder may also skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass both the transform and quantization, i.e., directly code the residual without applying the transform or quantization process.

[0134] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are inverse quantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. An in-loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0135] Figure 8 2 shows a block diagram of a video decoder 200. In the decoder 200, the bitstream is decoded by the decoder elements as described below. The video decoder 200 generally performs the same operations as described above. Figure 7 The encoding pass is described as the inverse of the decoding pass. Encoder 100 also typically performs video decoding as part of encoding the video data.

[0136] Specifically, the input to the decoder includes a video bitstream that may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other decoding information. Picture segmentation information indicates how the picture is segmented. Thus, the decoder can divide (235) the picture according to the decoded picture segmentation information. The transform coefficients are inverse quantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined (255) with the prediction block to reconstruct the image block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0137] The decoded picture may further undergo post-decoding processing (285), such as inverse color transformation (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or performing an inverse remapping of the remapping process performed in the pre-encoding process (101). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0138] Fig. 9A block diagram of an example of a system in which various aspects and embodiments are implemented is shown. System 1000 can be implemented as a device including various components described below, and is configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances and servers. The elements of system 1000 can be implemented individually or in combination in a single integrated circuit (IC), multiple ICs and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed on multiple ICs and / or discrete components. In various embodiments, system 1000 is coupled to one or more other systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port communication. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0139] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing, for example, various aspects described in this document. The processor 1010 may include embedded memory, input-output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include a non-volatile memory and / or a volatile memory, including but not limited to an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a magnetic disk drive, and / or an optical disk drive. As a non-limiting example, the storage device 1040 may include an internal storage device, an attached storage device (including a removable and non-removable storage device), and / or a network-accessible storage device.

[0140] The system 1000 includes an encoder / decoder module 1030, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module (or modules) that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000 or may be incorporated into the processor 1010 as a combination of hardware and software as known to those skilled in the art.

[0141] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of the various items during the execution of the processes described in this document. These stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0142] In some embodiments, memory within the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or a storage device 1040, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, an operating system for a television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations such as MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by JVET, Joint Video Experts Team).

[0143] As shown in block 1130, input to the elements of system 1000 may be provided through various input devices. Such input devices include, but are not limited to: (i) an RF section that receives a radio frequency (RF) signal transmitted over the air, for example, by a broadcaster; (ii) a component (COMP) input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Fig. 9 Other examples not shown include composite video.

[0144] In various embodiments, the input device of block 1130 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band); (ii) down-converting the selected signal; (iii) again band-limiting to a narrower frequency band to select a signal frequency band that may be referred to as a channel in some embodiments; (iv) demodulating the down-converted and band-limited signal; (v) performing error correction; and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and filter to the desired frequency band again by filtering, down-conversion and perform frequency selection. Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements that perform similar or different functions. For example, adding element can include inserting element between existing element, for example, inserting amplifier and analog-to-digital converter. In various embodiments, the RF part includes antenna.

[0145] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented as needed, for example, in a separate input processing IC or processor 1010. Similarly, various aspects of USB or HDMI interface processing may be implemented as needed, either in a separate interface IC or in processor 1010. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 1010 and encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on output devices.

[0146] The various components of system 1000 may be provided in an integrated housing. Within the integrated housing, the various components may be interconnected and transmit data between them using a suitable connection arrangement 1140 such as an internal bus (including an inter-IC (I2C) bus), wiring, and printed circuit boards as known in the art.

[0147] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data through the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0148] In various embodiments, a wireless network, such as a Wi-Fi network, such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers), is used to stream data or otherwise provide it to the system 1000. The Wi-Fi signals of these embodiments are received by a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 of these embodiments is usually connected to an access point or a router, which provides access to an external network including the Internet to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to the system 1000, which passes data through the HDMI connection of the input block 1130. Still other embodiments use the RF connection of the input block 1130 to provide streaming data to the system 1000. As described above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use a wireless network other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0149] System 1000 can provide output signals to various output devices, including display 1100, speaker 1110 and other peripheral devices 1120. Display 1100 of various embodiments includes one or more of, for example, a touch screen display, an organic light emitting diode (OLED) display, a curved display and / or a foldable display. Display 1100 can be used for a television, a tablet computer, a laptop computer, a cellular phone (mobile phone) or other devices. Display 1100 can also be integrated with other components (e.g., as in a smart phone), or applied separately (e.g., an external monitor for a laptop computer). In each example of each embodiment, other peripheral devices 1120 include one or more of an independent digital video disk (or digital versatile disk) (DVR, two items), a disk player, a stereo system and / or a lighting system. Various embodiments use one or more peripheral devices 1120, which provide functions based on the output of system 1000. For example, a disk player performs the function of playing the output of system 1000.

[0150] In various embodiments, control signals are transmitted between the system 1000 and the display 1100, speaker 1110, or other peripheral device 1120 using signaling such as AV.Link, consumer electronics control (CEC), or other communication protocols, which enables device-to-device control with or without user intervention. Output devices can be communicatively coupled to the system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, the output device can be connected to the system 1000 via the communication interface 1050 using the communication channel 1060. For example, the display 1100 and the speaker 1110 can be integrated into a single unit in an electronic device (e.g., a television) along with other components of the system 1000. In various embodiments, for example, the display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0151] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may alternatively be separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0152] These embodiments may be performed by computer software implemented by the processor 1010, hardware, or a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, as a non-limiting example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 1010 may be of any type suitable for the technical environment and may include, as a non-limiting example, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0153] Various implementations involve decoding. As used in this application, "decoding" may include, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also or alternatively includes a process performed by a decoder of various implementations described in this application, such as matrix-based intra-frame prediction that reduces memory usage using at least two products of matrices of reduced size.

[0154] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0155] Various implementations involve encoding. In a manner similar to the discussion above about "decoding", "encoding" as used in this application can include, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as segmentation, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such processes also or alternatively include processes performed by encoders of various implementations described in this application, such as matrix-based intra-frame prediction using at least two products of matrices of reduced size.

[0156] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will become clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0157] Note that the syntax elements used herein are descriptive terms. Therefore, they do not exclude the use of other syntax element names.

[0158] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0159] The implementations and aspects described herein may be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the features discussed may also be implemented in other forms (e.g., a device or program). For example, a device may be implemented with appropriate hardware, software, and firmware. The method may be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0160] References to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations thereof in various places throughout this application are not necessarily all referring to the same embodiment.

[0161] Additionally, the present application may refer to “determining” various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0162] Furthermore, the present application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0163] Additionally, the present application may involve "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is generally involved in one way or another.

[0164] It should be understood that, for example, in the case of "A / B", "A and / or B", and "at least one of A and B", the use of any of the following " / ", "and / or", and "at least one of..." is intended to cover the selection of only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to include the selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to many of the listed items, as will be clear to one of ordinary skill in this and related arts.

[0165] In addition, as used herein, the word "signal" especially refers to a corresponding decoder indicating something. For example, in some embodiments, the encoder sends a specific one of a plurality of parameters for matrix-based intra-frame prediction with a signal. In this way, in an embodiment, the same parameters are used on the encoder side and the decoder side. Therefore, for example, the encoder can transmit (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and other parameters, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select specific parameters. By avoiding the transmission of any actual function, bit saving is achieved in various embodiments. It should be understood that signaling can be achieved in various ways. For example, in various embodiments, one or more grammatical elements, flags, etc. are used to send information to the corresponding decoder with a signal. Although the foregoing relates to the verb form of the word "signal", the word "signal" can also be used as a noun in this article.

[0166] As will be appreciated by those skilled in the art, implementations may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0167] We have described a number of embodiments. Features of these embodiments may be provided individually or in any combination across various claim categories and types. In addition, embodiments may include one or more of the following features, devices, or aspects, alone or in any combination across various claim categories and types:

[0168] Modifying the encoding / decoding of blocks of pictures of a video, which blocks are encoded / decoded in intra prediction mode using linear weighted prediction to reduce the memory used to store the weight matrix;

[0169] obtaining intra-prediction samples of the block from: at least two matrix-vector products between at least two selected weight matrices among the set of weight matrices and associated bias vectors and the associated biases, and a set of neighboring reference samples;

[0170] Based on the linearly weighted intra prediction mode and the block shape, select at least two weight matrices G and H of size m×r and r×n from the set of weight matrices and associated bias vectors T and an associated bias vector b, the first selected weight matrix G has a size of m×r, and the second selected weight matrix G has a size of r×n, where m, n, r are integers, r

[0171] is less than m and r is less than n;

[0172] From the first matrix product H T x, the second matrix product G(H T x) and add the associated offset b to obtain the intra-frame prediction samples of the block The result is Where x is a set of n adjacent reference samples;

[0173] The first and second weight matrices G and H T is obtained from a low-rank processing of a weight matrix M of size m×n, and wherein the estimate of the weight matrix M are the first and second weight matrices G and H T The product of

[0174] The rank r is determined to reduce the complexity of matrix products;

[0175] The rank r is determined in response to the singular value decomposition of the weight matrix M;

[0176] For a block of size width*height, the set x of neighboring reference samples includes all reconstructed samples from the neighboring top line of size width and from the neighboring left line of size height, thus avoiding downsampling of the reference samples;

[0177] For classical intra prediction, for a block of size width*height, the set x of neighboring reference samples includes all reconstructed samples from the neighboring top line of size 2*width and from the neighboring left line of size 2*height;

[0178] Storing the disclosed structured matrices for matrix-based intra prediction;

[0179] • Deriving the intra-prediction samples of the block to be encoded or decoded from the stored structured matrix.

[0180] Increasing the bit depth of the product of the two matrices relative to the bit depth of the intra-prediction samples;

[0181] Increasing the bit depth of the matrix relative to the bit depth of intra prediction samples;

[0182] Enable or disable the processing of structured weight matrices used in matrix-based intra prediction methods in the decoder and / or encoder;

[0183] In signaling, inserting syntax elements that enable a decoder to identify possible weight matrices used in a matrix-based intra prediction method in a decoder and / or encoder;

[0184] In signaling, inserting at least one high-level syntax element representing a set of weight matrices and associated biases into a picture parameter set (PPS) such that all blocks in a frame using linear weighted prediction use the signaled set of weight matrices;

[0185] In signaling, inserting at least one high-level syntax element representing a set of weight matrices and associated biases into a sequence parameter set (SPS) such that all blocks in the sequence using linear weighted prediction use the signaled set of weight matrices;

[0186] A bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0187] A bitstream or signal comprising syntax conveying information generated according to any of the described embodiments.

[0188] • In the signaling, syntax elements are inserted to enable the decoder to apply matrix-based intra prediction in a manner corresponding to that used by the encoder.

[0189] • Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0190] • Creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments.

[0191] A method, process, apparatus, medium storing instructions, medium storing data, or signal according to any of the described embodiments.

[0192] • A TV, set-top box, cell phone, tablet or other electronic device that performs matrix-based intra prediction according to any of the described embodiments.

[0193] • A TV, set-top box, cell phone, tablet or other electronic device that performs matrix-based intra prediction according to any of the described embodiments and displays (eg, using a monitor, screen or other type of display) the resulting image.

[0194] • A TV, set-top box, cellular phone, tablet or other electronic device that selects (eg, using a tuner) a channel to receive a signal comprising an encoded image and performs matrix-based intra prediction according to any of the described embodiments.

[0195] • A TV, set-top box, cell phone, tablet or other electronic device that receives over the air (eg, using an antenna) a signal comprising an encoded image and performs matrix-based intra prediction according to any of the described embodiments.

Claims

1. A method comprising: Encoding a block of a picture of a video in an intra prediction mode using linear weighted prediction, the encoding comprising: - Determine a set x of n adjacent reference samples; - Based on the linearly weighted intra prediction mode and the block shape, at least a first weight matrix G of size m×r, a second weight matrix H of size r×n are selected from a set of weight matrices and associated bias vectors T and the associated bias vector b, the first selected weight matrix G has a size of m×r, and the second selected weight matrix H T has a size of r×n, where m, n, r are integers, r is less than m and r is less than n; - From the first matrix product H T x, the second matrix product G(H T x) and add the associated bias b to obtain the intra-frame prediction sample of the block turn out 2. The method according to claim 1, wherein the first and second weight matrices G and H T is obtained from a low-rank processing of a weight matrix M of size m×n, and wherein the estimate of the weight matrix M are the first and second weight matrices G and H T The product of The method of claim 2 , wherein r is determined to reduce matrix product complexity. The method of claim 2 , wherein r is determined in response to a singular value decomposition (SVD) of the weight matrix M. 5 . The method of claim 1 , wherein for a block of size width*height, the set x of neighboring reference samples comprises all reconstructed samples from a neighboring top line of size width and from a neighboring left line of size height.

6. The method of claim 1, wherein for a block of size width*height, the set x of neighboring reference samples includes all reconstructed samples from a neighboring top line of size 2*width and from a neighboring left line of size 2*height.

7. The method of claim 1, wherein at least one high-level syntax element representing the set of weight matrices and associated biases is signaled in a picture parameter set (PPS) such that all blocks in a frame using linear weighted prediction use the signaled set of weight matrices.

8. The method of claim 1, wherein at least one high-level syntax element representing the set of weight matrices and associated biases is signaled in a sequence parameter set (SPS) such that all blocks in a sequence using linear weighted prediction use the signaled set of weight matrices.

9. The method of claim 1, wherein a bit depth of the first matrix product and the second matrix product is increased relative to a bit depth of intra-prediction samples.

10. A method comprising: Decoding a block of a picture of a video, the block being coded in an intra prediction mode using linear weighted prediction, the decoding comprising: - Determine a set x of n adjacent reference samples; - Based on the linearly weighted intra prediction mode and the block shape, at least a first weight matrix G of size m×r, a second weight matrix H of size r×n are selected from a set of weight matrices and associated bias vectors T and the associated bias vector b, the first selected weight matrix G has a size of m×r, and the second selected weight matrix H T having size r×n, where m, n, r are integers, r is less than m and r is less than n; - from the first matrix product H T x, the second matrix product G(H T x) and add the associated bias b to obtain the intra-frame prediction sample of the block turn out 11. The method of claim 10, wherein the first and second weight matrices G and H T is obtained from a low-rank processing of a weight matrix M of size m×n, and wherein the estimate of the weight matrix M are the first and second weight matrices G and H T The product of 12. The method of claim 10, wherein r is determined to reduce matrix product complexity.

13. The method of claim 10, wherein r is determined in response to a singular value decomposition (SVD) of the weight matrix M.

14. The method of claim 10, wherein for a block of size width*height, the set x of neighboring reference samples includes all reconstructed samples from a neighboring top line of size width and from a neighboring left line of size height.

15. The method of claim 10, wherein for a block of size width*height, the set x of neighboring reference samples includes all reconstructed samples from a neighboring top line of size 2*width and from a neighboring left line of size 2*height.

16. The method of claim 10, wherein at least one high-level syntax element representing the set of weight matrices and associated biases is signaled in a picture parameter set (PPS) such that all blocks in a frame using linear weighted prediction use the signaled set of weight matrices.

17. The method of claim 10, wherein at least one high-level syntax element representing the set of weight matrices and associated biases is signaled in a sequence parameter set (SPS) such that all blocks in a sequence using linear weighted prediction use the signaled set of weight matrices.

18. The method of claim 11, wherein a bit depth of the first matrix product and the second matrix product is increased relative to a bit depth of intra-prediction samples.

19. An apparatus comprising one or more processors and at least one memory, wherein the one or more processors are configured to: Blocks of a picture of a video are encoded in intra prediction mode using linear weighted prediction by following these steps: - Determine a set x of n adjacent reference samples; - Based on the linearly weighted intra prediction mode and the block shape, at least a first weight matrix G of size m×r, a second weight matrix H of size r×n are selected from a set of weight matrices and associated bias vectors T and the associated bias vector b, the first selected weight matrix G has a size of m×r, and the second selected weight matrix H T has a size of r×n, where m, n, r are integers, r is less than m and r is less than n; - From the first matrix product H T x, the second matrix product G(H T x) and add the associated bias b to obtain the intra-frame prediction sample of the block turn out 20. The apparatus of claim 19, wherein the first and second weight matrices G and H T is obtained from a low-rank processing of a weight matrix M of size m×n, and wherein the estimate of the weight matrix M are the first and second weight matrices G and H T The product of 21. The apparatus of claim 19, wherein r is determined to reduce matrix product complexity.

22. The apparatus of claim 19, wherein r is determined in response to a singular value decomposition (SVD) of the weight matrix M.

23. The apparatus of claim 19, wherein for a block of size width*height, the set x of neighboring reference samples includes all reconstructed samples from a neighboring top line of size width and from a neighboring left line of size height.

24. The apparatus of claim 19, wherein for a block of size width*height, the set x of neighboring reference samples includes all reconstructed samples from a neighboring top line of size 2*width and from a neighboring left line of size 2*height.

25. The apparatus of claim 19, wherein at least one high-level syntax element representing the set of weight matrices and associated biases is signaled in a picture parameter set (PPS) such that all blocks in a frame using linear weighted prediction use the signaled set of weight matrices.

26. The apparatus of claim 19, wherein at least one high-level syntax element representing the set of weight matrices and associated biases is signaled in a sequence parameter set (SPS) such that all blocks in a sequence using linear weighted prediction use the signaled set of weight matrices.

27. The apparatus of claim 19, wherein a bit depth of the first matrix product and the second matrix product is increased relative to a bit depth of intra-prediction samples.

28. An apparatus comprising one or more processors and at least one memory, wherein the one or more processors are configured to: A block of a picture of a video, which is coded in intra prediction mode using linear weighted prediction, is decoded by the following steps: - Determine a set x of n adjacent reference samples; - Based on the linearly weighted intra prediction mode and the block shape, at least a first weight matrix G of size m×r, a second weight matrix H of size r×n are selected from a set of weight matrices and associated bias vectors T and the associated bias vector b, the first selected weight matrix G has a size of m×r, and the second selected weight matrix H T has a size of r×n, where m, n, r are integers, r is less than m and r is less than n; - From the first matrix product H T x, the second matrix product G(H T x) and add the associated bias b to obtain the intra-frame prediction sample of the block turn out 29. The apparatus of claim 28, wherein the first and second weight matrices G and H T is obtained from a low-rank processing of a weight matrix M of size m×n, and wherein the estimate of the weight matrix M are the first and second weight matrices G and H T The product of 30. The apparatus of claim 28, wherein r is determined to reduce matrix product complexity.

31. The apparatus of claim 28, wherein r is determined in response to a singular value decomposition (SVD) of the weight matrix M.

32. The apparatus of claim 28, wherein for a block of size width*height, the set x of neighboring reference samples includes all reconstructed samples from a neighboring top line of size width and from a neighboring left line of size height.

33. The apparatus of claim 28, wherein for a block of size width*height, the set x of neighboring reference samples includes all reconstructed samples from a neighboring top line of size 2*width and from a neighboring left line of size 2*height.

34. The apparatus of claim 28, wherein at least one high-level syntax element representing the set of weight matrices and associated biases is signaled in a picture parameter set (PPS) such that all blocks in a frame using linear weighted prediction use the signaled set of weight matrices.

35. The apparatus of claim 28, wherein at least one high-level syntax element representing the set of weight matrices and associated biases is signaled in a sequence parameter set (SPS) such that all blocks in a sequence using linear weighted prediction use the signaled set of weight matrices.

36. The apparatus of claim 28, wherein a bit depth of the first matrix product and the second matrix product is increased relative to a bit depth of intra-prediction samples.

Images