Matrix-based intra prediction

Abstract

The invention relates to matrix-based intra prediction. An apparatus for predicting a predetermined block (18) of an image using a plurality of reference samples (17a, c). The apparatus is configured to form (100) a sample value vector (102, 400) from the plurality of reference samples, derive from the sample value vector a further vector onto which the sample value vector is mapped by a predetermined reversible linear transformation, calculate a matrix vector product between the further vector and a predetermined prediction matrix to obtain a prediction vector, and determine (100) the prediction vector based on the matrix vector product. And predicting a sample of the predetermined block based on the prediction vector.

Description

[0001] This application is a divisional application of the invention patent application with national application number 202080049956.4, the application date of which is May 11, 2020, and the name of the invention is “Matrix-based internal prediction”. Technical Field

[0002] The present application relates to the field of block-based prediction. Embodiments relate to advantageous ways for determining a prediction vector. Background Art

[0003] There are different block-based intra prediction and inter prediction modes today. Samples of neighboring blocks to be predicted or samples obtained from other images can form a sample vector, which can undergo matrix multiplication to determine the prediction signal for the block to be predicted.

[0004] Matrix multiplication should preferably be performed in integer arithmetic, and matrices derived by some machine learning based training algorithm should be used for matrix multiplication.

[0005] However, such training algorithms typically only produce matrices given in floating point precision. Therefore, there is the problem of specifying integer operations so that they are well used to approximate matrix multiplications and / or achieve improvements in computational efficiency and / or make predictions more efficient in implementation. Summary of the invention

[0006] This is achieved through the subject matter of the independent technical solution of this application.

[0007] Further embodiments according to the invention are defined by the subject matter of the dependent claims of the present application.

[0008] According to the first aspect of the present invention, the inventors of the present application realize that one problem encountered when trying to determine a prediction vector by an encoder or a decoder is the possible lack of using integer arithmetic to calculate the prediction vector for a predetermined block. According to the first aspect of the present application, this difficulty is solved by deriving another vector from the sample value vector, the sample value vector is mapped to the other vector by a predetermined reversible linear transformation, so that the sample value vector is not directly applied to the matrix-vector product of the prediction vector. Instead, the matrix-vector product between the other vector and the predetermined prediction matrix is ​​calculated to calculate the prediction vector. For example, the other vector is derived so that the samples of the predetermined block can be predicted by the device using integer arithmetic operations and / or fixed-point arithmetic operations. This is based on the following idea: the components of the sample value vector are related, where a favorable predetermined reversible linear transformation can be used, for example, to obtain another vector with mainly small terms, so that an integer matrix and / or a matrix with fixed-point values ​​and / or a matrix with a small expected quantization error can be used as a predetermined prediction matrix.

[0009] Therefore, according to a first aspect of the present application, a device for predicting a predetermined block of an image using multiple reference samples is configured to form a sample value vector from multiple reference samples. For example, the reference samples are samples of a predetermined block adjacent to an intra prediction or samples in another image at an inter prediction. According to one embodiment, for example, the reference samples can be reduced by averaging to obtain a sample value vector having a reduced number of values. In addition, the device is configured to: derive another vector from the sample value vector, the sample value vector being mapped onto the other vector by a predetermined reversible linear transformation; calculate the matrix-vector product between the other vector and a predetermined prediction matrix to obtain a prediction vector; and predict samples of a predetermined block based on the prediction vector. Based on the other vector, the prediction of the samples of the predetermined block can represent an integer approximation of a direct matrix-vector product between the sample value vector and the matrix to obtain predicted samples of the predetermined block.

[0010] The direct matrix-vector product between the sample value vector and the matrix may be equal to the second matrix-vector product between another vector and the second matrix. The second matrix and / or matrix is, for example, a machine learning prediction matrix. According to one embodiment, the second matrix may be based on a predetermined prediction matrix and an integer matrix. For example, the second matrix is ​​equal to the sum of the predetermined prediction matrix and the integer matrix. In other words, the second matrix-vector product between another vector and the second matrix may be represented by a matrix-vector product between another vector and the predetermined prediction matrix and another matrix-vector product between the integer matrix and the other vector. For example, the integer matrix is ​​a matrix having a predetermined column i0 consisting of one and a column i≠i0 that is zero. Therefore, a good integer approximation and / or a good fixed-point value approximation of the first and / or second matrix-vector product may be achieved by the device. This is based on the idea that the predetermined prediction matrix may be quantized, or a quantized matrix, which is due to the fact that the other vector mainly includes small values, resulting in a slight effect of possible quantization errors in the approximation of the first and / or second matrix-vector product.

[0011] According to one embodiment, the sum of the reversible linear transformation multiplied by the predetermined prediction vector and the integer matrix may correspond to a quantized version of the machine learning prediction matrix. For example, the integer matrix is ​​a matrix having a predetermined column i0 consisting of one and a column i≠i0 that is zero.

[0012] According to an embodiment, the reversible linear transformation is defined such that a predetermined component of the other vector becomes a, and each of the other components of the other vector except the predetermined component is equal to the corresponding component of the sample value vector minus a, where a is a predetermined value. Therefore, another vector with a smaller value can be realized, thereby enabling quantization of the predetermined prediction matrix and producing a marginal impact of quantization errors in predicted samples of the predetermined block. In the case of this other vector, it is possible to predict the samples of the predetermined block by integer arithmetic operations and / or fixed-point arithmetic operations.

[0013] According to an embodiment, the predetermined value is one of an average value (such as an arithmetic mean or a weighted mean value) of the components of the sample value vector, a default value, a value signaled in a data stream obtained by encoding the image, and a component of the sample value vector corresponding to the predetermined component. For example, the sample value vector includes an average value of a plurality of reference samples or groups of reference samples among the plurality of reference samples. For example, a group of reference samples includes at least two reference samples, preferably adjacent reference samples.

[0014] For example, the predetermined value is the arithmetic mean or weighted mean of some components (e.g., at least two components) of the sample value vector or all components of the sample value vector. This is based on the following idea: the components of the sample value vector are related, that is, the values ​​of the components can be similar and / or at least some of the components can have equal values, wherein the component of the other vector is not equal to the predetermined component of the other vector, that is, the component i of i≠i0, where i0 represents the predetermined component, which may have an absolute value smaller than the corresponding component of the sample value vector. Therefore, another vector with a smaller value can be realized.

[0015] The predetermined value may be a default value, wherein the default value is for example selected from a list of default values ​​or is the same for all block sizes, prediction modes, etc. The components of the list of default values ​​may be associated with different block sizes, prediction modes, sample value vector sizes, average values ​​of the values ​​of the sample value vector, etc. Thus, for example, depending on the predetermined block, i.e. depending on the decoding or encoding settings associated with the predetermined block, an optimized default value is selected by the device from the list of default values.

[0016] Alternatively, the predetermined value may be a value signaled in a data stream from which the image is encoded. In this case, for example, the device for encoding determines the predetermined value. The determination of the predetermined value may be based on the same considerations as described above in the context of the default value.

[0017] A component of the other vector is not equal to a predetermined component of the other vector, i.e. a component i where i≠i0, where i0 represents the predetermined component, and the component of the other vector has, for example, an absolute value that is smaller than the corresponding component of the sample value vector, wherein a default value or a value signaled in the data stream is used as the predetermined value.

[0018] According to an embodiment, the predetermined value may be a component of the sample value vector corresponding to the predetermined component. In other words, by applying a reversible linear transformation, the value of the component of the sample value vector corresponding to the predetermined component does not change. Therefore, the value of the component of the sample value vector corresponding to the predetermined component is equal to, for example, the value of the predetermined component of another vector.

[0019] The predetermined component is for example selected by default, as for example described above with respect to the predetermined value as a default value. It is clear that the predetermined component can be selected by an alternative process. The predetermined component is for example selected to be similar to the predetermined value. According to an embodiment, the predetermined component is selected so that the value of the corresponding component of the sample value vector is equal to or has only a slight deviation from the average value of the values ​​of the sample value vector.

[0020] According to an embodiment, the matrix components of the predetermined prediction matrix corresponding to the predetermined components of the other vector in a column of the predetermined prediction matrix are all zero. The device is configured to calculate the matrix-vector product, i.e. the matrix-vector product between the other vector and the predetermined prediction matrix, by performing multiplication by calculating the matrix-vector product between the reduced prediction matrix generated from the predetermined prediction matrix by discarding columns (i.e. the columns consisting of zeros) and another vector generated from the other vector by discarding the predetermined components. This is based on the idea that the predetermined components of the other vector are set to a predetermined value, and if the values ​​of the sample value vector are relevant, this predetermined value is exactly the sample value in the prediction signal for the predetermined block or is close to the sample value. Therefore, the prediction of the samples of the predetermined block is optionally based on the predetermined prediction matrix multiplied by the other vector, or in fact based on the reduced prediction matrix multiplied by the other vector and the integer matrix multiplied by the other vector, the column i0 of the integer matrix corresponding to the predetermined components consisting of one, and all other columns i≠i0 of the integer matrix are zero. In other words, the transformed machine learning prediction matrix, for example by an inverse transformation of a predetermined reversible linear transformation, can be split into a predetermined prediction matrix or in fact a reduced prediction matrix and an integer matrix based on another vector. Therefore, only the prediction matrix should be quantized to obtain an integer approximation of the machine learning prediction matrix and / or the transformed machine learning prediction matrix, which is advantageous because the further vector does not include a predetermined component and all other components have an absolute value much smaller than the corresponding component of the sample value vector, thereby allowing for a slight effect of quantization errors in the resulting quantization of the machine learning prediction matrix and / or the transformed machine learning prediction matrix. In addition, in the case of the reduced prediction matrix and the further vector, almost no multiplication needs to be performed to obtain the prediction vector, thereby reducing complexity and resulting in higher computational efficiency. Optionally, a vector whose components are all predetermined values ​​can be added to the prediction vector when predicting samples of a predetermined block. As described above, this vector can be obtained by a matrix-vector product between an integer matrix and another matrix.

[0021] According to one embodiment, a matrix generated by summing each matrix component of a predetermined prediction matrix corresponding to a predetermined component of another vector within a column of the predetermined prediction matrix with one is multiplied by a predetermined reversible linear transformation corresponding to a quantized version of the machine learning prediction matrix. The summation of each matrix component of the predetermined prediction matrix corresponding to a predetermined component of another vector within a column of the predetermined prediction matrix with one represents, for example, a transformed machine learning prediction matrix. The transformed machine learning prediction matrix represents, for example, a machine learning prediction matrix transformed by an inverse transformation of a predetermined reversible linear transformation. The summation may correspond to the summation of a predetermined prediction matrix and an integer matrix, wherein the column i0 of the integer matrix corresponding to the predetermined component consists of one and all other columns i≠i0 thereof are zero.

[0022] According to an embodiment, the device is configured to represent a predetermined prediction matrix using prediction parameters, and to calculate a matrix-vector product by performing multiplication and summation on the components of another vector and the prediction parameters and the intermediate results generated therefrom. The absolute value of the prediction parameter can be represented by an n-bit fixed-point number representation, where n is equal to or lower than 14, or alternatively equal to or lower than 10, or alternatively equal to or lower than 8. In other words, the prediction parameter is multiplied by the elements of the matrix-vector product (such as another vector, a predetermined prediction matrix and / or a prediction vector) and / or summed with it. By multiplication and summation operations, a fixed-point format of predicted samples of, for example, a predetermined prediction matrix, a prediction vector and / or a predetermined block can be obtained.

[0023] An embodiment according to the present invention relates to a device for encoding an image, comprising a device for predicting a predetermined block of an image using multiple reference samples according to any one of the embodiments described herein to obtain a prediction signal. In addition, the device comprises an entropy encoder configured to encode a prediction residual for the predetermined block for correcting the prediction signal. In order to predict the predetermined block to obtain the prediction signal, the device is configured, for example, to: form a sample value vector from multiple reference samples; derive another vector from the sample value vector, the sample value vector being mapped onto the other vector by a predetermined reversible linear transformation; calculate the matrix-vector product between the other vector and a predetermined prediction matrix to obtain a prediction vector; and predict the sample of the predetermined block based on the prediction vector.

[0024] An embodiment according to the present invention relates to a device for decoding an image, comprising a device for predicting a predetermined block of an image using multiple reference samples according to any one of the embodiments described herein to obtain a prediction signal. In addition, the device comprises: an entropy decoder configured to decode a prediction residual for a predetermined block; and a prediction corrector configured to correct the prediction signal using the prediction residual. In order to predict the predetermined block to obtain the prediction signal, the device is configured, for example, to: form a sample value vector from multiple reference samples; derive another vector from the sample value vector, the sample value vector being mapped onto the other vector by a predetermined reversible linear transformation; calculate the matrix-vector product between the other vector and a predetermined prediction matrix to obtain a prediction vector; and predict samples of the predetermined block based on the prediction vector.

[0025] An embodiment according to the present invention relates to a method for predicting a predetermined block of an image using multiple reference samples, comprising: forming a sample value vector from multiple reference samples; deriving another vector from the sample value vector, wherein the sample value vector is mapped onto the other vector by a predetermined reversible linear transformation; calculating a matrix-vector product between the other vector and a predetermined prediction matrix to obtain a prediction vector; and predicting samples of the predetermined block based on the prediction vector.

[0026] An embodiment according to the present invention relates to a method for encoding an image, comprising predicting a predetermined block of the image using multiple reference samples according to the method described above to obtain a prediction signal, and entropy encoding the prediction residual for the predetermined block to correct the prediction signal.

[0027] An embodiment according to the present invention relates to a method for decoding an image, comprising predicting a predetermined block of the image using multiple reference samples according to one of the methods described above to obtain a prediction signal, entropy decoding a prediction residual for the predetermined block, and correcting the prediction signal using the prediction residual.

[0028] An embodiment according to the present invention relates to a data stream obtained by encoding an image using the method for encoding an image described herein.

[0029] An embodiment according to the present invention relates to a computer program with a program code, which is used to perform the method of any one of the embodiments described herein when the program code is run on a computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are not necessarily drawn to scale, and emphasis is generally placed on illustrating the principles of the present invention. In the following description, various embodiments of the present invention are described with reference to the following drawings, in which:

[0031] Figure 1 An embodiment of encoding into a data stream is shown;

[0032] Figure 2 An embodiment of an encoder is shown;

[0033] Figure 3 An embodiment showing reconstruction of an image;

[0034] Figure 4 An embodiment of a decoder is shown;

[0035] Figure 5 A schematic diagram showing prediction of a block for encoding and / or decoding according to an embodiment;

[0036] Figure 6 Matrix operations for prediction of blocks for encoding and / or decoding according to an embodiment are shown;

[0037] Figure 7.1 shows a prediction of a block with downscaled sample values ​​according to an embodiment;

[0038] Figure 7.2 illustrates prediction of a block using interpolation of samples according to an embodiment;

[0039] Figure 7.3shows a prediction of a block with a reduced sample value vector according to an embodiment, where only some boundary samples are averaged;

[0040] Figure 7.4 shows a prediction of a block with a reduced sample value vector according to an embodiment, wherein groups of four boundary samples are averaged;

[0041] Figure 8 A schematic diagram showing an apparatus for predicting a block according to an embodiment;

[0042] Fig. 9 illustrates a matrix operation performed by a device according to an embodiment;

[0043] Figures 10a to 10c shows detailed matrix operations performed by a device according to an embodiment;

[0044] Fig.11 illustrates detailed matrix operations performed by a device using offset and scaling parameters according to an embodiment;

[0045] Fig.12 shows detailed matrix operations performed by a device using offset and scaling parameters according to various embodiments; and

[0046] Fig.13 A block diagram illustrating a method for predicting a predetermined block according to an embodiment is shown. DETAILED DESCRIPTION

[0047] In the following description, equal or equivalent elements having equal or equivalent functionality are denoted by equal or equivalent reference numerals even if they appear in different drawings.

[0048] In the following description, a number of details are set forth to provide a more thorough explanation of embodiments of the present invention. However, it will be apparent to those skilled in the art that embodiments of the present invention may be practiced without these specific details. In other cases, well-known structures and devices are shown in block diagram form rather than in detail to avoid confusing embodiments of the present invention. In addition, unless otherwise specifically noted, the features of the different embodiments described herein may be combined with one another.

[0049] 1. Introduction

[0050] In the following, different examples, embodiments and aspects of the invention will be described. At least some of these examples, embodiments and aspects particularly refer to methods and / or apparatus for video coding and / or for block-based prediction techniques, for example using linear or affine transformations and neighboring sample reduction, and / or for optimizing video delivery (e.g., broadcasting, streaming, file playback, etc.), for example, for video applications and / or for virtual reality applications.

[0051] Furthermore, the examples, embodiments and aspects may represent High Efficiency Video Coding (HEVC) or a successor.Furthermore, other embodiments, examples and aspects will be defined by the appended claims.

[0052] It should be noted that any embodiment, example and aspect as defined in the claims may be supplemented by any of the details (features and functionalities) described in the following sections.

[0053] Furthermore, the embodiments, examples and aspects described in the following sections may be used individually and may also be supplemented by any of the features in another section or by any of the features included in the claims.

[0054] Furthermore, it should be noted that the individuals, examples, embodiments, and aspects described herein can be used individually or in combination. Therefore, details can be added to each of the various aspects without adding details to another of the examples, embodiments, and aspects.

[0055] It should also be noted that the present disclosure explicitly or implicitly describes features of decoding and / or encoding systems and / or methods.

[0056] In addition, the features and functionality disclosed herein with respect to a method may also be used in an apparatus. In addition, any features and functionality disclosed herein with respect to an apparatus may also be used in a corresponding method. In other words, the method disclosed herein may be supplemented by any of the features and functionality described with respect to the apparatus.

[0057] Furthermore, any of the features and functionalities described herein may be implemented in hardware or software, or using a combination of hardware and software, as will be described in the section “Implementation Alternatives”.

[0058] Furthermore, in some examples, embodiments or aspects, any of the features described in brackets ("(...)" or "[...]") may be considered optional.

[0059] 2. Encoder, Decoder

[0060] In the following, various examples are described that can assist in achieving more efficient compression when using block-based prediction techniques. Some examples achieve high compression efficiency by using a set of intra prediction modes. The latter examples can be added to other intra prediction modes designed, for example, exploratory, or can be provided specifically. And even other examples simultaneously utilize both of the special cases just discussed. However, as a variation of these embodiments, the intra prediction can be changed to inter prediction by using reference samples in another image.

[0061] For easy understanding of the following examples of the present application, the present description starts by presenting a possible encoder and a decoder suitable therefor in which the subsequently outlined examples of the present application may be constructed. Figure 1 An apparatus is shown for encoding an image 10 block by block into a data stream 12. The apparatus is indicated using reference numeral 14 and may be a still image encoder or a video encoder. In other words, when the encoder 14 is configured to encode a video 16 including the image 10 into the data stream 12, or the encoder 14 may exclusively encode the image 10 into the data stream 12, the image 10 may be the current image in the video 16.

[0062] As mentioned, the encoder 14 performs encoding in a block-by-block manner or on a block basis. To this end, the encoder 14 subdivides the image 10 into blocks, and the units of the encoder 14 encode the image 10 into a data stream 12. Examples of possible subdivisions of the image 10 into blocks 18 are explained in more detail below. In general, the subdivision may end up with blocks 18 of constant size, such as an array of blocks configured in rows and columns, such as by using a hierarchical multi-tree subdivision, or may end up with blocks 18 of different block sizes, wherein the multi-tree subdivision starts with the entire image area of ​​the image 10 or starts with the image 10 pre-divided into an array of tree-shaped blocks, wherein these examples should not be regarded as excluding other possible ways of subdividing the image 10 into blocks 18.

[0063] Furthermore, the encoder 14 is a predictive encoder configured to predictively encode the image 10 into the data stream 12. For a particular block 18, this means that the encoder 14 determines a prediction signal for the block 18 and encodes a prediction residual (i.e., a prediction error of the prediction signal from the actual image content within the block 18) into the data stream 12.

[0064] The encoder 14 may support different prediction modes in order to derive a prediction signal for a particular block 18. The prediction mode of importance in the following example is the intra prediction mode, according to which the interior of the block 18 is spatially predicted based on neighboring already encoded samples of the image 10. The encoding of the image 10 into the data stream 12 and thus the corresponding decoding process may be based on a specific coding order 20 defined within the block 18. For example, the coding order 20 may traverse the block 18 in a raster scan order (such as row by row from top to bottom), where each row is traversed from left to right, for example. In the case of a hierarchical multi-tree based subdivision, a raster scan order may be applied within each hierarchical level, where a depth-first traversal order may be applied, i.e., leaf nodes within a block of a particular hierarchical level may precede blocks of the same hierarchical level having the same parent block according to the coding order 20. Depending on the coding order 20, the neighboring already encoded samples of the block 18 may generally be located at one or more sides of the block 18. In the case of the examples presented herein, the neighboring already encoded samples of block 18 , for example, are positioned at the top and to the left of block 18 .

[0065] The intra-prediction mode may not be the only mode supported by the encoder 14. In the case where the encoder 14 is a video encoder, for example, the encoder 14 may also support an inter-prediction mode according to which the block 18 is temporally predicted from previously encoded pictures of the video 16. The inter-prediction mode may be a motion compensated prediction mode according to which a motion vector is signaled to this block 18, indicating the relative spatial offset of the portion from which the prediction signal for the block 18 is to be derived as a copy. Additionally or alternatively, other non-intra-prediction modes may also be available, such as an inter-prediction mode in the case where the encoder 14 is a multi-view encoder, or a non-predictive mode according to which the interior of the block 18 is encoded as is, i.e. without any prediction.

[0066] Before beginning to focus the description of the present application on the intra-prediction mode, a more specific example for a possible block-based encoder (i.e., a possible implementation for encoder 14) is described as follows: Figure 2 described, wherein suitable Figure 1 and Figure 2 Two corresponding examples of the decoder.

[0067] Figure 2 Show Figure 1 A possible implementation of the encoder 14 is one in which the encoder is configured to use transform coding for encoding the prediction residual, but this is an approximate example and the present application is not limited to this type of prediction residual coding. Figure 2, the encoder 14 comprises a subtractor 22 configured to subtract the corresponding prediction signal 24 from the inbound signal (i.e. the image 10, or the current block 18 when on a block basis) to obtain a prediction residual signal 26, which is then encoded by a prediction residual encoder 28 into the data stream 12. The prediction residual encoder 28 consists of a lossy encoding stage 28a and a lossless encoding stage 28b. The lossy stage 28a receives the prediction residual signal 26 and comprises a quantizer 30, which quantizes the samples of the prediction residual signal 26. As already mentioned above, the example of the invention uses transform coding of the prediction residual signal 26, and therefore the lossy encoding stage 28a comprises a transform stage 32 connected between the subtractor 22 and the quantizer 30 in order to transform this spectrally decomposed prediction residual 26, and to perform quantization of the transformed coefficients representing the residual signal 26 by the quantizer 30. The transform may be a DCT, a DST, a FFT, a Hadamard transform, or the like. The transformed and quantized prediction residual signal 34 is then losslessly encoded by a lossless encoding stage 28b, which is an entropy encoder that entropy encodes the quantized prediction residual signal 34 into the data stream 12. The encoder 14 further comprises a prediction residual signal reconstruction stage 36, which is connected to the output of the quantizer 30 in order to reconstruct the prediction residual signal from the transformed and quantized prediction residual signal 34 in a manner that can also be used at the decoder, that is, taking into account the coding losses of the quantizer 30. For this purpose, the prediction residual reconstruction stage 36 comprises an inverse quantizer 38, which performs the inverse operation of the quantization of the quantizer 30, and then an inverse transformer 40, which performs an inverse transformation with respect to the transformation performed by the transformer 32, such as the inverse operation of the spectral decomposition, such as the inverse operation of any of the specific transformation examples mentioned above. The encoder 14 comprises an adder 42, which adds the reconstructed prediction residual signal output by the inverse transformer 40 and the prediction signal 24 in order to output a reconstructed signal, that is, reconstructed samples. This output is fed into the predictor 44 of the encoder 14, which then determines the prediction signal 24 based on the output. The predictor 44 supports the Figure 1 All prediction models discussed. Figure 2 It is also illustrated that in the case where the encoder 14 is a video encoder, the encoder 14 may also include an in-loop filter 46 which completely filters the reconstructed image which, after having been filtered, forms a reference image for the predictor 44 relative to the inter-predicted blocks.

[0068] As already mentioned above, the encoder 14 operates on a block basis. For the subsequent description, the block basis of interest is the basis of subdividing the image 10 into blocks, selecting an intra prediction mode for the blocks from a set or multiple intra prediction modes supported by the predictor 44 or the encoder 14, respectively, and executing the selected intra prediction mode separately. However, there may also be blocks that subdivide the image 10 into other categories. For example, whether the image 10 is inter-coded or intra-coded, the above-mentioned decisions can be made at the granularity or unit of blocks that deviate from the block 18. For example, the inter / intra mode decision can be performed at the level of the coding blocks into which the image 10 is subdivided, and each coding block is subdivided into prediction blocks. Each of the prediction blocks of the coding blocks for which it has been decided to use intra prediction is subdivided into an intra prediction mode decision. To this end, for each of these prediction blocks, it is decided which supported intra prediction mode should be used for the corresponding prediction block. These prediction blocks will form the block 18 of interest here. The prediction blocks within the coding blocks associated with inter prediction will be processed differently by the predictor 44. The prediction block will be inter-predicted from the reference picture by determining a motion vector and copying the prediction signal for this block from the position pointed to by the motion vector in the reference picture. Another block subdivision is the subdivision into transform blocks, wherein the transform is performed in units of the transform blocks by the transformer 32 and the inverse transformer 40. The transformed block may, for example, be the result of a further subdivision of the coding block. Of course, the examples set forth herein should not be considered restrictive, and other examples also exist. Just for the sake of completeness, it should be noted that the subdivision into coding blocks may, for example, use a multi-tree subdivision, and that the prediction blocks and / or transform blocks may also be obtained by further subdividing the coding blocks using a multi-tree subdivision.

[0069] Figure 3 Suitable for depiction Figure 1 A decoder 54 or device for block-by-block decoding of the encoder 14. This decoder 54 is the opposite of the encoder 14, that is, it decodes the image 10 from the data stream 12 in a block manner and supports multiple intra-prediction modes for this purpose. For example, the decoder 54 may include a residual provider 156. Figure 1All other possibilities discussed are also valid for the decoder 54. For this purpose, the decoder 54 can be a still image decoder or a video decoder and all prediction modes and prediction possibilities are also supported by the decoder 54. The difference between the encoder 14 and the decoder 54 mainly lies in the fact that the encoder 14 selects or chooses the encoding decision according to some optimization, such as, for example, in order to minimize some cost function that can depend on the encoding rate and / or the encoding distortion. One of these encoding options or encoding parameters can involve a series of intra-prediction modes to be used for the current block 18 among the available or supported intra-prediction modes. The selected intra-prediction mode can then be signaled to the current block 18 by the encoder 14 in the data stream 12, wherein the decoder 54 reselects using this signaling in the data stream 12 for the block 18. Similarly, the subdivision of the image 10 into blocks 18 can be optimized in the encoder 14, and the corresponding subdivision information can be transmitted in the data stream 12, wherein the decoder 54 recovers the subdivision of the image 10 into blocks 18 based on the subdivision information. In summary, the decoder 54 may be a predictive decoder operating on a block basis, and in addition to the intra prediction mode, the decoder 54 may support other prediction modes, such as the inter prediction mode in the case where the decoder 54 is a video decoder. Figure 1 The coding order 20 discussed is discussed, and since both the encoder 14 and the decoder 54 comply with this coding order 20, the same neighboring samples are available for the current block 18 at the encoder 14 and the decoder 54. Therefore, in order to avoid unnecessary repetition, the description of the mode of operation of the encoder 14, as far as the subdivision of the image 10 into blocks is concerned, for example as far as the prediction is concerned and as far as the coding of the prediction residual is concerned, should also apply to the decoder 54. The difference lies in the fact that the encoder 14 selects some coding options or coding parameters by optimization and signals the coding parameters within the data stream 12 or inserts them into the data stream 12, which are then derived from the data stream 12 by the decoder 54 for re-prediction, subdivision, etc.

[0070] Figure 4 Show Figure 3 A possible implementation of the decoder 54 is suitable for Figure 2 As shown in Figure 1 The implementation of the encoder 14 is implemented. Figure 4 Many components of the encoder 54 are related to Figure 2 The same elements appear in the corresponding encoder, so Figure 4 The same reference numerals with primes are used in order to indicate these elements. Specifically, the adder 42', the optional in-loop filter 46' and the predictor 44' are shown in the same manner as in Figure 2is connected to the prediction loop in the same way as in the encoder of the present invention. The reconstructed, i.e. dequantized and retransformed prediction residual signal applied to the adder 42' is derived via a sequence of the following components: an entropy decoder 56, which performs the inverse operation of the entropy coding of the entropy encoder 28b; followed by a residual signal reconstruction stage 36', which consists of an inverse quantizer 38' and an inverse transformer 40', as is the case on the encoding side. The output of the decoder is a reconstruction of the image 10. The reconstruction of the image 10 can be obtained directly at the output of the adder 42' or alternatively at the output of the in-loop filter 46'. Some post-filters can be arranged at the output of the decoder in order to perform some post-filtering on the reconstruction of the image 10 in order to improve the image quality, but Figure 4 This option is not depicted in .

[0071] Again, about Figure 4 , the above about Figure 2 The description proposed for Figure 4 should also be valid, except that only the encoder performs the optimization tasks and the associated decisions about coding options. However, all descriptions of block subdivision, prediction, inverse quantization and retransformation are for Figure 4 The decoder 54 is also effective.

[0072] 3. ALWIP (Affine Linear Weighted Intra-Predictor)

[0073] Some non-limiting examples regarding ALWIP are discussed herewith, even though ALWIP is not always necessary to specifically implement the techniques discussed herein.

[0074] The present application relates in particular to an improved block-based prediction mode concept for block-by-block image coding, such as may be used in a video codec such as HEVC or any successor to HEVC. The prediction mode may be an intra-prediction mode, but in theory the concepts described herein may also be transferred to an inter-prediction mode, where the reference sample is part of another image.

[0075] A block-based prediction concept is sought that allows for an efficient implementation such as a hardware-friendly implementation.

[0076] This object is achieved through the subject matter of the independent technical solution of this application.

[0077] Intra-prediction mode is widely used in image and video coding. In video coding, intra-prediction mode competes with other prediction modes such as inter-prediction mode, such as motion compensated prediction mode. In intra-prediction mode, the current block is predicted based on neighboring samples, that is, samples that have been encoded on the encoder side and decoded on the decoder side. The neighboring sample values ​​are extrapolated into the current block in order to form a prediction signal for the current block, where the prediction residual is transmitted in the data stream for the current block. The better the prediction signal, the lower the prediction residual, and correspondingly a lower number of bits are required to encode the prediction residual.

[0078] To be effective, several aspects should be considered in order to form an efficient framework for intra prediction in a block-by-block image coding environment. For example, the larger the number of intra prediction modes supported by the codec, the larger the side information rate consumption in order to signal the selection to the decoder. On the other hand, the set of supported intra prediction modes should be able to provide a good prediction signal, i.e. a prediction signal that results in a low prediction residual.

[0079] In the following, as a comparative embodiment or basic example, a device (encoder or decoder) for decoding an image block by block from a data stream is disclosed, wherein the device supports at least one intra prediction mode, according to which an intra prediction signal for a block of a predetermined size of the image is determined by applying a first template of samples adjacent to a current block to an affine linear predictor, which will be referred to as an affine linear weighted intra predictor (ALWIP) in the following.

[0080] The apparatus may have at least one of the following properties (which may be applicable, for example, to a method or another technology implemented in a non-transitory storage unit storing instructions that, when executed by a processor, cause the processor to implement the method and / or function as the apparatus):

[0081] 3.1 Predictors can complement other predictors

[0082] The intra prediction modes that may be formed to further describe the subject matter of the implementation improvements below may be complementary to other intra prediction modes of the codec. Therefore, these intra prediction modes may be complementary to the DC prediction mode, the plane prediction mode, and the angle prediction mode defined in the HEVC codec, respectively, in the JEM reference software. The latter three types of intra prediction modes will be referred to as traditional intra prediction modes in the future. Therefore, for a given block in an intra mode, a flag needs to be parsed by the decoder to indicate that one of the intra prediction modes supported by the device will be used or not.

[0083] 3.2 More than one proposed prediction model

[0084] The device may contain more than one ALWIP mode. Therefore, in the case where the decoder knows that one of the ALWIP modes supported by the device will be used, the decoder needs to parse additional information indicating which of the ALWIP modes supported by the device will be used.

[0085] The signaling of supported modes may have the property that the encoding of some ALWIP modes may require fewer bins than other ALWIP modes. Which of these modes requires fewer bins and which requires more bins may depend on information that can be extracted from the decoded bitstream, or may be fixed in advance.

[0086] 4. Some aspects

[0087] Figure 2 A decoder 54 is shown for decoding an image from a data stream 12. The decoder 54 may be configured to decode a predetermined block 18 of an image. Specifically, the predictor 44 may be configured to map a set of P neighboring samples adjacent to the predetermined block 18 to a set of Q predicted values ​​for samples of the predetermined block using a linear or affine linear transform [e.g., ALWIP].

[0088] like Figure 5 As shown in , a predetermined block 18 includes Q values ​​to be predicted (which will be "predicted values" at the end of the operation). If block 18 has M rows and N columns, then Q=M·N. The Q values ​​of block 18 can be in the spatial domain (e.g., pixels) or in the transform domain (e.g., DCT, discrete wavelet transform, etc.). The Q values ​​of block 18 can be predicted based on P values ​​obtained from neighboring blocks 17a to 17c that are generally adjacent to block 18. The P values ​​of neighboring blocks 17a to 17c can be in the position closest to (e.g., adjacent to) block 18. The P values ​​of neighboring blocks 17a to 17c have been processed and predicted. The P values ​​are indicated as values ​​in portions 17'a to 17'c to distinguish the portions from the blocks to which they belong (in some examples, 17'b is not used).

[0089] like Figure 6As shown in , in order to perform the prediction, it is possible to operate with a first vector 17P having P entries (each associated with a specific position in the neighboring parts 17'a to 17'c), a second vector 18Q having Q entries (each associated with a specific position in the block 18) and a mapping matrix 17M (each row is associated with a specific position in the block 18, and each column is associated with a specific position in the neighboring parts 17'a to 17'c). Therefore, the mapping matrix 17M performs the prediction of the P values ​​of the neighboring parts 17'a to 17'c into the values ​​of the block 18 according to a predetermined pattern. The entries in the mapping matrix 17M can therefore be understood as weighting factors. In the following paragraphs, we will use the symbols 17a to 17c instead of 17'a to 17'c to represent the neighboring parts of the boundary.

[0090] In the art, several conventional modes are known, such as DC mode, planar mode, and 65 directional prediction modes. For example, 67 modes may be known.

[0091] However, it is noted that it is also possible to use a different mode, which is referred to here as a linear or affine linear transformation. A linear or affine linear transformation comprises P·Q weighting factors, among which at least 1 / 4P·Q weighting factors are non-zero weight values, which for each of the Q predicted values, include a series of P weighting factors associated with the corresponding predicted value. The series, when arranged one by one according to the raster scan order among the samples of the predetermined block, forms an omnidirectional nonlinear envelope.

[0092] It is possible to map P positions of neighboring values ​​17'a to 17'c (templates), Q positions of neighboring samples 17'a to 17'c, and to map at the values ​​of P*Q weighting factors of the matrix 17M. A plane is an example of an envelope of a series for DC transformation (it is a plane for DC transformation). The envelope is obviously planar and is therefore excluded by the definition of linear or affine linear transformation (ALWIP). Another example is a matrix that generates a simulation of an angular pattern: the envelope will be excluded from the ALWIP definition and, to put it bluntly, will look like a hill that slopes from top to bottom along a direction in the P / Q plane. The planar mode and the 65 directional prediction modes will have different envelopes, however, they will be linear in at least one direction, that is, for example, all directions for the illustrated DC and for example, the hill direction for the angular pattern.

[0093] In contrast, the envelope of a linear or affine transformation will not be omnidirectionally linear. It is understood that in some cases such a transformation may be optimal for performing the prediction of block 18. It is noted that preferably at least 1 / 4 is different from zero (ie, at least 25% of the P*Q weighting factors are different from 0).

[0094] According to any conventional mapping rule, the weighting factors may be unrelated to each other. Therefore, the matrix 17M may be such that the values ​​of its items have no clearly identifiable relationship. For example, the weighting factors cannot be described by any analytical or differential function.

[0095] In an example, the ALWIP transformation is such that the mean of the maximum values ​​of the cross-correlations between the first series of weighting factors for the corresponding predicted values ​​and the second series of weighting factors for the predicted values ​​other than the corresponding predicted values ​​or the inverted versions of the latter series - whichever results in a higher maximum value - can be below a predetermined threshold (e.g., 0.2 or 0.3 or 0.35 or 0.1, for example, a threshold in the range between 0.05 and 0.035). For example, for each pair of rows (i1, i2) of the ALWIP matrix 17M, the cross-correlations can be calculated by multiplying the P values ​​of the i1-th row by the P values ​​of the i2-th row. For each obtained cross-correlation, the maximum value can be obtained. Therefore, the mean (average value) can be obtained for the entire matrix 17M (that is, the maximum values ​​of the cross-correlations in all combinations are averaged). Thereafter, the threshold can be, for example, 0.2 or 0.3 or 0.35 or 0.1, for example, a threshold in the range between 0.05 and 0.035.

[0096] The P adjacent samples of blocks 17a to 17c may be located along a one-dimensional path extending along a boundary (e.g., 18c, 18a) of a predetermined block 18. For each of the Q predicted values ​​of the predetermined block 18, the series of P weighting factors associated with the corresponding predicted value may be ordered in a manner that traverses the one-dimensional path in a predetermined direction (e.g., from left to right, from top to bottom, etc.).

[0097] In an example, the ALWIP matrix 17M may be non-diagonal or non-block diagonal.

[0098] An example of an ALWIP matrix 17M for predicting a 4×4 block 18 from 4 predicted neighboring samples may be:

[0099] {

[0100] {37, 59, 77, 28},

[0101] {32, 92, 85, 25},

[0102] {31, 69, 100, 24},

[0103] {33, 36, 106, 29},

[0104] {24, 49, 104, 48},

[0105] {24, 21, 94, 59},

[0106] {29, 0, 80, 72},

[0107] {35, 2, 66, 84},

[0108] {32, 13, 35, 99},

[0109] {39, 11, 34, 103},

[0110] {45, 21, 34, 106},

[0111] {51, 24, 40, 105},

[0112] {50, 28, 43, 101},

[0113] {56, 32, 49, 101},

[0114] {61, 31, 53, 102},

[0115] {61, 32, 54, 100}

[0116] }.

[0117] (Here, {37, 59, 77, 28} is the first row of matrix 17M; {32, 92, 85, 25} is the second row; and {61, 32, 54, 100} is the 16th row). Matrix 17M has a size of 16×4 and includes 64 weighting factors (since 16*4=64). This is because matrix 17M has a size of Q×P, where Q=M*N, which is the number of samples of block 18 to be predicted (block 18 is a 4×4 block), and P is the number of samples of predicted samples. Here, M=4, N=4, Q=16 (since M*N=4*4=16), and P=4. The matrix is ​​non-diagonal and non-block diagonal, and is not described by a particular rule.

[0118] As can be seen, less than 1 The weighting factor of / 4 is 0 (in the case of the matrix shown above, one weighting factor out of sixty-four is zero). The envelope formed by these values, when arranged one after another according to the raster scan order, forms an omnidirectional nonlinear envelope.

[0119] Even though the above interpretation is primarily discussed with reference to a decoder (eg, decoder 54), the interpretation may be performed at an encoder (eg, encoder 14).

[0120] In some examples, for each block size (in the set of block sizes), the ALWIP transforms of the intra-prediction modes within the second set of intra-prediction modes for the corresponding block size are different from each other. Additionally or alternatively, the cardinality of the second set of intra-prediction modes for the block sizes in the set of block sizes may be consistent, but the associated linear or affine linear transforms of the intra-prediction modes within the second set of intra-prediction modes for different block sizes may not be convertible into each other by scaling.

[0121] In some examples, an ALWIP transform may be defined in such a way that it shares “nothing” with a legacy transform (eg, an ALWIP transform may share “nothing” with a corresponding legacy transform even if the transform has been mapped by one of the above mappings).

[0122] In an example, the ALWIP mode is used for both the luma component and the chroma components, but in other examples, the ALWIP mode is used for the luma component but not for the chroma components.

[0123] 5. Affine linear weighted intra prediction mode with encoder acceleration (e.g., test CE3-1.2.1)

[0124] 5.1 Description of method or device

[0125] The affine linear weighted intra prediction (ALWIP) mode tested in CE3-1.2.1 may be the same as the mode proposed in JVET-L0199 under test CE3-2.2.2, except for the following changes:

[0126] • Coordination with Multiple Reference Line (MRL) intra prediction (especially encoder estimation and signaling), ie MRL is not combined with ALWIP and transmission of MRL index is restricted to non-ALWIP blocks.

[0127] - All blocks with WxH ≥ 32x32 must now be subsampled (previously optional for 32x32); therefore, the extra test to send the subsampled flag at the encoder has been removed.

[0128] • ALWIP for 64xN and Nx64 blocks (where N≤32) has been added by downsampling to 32xN and Nx32 respectively and applying the corresponding ALWIP modes.

[0129] Additionally, test CE3-1.2.1 includes the following encoder optimizations for ALWIP:

[0130] • Combined mode estimation: Conventional and ALWIP modes use a shared Hadamard candidate list for all RD estimations, ie ALWIP mode candidates are added to the same list as conventional (and MRL) mode candidates based on Hadamard cost.

[0131] Support for fast within EMT and fast within PB for combined pattern lists, with additional optimizations to reduce the number of total RD checks.

[0132] Following the same approach as the conventional mode, only the available MPMs of the left and above blocks are added to the list for full RD estimation of ALWIP.

[0133] 5.2 Complexity Assessment

[0134] In test CE3-1.2.1, excluding the computation of the discrete cosine transform, up to 12 multiplications are required per sample to generate the prediction signal. In addition, a total of 136492 parameters of 16 bits per parameter are required. This corresponds to 0.273 Mbytes of memory.

[0135] 5.3 Experimental Results

[0136] The evaluation was performed on intra-only (AI) and random-access (RA) configurations with VTM software version 3.0.1 according to the common test conditions JVET-J1010 [2]. The corresponding simulations were performed on an Intel Xeon cluster (E5-2697Av4, AVX2 enabled, Intel turbo boost disabled) with Linux OS and GCC 7.2.1 compiler.

[0137] Table 1. Results for CE3-1.2.1 configured for VTMAI

[0138]

[0139]

[0140] Table 2. Results for CE3-1.2.1 configured for VTM RA

[0141] Y U V Coding Time Decoding time Category A1 -1,25％ -1,80％ -1,95％ 113％ 100％ Category A2 -0,68％ -0,54％ -0,21％ 111％ 100％ Category B -0,82％ -0,72％ -0,97％ 113％ 100％ Category C -0,70％ -0,79％ -0,82％ 113％ 99％ Category E total -0,85％ -0,92％ -0,98％ 113％ 100％ Category D -0,65％ -1,06％ -0,51％ 113％ 102％ Category F (optional) -1,07％ -1,04％ -0,96％ 117％ 99％

[0142] 5.4 Affine Linear Weighted Intra-Prediction with Reduced Complexity (e.g. Test CE3-1.2.2)

[0143] The technique tested in CE2 is related to the “affine linear intra prediction” described in JVET-L0199 [1], but simplifies the prediction in terms of memory requirements and computational complexity:

[0144] There may be only three different sets of prediction matrices (e.g. S0, S1, S2, see also below) and bias vectors (e.g. for providing offset values) covering all block shapes. Thus, the number of parameters is reduced to 14400 10-bit values, which is less memory than is stored in a 128x128 CTU.

[0145] The input and output sizes of the predictor are further reduced. In addition, instead of transforming the boundaries via DCT, averaging or downsampling can be performed on the boundary samples, and the generation of the prediction signal can use linear interpolation instead of inverse DCT. Therefore, to generate the prediction signal, up to four multiplications may be required per sample.

[0146] 6. Examples

[0147] Here we discuss how to use ALWIP prediction to perform some predictions (e.g. Figure 6 ).

[0148] In principle, reference Figure 6 , in order to obtain Q=M*N values ​​of the M×N block 18 to be predicted, a multiplication of Q*P samples of the Q×PALWIP prediction matrix 17M by P samples of the P×1 neighboring vector 17P should be performed. Therefore, in general, in order to obtain each of the Q=M*N values ​​of the M×N block 18 to be predicted, at least P=M+N value multiplications are required.

[0149] These multiplications have extremely undesirable effects. The size P of the boundary vector 17P generally depends on the number M+N of boundary samples (bit cells or pixels) 17a, 17c that are adjacent to (e.g., neighboring) the M×N block 18 to be predicted. This means that if the size of the block 18 to be predicted is larger, the number M+N of boundary pixels (17a, 17c) is correspondingly larger, thus increasing the size P=M+N of the P×1 boundary vector 17P and the length of each row of the Q×PALWIP prediction matrix 17M, and correspondingly also the number of necessary multipliers (in general, Q=M*N=W*H, where the width (W) is another sign of N and the height (H) is another sign of M; in the case where the boundary vector is formed by only one row and / or one column of samples, P is P=M+N=H+W).

[0150] This problem is usually exacerbated by the fact that in microprocessor-based systems (or other digital processing systems), multiplication is usually a power-consuming operation. As you can imagine, a large number of multiplications for a very large number of samples in a large number of blocks will result in a waste of computational power, which is usually undesirable.

[0151] Therefore, it is preferable to reduce the number of multiplications Q*P necessary to predict the M×N block 18.

[0152] It will be appreciated that it is possible to reduce to some extent the computational power necessary for each intra prediction of each block 18 to be predicted by intelligently selecting an operation that replaces the multiplication and is easier to process.

[0153] Specifically, refer to Figure 7.1 to Figure 7.4 It is understood that the encoder or decoder can use multiple adjacent samples (such as 17a, 17c) to predict a predetermined block (such as 18) of the picture by the following operations:

[0154] reducing (e.g. at step 811) (e.g. by averaging or downsampling) a plurality of adjacent samples (e.g. 17a, 17c) to obtain a reduced set of sample values ​​having a lower number of samples than the plurality of adjacent samples,

[0155] The downscaled set of sample values ​​is subjected to (eg, at step 812) a linear or affine linear transformation to obtain predicted values ​​for predetermined samples of a predetermined block.

[0156] In some cases, the decoder or encoder may also derive predicted values ​​for other samples of the predetermined block based on the predicted values ​​for the predetermined sample and a plurality of neighboring samples, for example by interpolation.Thus, an upsampling strategy may be obtained.

[0157] In the example, it is possible to perform (eg, at step 811) some averaging on the samples of the boundary 17 to obtain a reduced sample set 102 ( Figure 7.1 to Figure 7.4 ) (At least one of the samples of the reduced number of samples 102 may be an original boundary sample or an average of two samples in a series of original boundary samples). For example, if the original boundary has P=M+N samples, the reduced sample set may have P red =M red +N red , which has M red <M and N red < at least one of N, such that P red <P. Therefore, the boundary vector 17P that will actually be used for prediction (e.g., at step 812b) will not have P×1 entries, but will have P red ×1 item, where P red <P. Similarly, the ALWIP prediction matrix 17M selected for prediction will not have a size of Q×P, but will have a size of Q×P red (or Q red ×P red , see below), which has a reduced number of elements of the matrix, at least because P red <P(with the help of M red <M and N red<at least one of N).

[0158] In some examples (e.g. Figure 7.2 , Figure 7.3 ), if the block obtained by ALWIP (at step 812) is of size M′ red ×N′ red (where M′ red <M and / or N' red <N), it is even possible to further reduce the number of multiplications (i.e. the samples directly predicted by ALWIP are fewer in number than the samples of block 18 that are actually predicted). red =M′ red *N′ red , which will be achieved by not using Q*P red multiplication and use Q red *P red multiplications (where Q red *P red <Q*P red <Q*P) to obtain ALWIP prediction. This multiplication will predict the reduced block, where the size M′ red ×N′ red Nevertheless, it will be possible to perform (for example at subsequent step 813) a reduction from the reduced M′ red ×N′ red Upsampling (eg, obtained by interpolation) of the predicted block into the final M×N predicted block.

[0159] These techniques may be advantageous because, although matrix multiplication involves a reduced number of multiplications (Q red *P red or Q*P red ), but both the initial reduction (e.g., averaging or downsampling) and the final transformation (e.g., interpolation) can be performed by reducing (or even avoiding) multiplications. For example, downsampling, averaging and / or interpolation can be performed (e.g., at steps 811 and / or 813) by employing computationally inefficient binary operations (such as addition and shifting).

[0160] Furthermore, the addition is an extremely easy operation that can be easily performed without requiring a great deal of computational effort.

[0161] This shift operation can be used, for example, to average two boundary samples and / or to interpolate two samples (support values) of a downscaled predicted block (or taken from the boundary) to obtain the final predicted block. (For interpolation, two sample values ​​are necessary. Within the block, we always have two predetermined values, but for interpolating samples along the left and upper boundaries of the block, we only have one predetermined value, as Figure 7.2Therefore, we use boundary samples as support values ​​for interpolation. )

[0162] A two-step process may be used, such as:

[0163] First, sum the values ​​of the two samples;

[0164] The value of the sum is then halved (eg, by shifting right).

[0165] Alternatively, it is possible to:

[0166] First, each of the samples is halved (e.g., by shifting left);

[0167] The values ​​of the two halved samples are then summed.

[0168] When downsampling (eg, at step 811 ), an even easier operation may be performed because only one sample size needs to be selected among a group of samples (eg, samples that are adjacent to each other).

[0169] Therefore, it is now possible to define techniques for reducing the number of multiplications to be performed. Some of these techniques may be based on at least one of the following principles:

[0170] Even though in reality the predicted block 18 has a size M×N, the block may be reduced (in at least one of the two dimensions) and a method having a reduced size Q may be applied. red ×P red The ALWIP matrix (where Q red =M′ red *N′ red , P red =N red +M red , where M′ red <M and / or N' red <N and / or M red <M and / or N red <N). Therefore, the boundary vector 17P will have a size P red ×1, only means P red <P multiplication (where P red =M red +N red And P = M + N).

[0171] P red The ×1 boundary vector 17P can be easily obtained from the original boundary 17, for example:

[0172] By downsampling (e.g. by selecting only some samples at the border); and / or

[0173] By averaging multiple samples of the boundary (the boundary can be easily obtained by addition and shifting without using multiplication).

[0174] Additionally or alternatively, instead of predicting all Q=M*N values ​​of the block 18 to be predicted by multiplication, it is possible to predict only the reduced block (e.g. Q red =M′ red *N′ red , where M′ red <M and / or N' red <N). By interpolation, for example, using Q red samples as the remaining QQ to be predicted red The remaining samples of the block 18 to be predicted are obtained by using the support values ​​of the values.

[0175] according to Figure 7.1 In the example illustrated in , a 4×4 block 18 (M=4, N=4, Q=M*N=16) is to be predicted, and a neighborhood 17 (neighborhoods 17a and 17c may be collectively indicated by 17) of samples 17a (a vertical row with four already predicted samples) and 17c (a horizontal row with four already predicted samples) has already been predicted at a previous iteration. A priori, by using Figure 5 , the prediction matrix 17M should be a Q×P=16×8 matrix (by virtue of Q=M*N=4*4 and P=M+N=4+4=8), and the boundary vector 17P should have an 8×1 size (by virtue of P=8). However, this will drive the necessity of performing 8 multiplications for each of the 16 samples of the 4×4 block 18 to be predicted, thus resulting in a total of 16*8=128 multiplications that need to be performed. (It should be noted that the average number of multiplications per sample is a good estimate of the computational complexity. For conventional intra prediction, four multiplications are required per sample, and this increases the computational work involved. Therefore, this can be used as an upper limit for ALWIP, which will ensure that the complexity is reasonable and does not exceed the complexity of conventional intra prediction.)

[0176] Nevertheless, it is understood that by using the present technique, it is possible to reduce the number of samples 17a and 17c adjacent to the block 18 to be predicted from P to P at step 811. red Specifically, it is understood that it is possible to average (e.g. Figure 7.1100 in the figure) to obtain a reduced boundary 102 with two horizontal rows and two vertical columns, so that a 2×2 block is used as block 18 (the reduced boundary is formed by an average value). Alternatively, it is possible to perform downsampling, so that two samples for row 17c and two samples for column 17a are selected. Therefore, instead of a horizontal row 17c with four original samples being processed as having two samples (e.g. an averaged sample), a vertical column 17a that originally had four samples is processed as having two samples (e.g. an averaged sample). It is also possible to understand that after subdividing the rows 17c and columns 17a in each group 110 of two samples, a single sample is maintained (e.g. an average of the samples of the group 110 or a simple selection among the samples of the group 110). Therefore, a so-called reduced set of sample values ​​102 (M) is obtained by means of a set 102 with only four samples. red =2, N red =2,P red =M red +N red =4, where P red <P).

[0177] It is understood that it is possible to perform operations such as averaging or downsampling 100 without performing too many multiplications at the processor level: the averaging or downsampling 100 performed in step 811 can simply be obtained by direct and computationally inexpensive operations such as additions and shifts.

[0178] It is understood that at this point, the reduced sample value set 102 may be subjected to a linear or affine linear (ALWIP) transform 19 (eg, using a method such as Figure 5 In this case, the ALWIP transform 19 directly maps the four samples 102 onto the sample values ​​104 of the block 18. In the present case, no interpolation is required.

[0179] In this case, the ALWIP matrix 17M has a size of Q×P red =16×4: This follows from the fact that all Q=16 samples of block 18 to be predicted are directly obtained by ALWIP multiplication (no interpolation is required).

[0180] Thus, at step 812a, a suitable ALWIP matrix 17M having a size of Q×Pred is selected. The selection may be based at least in part on, for example, signaling from the data stream 12. The selected ALWIP matrix 17M may also utilize A k , where k can be understood as an index, which can be signaled in the data stream 12 (in some cases, the matrix is ​​also indicated as See below). The selection may be performed according to the following scheme: for each size (e.g. a height / width pairing of a block 18 to be predicted), an ALWIP matrix 17M is selected among, for example, one of three sets S0, S1, S2 of matrices (each of the three sets S0, S1, S2 may group a plurality of ALWIP matrices 17M having the same size, and the ALWIP matrix to be selected for the prediction will be one of them).

[0181] At step 812b, the selected Q×P red ALWIP matrix 17M (also indicated as A k) With P red ×1 multiplication between boundary vectors 17P.

[0182] At step 812c, the offset value (eg, b k ) is added to all obtained values ​​104 of the vector 18Q obtained, for example, by ALWIP. The value of the offset (b k Or in some cases, use instructions, see below) can be used with a specific selected ALWIP matrix (A k ) and may be based on an index (eg, which may be signaled in the data stream 12).

[0183] Therefore, the comparison between using the technology of the present invention and not using the technology of the present invention is continued here:

[0184] Without the technology of the present invention:

[0185] a block 18 to be predicted, said block having size M=4, N=4;

[0186] Q = M * N = 4 * 4 = 16 values ​​to be predicted;

[0187] P = M + N = 4 + 4 = 8 boundary samples

[0188] P=8 multiplications for each of the Q=16 values ​​to be predicted

[0189] P*Q=8*16=128 total number of multiplications;

[0190] In the case of the present invention technology, we have:

[0191] a block 18 to be predicted, said block having size M=4, N=4;

[0192] Q = M * N = 4 * 4 = 16 values ​​predicted at the end;

[0193] The reduced size of the boundary vector: P red =Mred +N red =2+2=4;

[0194] P for each of the Q=16 values ​​to be predicted by ALWIP red = 4 multiplications, P red *Q = 4*16 = total number of 64 multiplications (half of 128!)

[0195] The ratio between the number of multiplications and the number of final values ​​to be obtained is P red *Q / Q=4, i.e. less than half of the P=8 multiplications for each sample to be predicted!

[0196] As can be appreciated, it is possible to obtain appropriate values ​​at step 812 by relying on straightforward and computationally inefficient operations such as averaging (and, if applicable, addition and / or shifting and / or downsampling).

[0197] refer to Figure 7.2 , the block 18 to be predicted is here an 8×8 block of 64 samples (M=8, N=8). Here, a priori, the prediction matrix 17M should have a size Q×P=64×16 (Q=64, by virtue of Q=M*N=8*8=64, M=8 and N=8 and by virtue of P=M+N=8+8=16). Therefore, a priori, P=16 multiplications will be needed for each of the Q=64 samples of the 8×8 block 18 to be predicted, to obtain 64*16=1024 multiplications for the entire 8×8 block 18!

[0198] However, if Figure 7.2 As can be seen in FIG. 8 , a method 820 may be provided according to which, instead of using all 16 samples of the boundary, only 8 values ​​are used (e.g., 4 in the horizontal boundary row 17 c and 4 in the vertical boundary column 17 a between the original samples of the boundary). From the boundary row 17 c, 4 samples may be used instead of 8 samples (e.g., the 4 samples may be a two-by-two average and / or one sample may be selected from two samples). Thus, the boundary vector is not a P×1=16×1 vector, but only a P×1=16×1 vector. red ×1=8×1 vector (P red =M red +N red =4+4). It is understood that it is possible to select or average (eg, two by two) the samples of the horizontal rows 17c and the samples of the vertical columns 17a to have only P red = 8 boundary values ​​instead of the original P = 16 samples, thereby forming a reduced sample value set 102. The reduced set 102 will allow a reduced version of block 18 to be obtained, the reduced version having Q red =M red *N red= 4*4 = 16 samples (instead of Q = M*N = 8*8 = 64). It is possible to apply the ALWIP matrix for predicting samples with size M red ×N red =4×4 block. The reduced version of block 18 consists of Figure 7.2 Samples indicated in gray in scheme 106: Samples indicated in gray squares (including samples 118' and 118") form a 4×4 reduced block with Q obtained in step 812. red = 16 values. The 4x4 reduced block is obtained by applying the linear transformation 19 when performing step 812. After obtaining the values ​​of the 4x4 reduced block, it is possible to obtain the values ​​of the remaining samples (the samples indicated with white samples in scheme 106), for example by interpolation.

[0199] about Figure 7.1 The method 810 may additionally include deriving the remaining QQ for the M×N=8×8 block 18 to be predicted, for example by interpolation. red =64-16=48 samples (white squares) of predicted values ​​in step 813. The remaining QQ red = 64-16 = 48 samples can be interpolated from Q red = 16 directly obtained samples (the interpolation can also use the values ​​of boundary samples, for example). Figure 7.2 As can be seen in FIG. 8 , although samples 118′ and 118″ (as indicated by the gray squares) have been obtained at step 812, sample 108′ (in between samples 118′ and 118″ and indicated by the white squares) is obtained at step 813 by interpolation between samples 118′ and 118″. It is understood that interpolation can also be obtained by operations similar to those used for averaging (such as shifting and adding). Therefore, in Figure 7.2 In the example, value 108' may generally be determined as a midpoint (which may be an average) between the value of sample 118' and the value of sample 118".

[0200] By performing interpolation, it is also possible at step 813 to obtain a final version of the M×N=8×8 block 18 based on the plurality of sample values ​​indicated in 104 .

[0201] Therefore, the comparison between using the technology of the present invention and not using the technology of the present invention is:

[0202] Without the technology of the present invention:

[0203] a block 18 to be predicted, the block having dimensions M=8, N=8, and Q=M*N=8*8=64 samples to be predicted in the block 18;

[0204] P = M + N = 8 + 8 = 16 samples in boundary 17;

[0205] P = 16 multiplications for each of the Q = 64 values ​​to be predicted,

[0206] P*Q=16*64=total number of 1028 multiplications

[0207] The ratio between the number of multiplications and the number of final values ​​to be obtained is P*Q / Q=16

[0208] With the technology of the present invention:

[0209] The block to be predicted 18 has a size of M=8, N=8

[0210] Q = M * N = 8 * 8 = 64 values ​​predicted at the end;

[0211] But will use Q red ×P red ALWIP matrix, where P red =M red +N red , Q red =M red *N red , M red =4,N red =4

[0212] P in the border red =M red +N red =4+4=8 samples, where P red <P

[0213] Q for the 4×4 reduced block to be predicted (formed by the grey squares in scheme 106) red = P for each of the 16 values red = 8 multiplications,

[0214] P red *Q red = 8*16 = total number of 128 multiplications (much smaller than 1024!)

[0215] The ratio between the number of multiplications and the number of final values ​​to be obtained is P red *Q red / Q=128 / 64=2 (much smaller than the 16 obtained without the present technique!).

[0216] Therefore, the technology presented here requires 8 times less power than the previous one.

[0217] Figure 7.3Another example is shown (which may be based on method 820) where the block 18 to be predicted is a rectangular 4x8 block (M=8, N=4) with Q=4*8=32 samples to be predicted. The boundary 17 is formed by horizontal rows 17c with N=8 samples and vertical columns 17a with M=4 samples. Therefore, a priori, the boundary vector 17P will have dimensions Px1=12x1, and the prediction ALWIP matrix should be a QxP=32x12 matrix, thus resulting in the need for Q*P=32*12=384 multiplications.

[0218] However, it is possible, for example, to average or downsample at least 8 samples of the horizontal row 17c to obtain a reduced horizontal row with only 4 samples (e.g., averaged samples). In some examples, the vertical column 17a will remain intact (e.g., no averaging). In summary, the reduced border will have a size P red =8, where P red <P. Therefore, the boundary vector 17P will have a size of P red ×1=8×1. The ALWIP prediction matrix 17M will have a size of M*N red *P red =4*4*8=64. The 4×4 reduced block obtained directly when performing step 812 (formed by gray columns in scheme 107) will have a size Q red =M*N red =4*4=16 samples (instead of Q=4*8=32 of the original 4×8 block 18 to be predicted). Once the downscaled 4×4 block is obtained by ALWIP, it is possible to add the offset value b at step 813 k (Step 812c) and perform interpolation. Figure 7.3 As can be seen at step 813 in FIG. 8 , the reduced 4×4 block is expanded to a 4×8 block 18 , wherein the value 108 ′ not obtained at step 812 is obtained at step 813 by interpolating the values ​​118 ′ and 118 ″ (grey squares) obtained at step 812 .

[0219] Therefore, the comparison between using the technology of the present invention and not using the technology of the present invention is:

[0220] Without the technology of the present invention:

[0221] The block to be predicted 18 has a size of M=4, N=8

[0222] Q = M * N = 4 * 8 = 32 values ​​to be predicted;

[0223] P = M + N = 4 + 8 = 12 samples in the boundary;

[0224] P = 12 multiplications for each of the Q = 32 values ​​to be predicted,

[0225] P*Q=12*32=total number of 384 times

[0226] The ratio between the number of multiplications and the number of final values ​​to be obtained is P*Q / Q=12

[0227] With the technology of the present invention:

[0228] The block to be predicted 18 has a size of M=4, N=8

[0229] The predicted Q = M*N = 4*8 = 32 values ​​at the end;

[0230] But you can use Q red ×P red =16×8 ALWIP matrix, where M=4, N red =4,Q red =M*N red =16,

[0231] P red =M+N red =4+4=8

[0232] P in the border red =M+N red =4+4=8 samples, where P red <P

[0233] Q for the reduced block to be predicted red = P for each of the 16 values red = 8 multiplications,

[0234] Q red *P red = 16*8 = 128 total number of multiplications (less than 384!)

[0235] The ratio between the number of multiplications and the number of final values ​​to be obtained is P red *Q red / Q = 128 / 32 = 4 (much smaller than the 12 obtained without the present technique).

[0236] Therefore, with the technology of the present invention, the computing work is reduced to one third.

[0237] Figure 7.4Shown is the case of a block 18 with size M×N=16×16 to be predicted and Q=M*N=16*16=256 values ​​to be predicted at the end, the block having P=M+N=16+16=32 boundary samples. This will produce a prediction matrix with size Q×P=256×32, which will imply 256*32=8192 multiplications!

[0238] However, by applying the method 820, it is possible to reduce the number of boundary samples, for example, from 32 to 8 at step 811 (e.g., by averaging or downsampling): for example, for each group 120 of four consecutive samples of row 17a, there is still a single sample (e.g., selected among the four samples, or the average of the samples). Similarly, for each group 120 of four consecutive samples of column 17c, there is still a single sample (e.g., selected among the four samples, or the average of the samples).

[0239] Here, the ALWIP matrix 17M is Q red ×P red =64×8 matrix: This is due to the selection of P red =8 (by using 8 averaged or selected samples of the 32 samples from the boundary) and the fact that the reduced block to be predicted at step 812 is an 8×8 block (in scheme 109, the grey square is 64).

[0240] Therefore, once the 64 samples of the downscaled 8×8 block are obtained at step 812, it is possible to derive the remaining QQ of the block 18 to be predicted at step 813 red =256-64=192 values ​​104.

[0241] In this case, in order to perform the interpolation, a choice has been made to use all samples of the boundary columns 17a and only replaced samples in the boundary rows 17c. Other choices may be made.

[0242] With the method of the invention, the ratio between the number of multiplications and the number of values ​​finally obtained is Q red *P red / Q = 8*64 / 256 = 2, which is much smaller than the 32 multiplications for each value without the present technique!

[0243] The comparison between using the technology of the present invention and not using the technology of the present invention is:

[0244] Without the technology of the present invention:

[0245] The block to be predicted 18 has a size of M=16, N=16

[0246] Q = M * N = 16 * 16 = 256 values ​​to be predicted;

[0247] P = M + N = 16 * 16 = 32 samples in the boundary;

[0248] P = 32 multiplications for each of the Q = 256 values ​​to be predicted,

[0249] P*Q=32*256=total number of 8192 multiplications;

[0250] The ratio between the number of multiplications and the number of final values ​​to be obtained is P*Q / Q=32

[0251] With the technology of the present invention:

[0252] The block to be predicted 18 has a size of M=16, N=16

[0253] Q = M * N = 16 * 16 = 256 values ​​predicted at the end;

[0254] But will use Q red ×P red =64×8ALWIP matrix, where M red =4,N red =4, Q to be predicted by ALWIP red =8*8=64 samples, P red =M red +N red =4+4=8

[0255] P in the border red =M red +N red =4+4=8 samples, where P red <P

[0256] Q for the reduced block to be predicted red = P for each of the 64 values red = 8 multiplications,

[0257] Q red *P red = 64*4 = total number of 256 multiplications (less than 8192!)

[0258] The ratio between the number of multiplications and the number of final values ​​to be obtained is PP red *Q red / Q=8*64 / 256=2

[0259] (Much smaller than the 32 obtained without the present invention's technique).

[0260] Therefore, the computing power required by the present technology is 16 times less than that of traditional technology.

[0261] Therefore, it is possible to predict a predetermined block (18) of an image using a number of neighboring samples (17) by the following operation:

[0262] downscaling (100, 813) a plurality of adjacent samples to obtain a downscaled sample value set (102) having a lower number of samples than the plurality of adjacent samples (17),

[0263] A linear or affine linear transformation (19, 17M) is performed (812) on the reduced set of sample values ​​(102) to obtain predicted values ​​of predetermined samples (104, 118', 188") for a predetermined block (18).

[0264] In particular, it is possible to perform downscaling (100, 813) by downsampling the plurality of neighboring samples to obtain a downscaled set of sample values ​​(102) that is lower in number of samples compared to the plurality of neighboring samples (17).

[0265] Alternatively, it is possible to perform the downscaling (100, 813) by averaging a plurality of neighboring samples to obtain a downscaled set of sample values ​​(102) that is lower in number of samples than the plurality of neighboring samples (17).

[0266] Furthermore, it is possible to derive (813) predicted values ​​for other samples (108, 108') of the predetermined block (18) by interpolating predicted values ​​for the predetermined sample (104, 118', 118") and a plurality of neighboring samples (17).

[0267] A plurality of adjacent samples (17a, 17c) may be arranged along two sides (eg Figure 7.1 to Figure 7.4 The predetermined samples (e.g., samples obtained by ALWIP in step 812) may also be arranged in rows and columns, and along at least one of the rows and columns, the predetermined samples may be located at each nth position starting from the samples (112) of the predetermined samples 112 adjacent to both sides of the predetermined block 18.

[0268] Based on the plurality of adjacent samples (17), it is possible to determine, for each of at least one of the rows and columns, a support value (118) for one (118) of the plurality of adjacent positions, the support value being aligned with a corresponding one of the at least one of the rows and columns. It is also possible to derive, by interpolation, predicted values ​​118 for other samples (108, 108') of the predetermined block (18) based on the predicted value for the predetermined sample (104, 118', 118") and the support value for the adjacent sample (118) aligned with at least one of the rows and columns.

[0269] The predetermined samples (104) may be located along the rows at every n-th position starting from the samples (112) adjoining two sides of the predetermined block 18, and the predetermined samples may be located along the columns at every m-th position starting from the samples (112) adjoining two sides of the predetermined block (18), where n, m>1. In some cases, n=m (e.g., in Figure 7.2 and Figure 7.3 , where samples 104 , 118 ′, 118 ″ obtained directly by ALWIP at 812 and indicated by grey squares alternate along rows and columns to samples 108 , 108 ′ obtained subsequently at step 813 ).

[0270] Along at least one of the rows (17c) and columns (17a), determining the support value may be performed, for example, by downsampling or averaging (122) for each support value a group (120) of neighboring samples within a plurality of neighboring samples, the group comprising the neighboring sample (118) for which the corresponding support value was determined. Figure 7.4 In step 813, it is possible to obtain the value of sample 119 by using the predetermined sample 118'" (previously obtained at step 812) and the values ​​of the neighboring samples 118 as supporting values.

[0271] The plurality of adjacent samples may be extended in one dimension along two sides of the predetermined block (18). The reduction (811) may be performed by grouping the plurality of adjacent samples (17) into one or more groups (110) of consecutive adjacent samples and performing downsampling or averaging on each of the one or more groups (110) of adjacent samples, the groups having two or more adjacent samples.

[0272] In an example, a linear or affine linear transformation may include P red *Q red or P red *Q weighting factors, where P red is the number of sample values ​​(102) in the reduced set of sample values, and Q red Or Q is the number of predetermined samples in a predetermined block (18). 1 / 4P red *Q red or 1 / 4P red *Q weighting factors are non-zero weight values. red *Q red or P red *Q weighting factors can be applied to Q or Q red Each of the predetermined samples includes a series of P redweighting factors, wherein the series, when arranged one below the other according to a raster scan order among predetermined samples of a predetermined block (18), form an omnidirectional nonlinear envelope. red *Q or P red *Q red The weighting factors may be uncorrelated with each other via any common mapping rule. The mean of the maximum values ​​of the cross-correlations between the weighting factors of the first series for the corresponding predetermined samples and the weighting factors of the second series for the predetermined samples other than the corresponding predetermined samples or the inverse version of the latter series (whichever produces the higher maximum value) is below a predetermined threshold. The predetermined threshold may be 0.3 [or 0.2 or 0.1 in some cases]. red The adjacent samples (17) may be located along a one-dimensional path extending along two sides of the predetermined block (18) and for Q or Q red For each of the predetermined samples, P of the series of corresponding predetermined samples red The weighting factors are ordered in such a way as to traverse a one-dimensional path in a predetermined direction.

[0273] 6.1 Description of methods and apparatus

[0274] To predict samples of a rectangular block with width W (also indicated by N) and height H (also indicated by M), affine linear weighted intra prediction (ALWIP) may take as input a column of H reconstructed neighboring boundary samples to the left of the block and a row of W reconstructed neighboring boundary samples above the block. If reconstructed samples are not available, they are generated as done in traditional intra prediction.

[0275] Generating a prediction signal (e.g., a value for a complete block 18) may be based on at least some of the following three steps:

[0276] 1. In the boundary samples 17, samples 102 (eg, four samples in the case of W=H=4 and / or eight samples in other cases) may be obtained by averaging or downsampling (eg, step 811).

[0277] 2. A matrix-vector multiplication may be performed using the averaged samples (or samples remaining from downsampling) as input, followed by the addition of an offset. The result may be a scaled prediction signal for the subsampled set of samples in the original block (eg, step 812).

[0278] 3. The prediction signals at the remaining positions may be generated, for example by upsampling, from the prediction signals for the subsampled set, for example by linear interpolation (eg, step 813).

[0279] Due to steps 1.(811) and / or 3.(813), the total number of multiplications required when calculating the matrix-vector product can be such that the total number is always less than or equal to 4*W*H. In addition, the averaging operation on the boundaries and the linear interpolation of the reduced prediction signal are performed only by using additions and bit shifts. In other words, in the example, the ALWIP mode requires at most four multiplications per sample.

[0280] In some examples, the matrix (e.g., 17M) and offset vector (e.g., b) required to generate the prediction signal k ) can be taken from a set of matrices (e.g., three sets), e.g., S0, S1, S2, which are stored in storage units of a decoder and an encoder, for example.

[0281] In some examples, the set S0 may include n0 (eg, n0=16 or n0=18 or another number) matrices (e.g., consisting of), each of the matrices may have 16 rows and 4 columns and 18 offset vectors each having a size of 16 Based on Figure 7.1 The technique is performed. This set of matrices and offset vectors is for a block 18 of size 4×4. Once the boundary vector has been reduced to P red =4 vector (for Figure 7.1 811), it is possible to convert P of the reduced sample set 102 red =4 samples are directly mapped to Q=16 samples of the 4×4 block 18 to be predicted.

[0282] In some examples, set S1 may include n1 (eg, n1=8 or n1=18 or another number) matrices Each of the matrices may have 16 rows and 8 columns and 18 offset vectors each having a size of 16. Based on Figure 7.2 or Figure 7.3 The technique is performed. The matrices and offset vectors of this set S1 can be used for blocks of size 4×8, 4×16, 4×32, 4×64, 16×4, 32×4, 64×4, 8×4 and 8×8. In addition, it can also be used for blocks of size WxH (where max(W,H)>4 and min(W,H)=4), that is, for blocks of size 4×16 or 16×4, 4×32 or 32×4 and 4x×64 or 64×4. The 16×8 matrix refers to a reduced version of block 18 (which is a 4×4 block), as shown in Figure 7.2 and Figure 7.3 Obtained in.

[0283] Additionally or alternatively, the set S2 may include n2 (eg, n2=6 or n2=18 or another number) matrices (e.g., consisting of), each of the matrices may have 64 rows and 8 columns and 18 offset vectors having a size of 64 The 64×8 matrix refers to a reduced version of block 18 (which is an 8×8 block), such as in Figure 7.4 This set of matrices and offset vectors can be used for blocks of size 8×16, 8×32, 8×64, 16×8, 16×16, 16×32, 16×64, 32×8, 32×16, 32×32, 32×64, 64×8, 64×16, 64×32, 64×64.

[0284] This set of matrices and offset vectors, or portions of these matrices and offset vectors, may be used for all other block shapes.

[0285] 6.2 Averaging or Downsampling Boundaries

[0286] Here, features regarding step 811 are provided.

[0287] As explained above, the boundary samples (17a, 17c) may be averaged and / or downsampled (eg, from P samples to P samples). red <P samples).

[0288] In the first step, enter the boundary bdry top (e.g., 17c) and bdry left (e.g., 17a) can be reduced to a smaller boundary and To obtain the reduced set 102. Here, in the case of a 4×4 block, and Both consist of 2 samples, and in other cases, both consist of 4 samples.

[0289] In the case of 4×4 blocks, it is possible to define

[0290]

[0291] And similarly define therefore, is an average value obtained using, for example, a bit shift operation.

[0292] In all other cases (e.g., for blocks with width or height different from 4), if the block width W is given as W=4*2 k , then for 0≤i<4, we limit

[0293]

[0294] And similarly define

[0295] In still other cases, it is possible to downsample the boundary (e.g., by selecting a particular boundary sample from a group of boundary samples) to obtain a reduced number of samples. top [0] and bdry top [1] Select And can be in bdry top [2] and bdry top [3] Choose It is also possible to similarly define

[0296] Two reduced boundaries and Can be concatenated to the reduced boundary vector bdry red (associated with the reduced set 102) (also indicated by 17P). Reduced boundary vector bdry red So we can have size four for blocks of shape 4×4 (P red =4)( Figure 7.1 example) and have blocks of size eight for all other shapes (P red =8)( Figure 7.2 to Figure 7.4 for example).

[0297] Here, if mode < 18 (or the number of matrices in the matrix set), it is possible to limit

[0298]

[0299] If mode≥18, which corresponds to the transposed mode of mode-17, then it is possible to define

[0300]

[0301] Therefore, depending on the specific state (one state: mode<18; another state: mode≥18), it is possible to scan along different scan orders (for example, one scan order: Another scanning order: ) assigns the predicted values ​​of the output vector.

[0302] Other strategies may be implemented. In other examples, the mode index "mode" need not be in the range of 0 to 35 (other ranges may be defined). In addition, it is not necessary for each of the three sets S0, S1, S2 to have 18 matrices (therefore, instead of an expression such as mode≥18, mode≥n0, n1, n2 is possible, the mode being the number of matrices of each matrix set S0, S1, S2, respectively). In addition, the sets may each have a different number of matrices (e.g., S0 may have 16 matrices, S1 may have eight matrices, and S2 may have six matrices).

[0303] The mode and transposed information are not necessarily stored and / or transmitted as a combined mode index "mode": in some examples, it is possible to signal it explicitly as a transposed flag and a matrix index (0-15 for S0, 0-7 for S1, and 0-5 for S2).

[0304] In some cases, the combination of the transposed flag and the matrix index may be interpreted as a set index. For example, there may be one bit that operates as a transposed flag and some bits that indicate a matrix index, which are collectively indicated as a "set index".

[0305] 6.3 Generating Reduced Prediction Signal by Matrix-Vector Multiplication

[0306] Here, features regarding step 812 are provided.

[0307] In the reduced input vector bdry red (boundary vector 17P), we can generate the reduced prediction signal pred red The latter signal can be about a signal with width W red and height H red Here, W red and H red Can be limited to:

[0308] W red =4,H red =4; if max(W,H)≤8,

[0309] Otherwise, W red =min(W,8),H red =min(H,8).

[0310] The reduced prediction signal pred can be calculated by calculating the matrix-vector product and adding an offset red :

[0311] pred red =A.bdry red +b.

[0312] Here, A is a matrix (e.g., prediction matrix 17M), which may have W red *H red rows and 4 columns (if W=H=4, and 8 columns in all other cases), and b may have size W red *H red Vector.

[0313] If W=H=4, then A may have 4 columns and 16 rows, and thus 4 multiplications per sample may be required in this case to compute pred red In all other cases, A can have 8 columns, and no one can prove that in these cases we have 8*W red *H red ≤4*W*H, that is, in these cases, at most 4 multiplications are required per sample to calculate pred red .

[0314] The matrix A and the vector b can be taken from one of the sets S0, S1, S2 as follows. We restrict the index idx=idx(W,H) by setting idx(W,H)=0 if W=H=4, setting idx(W,H)=1 if max(W,H)=8, and setting idx(W,H)=2 in all cases. In addition, we can make m=mode if mode<18, and m=mode-17 otherwise. Then, if idx≤1 or idx=2 and min(W,H)>4, we can make and In the case where idx=2 and min(W,H)=4, we let A be the matrix formed by discarding each row And each line of In the case of W=4 it corresponds to odd x-coordinates in the downsampled block or in the case of H=4 it corresponds to odd y-coordinates in the downsampled block. If mode≥18, we replace the downsampled prediction signal by its transposed signal. In alternative examples, different strategies can be implemented. For example, instead of reducing the size of the larger matrix ("discarding"), a smaller matrix S1 (idx=1) is used, where W red =4 and H red = 4. That is, such blocks are now assigned to S1 instead of S2.

[0315] Other strategies may be implemented. In other examples, the mode index "mode" need not be in the range of 0 to 35 (other ranges may be defined). Furthermore, it is not necessary for each of the three sets S0, S1, S2 to have 18 matrices (thus, instead of an expression such as mode < 18, mode < n0, n1, n2 is possible, the mode being the number of matrices of each matrix set S0, S1, S2, respectively). Furthermore, the sets may each have a different number of matrices (e.g., S0 may have 16 matrices, S1 may have eight matrices, and S2 may have six matrices).

[0316] 6.4 Linear Interpolation for Generating the Final Prediction Signal

[0317] Here, features regarding step 812 are provided.

[0318] For interpolation of subsampled prediction signals of larger blocks, a second version of the averaged boundary may be needed. That is, if min(W, H) > 8 and W ≥ H, then we write W = 8*2 l , and for 0≤i<8, limit

[0319]

[0320] If min(W,H)>8 and H>W, then we similarly define

[0321] Additionally or alternatively, it is possible to have "hard downsampling", where equal

[0322]

[0323] Furthermore, one can similarly define

[0324] In generating pred red At the sample positions discarded when , the final prediction signal can be obtained by linear interpolation from pred red Produce (for example, Figure 7.2 to Figure 7.4 In some examples, if W=H=4, this linear interpolation may be unnecessary (e.g., Figure 7.1 for example).

[0325] The linear interpolation may be given as follows (although other examples are possible). Assume W ≥ H. Then, if H > H red , you can execute pred red In this case, pred red We can extend a column to the top as follows. If W = 8, then pred redMay have a width W red = 4 and can be obtained by averaging the boundary signal Extend to the top, e.g. as defined above. If W>8, then pred red With width W red = 8, and it passes through the average boundary signal Expand to the top, for example as defined above. We can red Write pred red [x][-1]. Then, with respect to the red and height 2*H red The signal of the block Can be given as

[0326]

[0327] Where 0≤x<W red And 0≤y<H red The latter process can be performed k times, up to 2 k *H red =H. Thus, if H=8 or H=16, it can be performed at most once. If H=32, it can be performed twice. If H=64, it can be performed three times. Next, the horizontal upsampling operation can be applied to the result of the vertical upsampling. The latter upsampling operation can use the entire boundary on the left of the prediction signal. Finally, if H>W, we can proceed similarly by first upsampling in the horizontal direction (if necessary) and then in the vertical direction.

[0328] This is an example of interpolation using scaled boundary samples for a first interpolation (horizontally or vertically) and the original boundary samples for a second interpolation (vertically or horizontally). Depending on the block size, only the second interpolation is needed or no interpolation is needed. If both horizontal and vertical interpolation are needed, the order depends on the width and height of the block.

[0329] However, different techniques may be implemented: for example, the original boundary samples may be used for both the first and second interpolation, and the order may be fixed, such as first horizontal then vertical (in other cases first vertical then horizontal).

[0330] Therefore, the interpolation order (horizontal / vertical) and use of downscaled / original boundary samples may vary.

[0331] 6.5 Example of the entire ALWIP process

[0332] against Figure 7.1 to Figure 7.4The different shapes in illustrate the overall process of averaging, matrix-vector multiplication, and linear interpolation. Note that the remaining shapes are considered to be one of the depicted cases.

[0333] 1. Given a 4×4 block, ALWIP can be implemented by using Figure 7.1 The technique of is used instead of two average values ​​along each axis of the boundary. The resulting four input samples enter the matrix-vector multiplication. The matrix is ​​taken from the set S0. After adding the offset, this can produce 16 final prediction samples. Linear interpolation is not necessary to generate the prediction signal. Therefore, a total of (4*16) / (4*4)=4 multiplications are performed for each sample. See for example Figure 7.1 .

[0334] 2. Given an 8×8 block, ALWIP can take four averages along each axis of the boundary. The eight input samples are obtained by using Figure 7.2 The technique goes into matrix-vector multiplication. The matrix is ​​taken from set S1. This results in 16 samples at odd positions of the prediction block. Therefore, a total of (8*16) / (8*8)=2 multiplications are performed per sample. After adding the offset, these samples can be interpolated vertically, for example by using the top boundary and horizontally, for example by using the left boundary. See e.g. Figure 7.2 .

[0335] 3. Given an 8×4 block, ALWIP can be implemented by using Figure 7.3 The technique uses four average values ​​along the horizontal axis of the boundary and four original boundary values ​​on the left boundary. The resulting eight input samples enter the matrix-vector multiplication. The matrix is ​​taken from set S1. This will produce 16 samples at odd horizontal positions and each vertical position of the prediction block. Therefore, a total of (8*16) / (8*4)=4 multiplications are performed for each sample. After adding the offset, these samples are interpolated horizontally, for example by using the left boundary. See, for example Figure 7.3 .

[0336] The transposed case is handled accordingly.

[0337] 4. Given a 16×16 block, ALWIP can take four averages along each axis of the boundary. The resulting eight input samples are obtained by using Figure 7.2 The technique goes into matrix-vector multiplication. The matrix is ​​taken from set S2. This produces 64 samples at odd positions of the prediction block. Therefore, a total of (8*64) / (16*16)=2 multiplications are performed per sample. After adding the offset, the samples are interpolated vertically by using the top boundary and horizontally by using the left boundary, for example. See e.g. Figure 7.2 . See e.g. Figure 7.4 .

[0338] For larger shapes the process can be essentially the same, and it is easy to check that the number of multiplications per sample is less than two.

[0339] For Wx8 blocks, only horizontal interpolation is necessary, since samples are given at odd horizontal and every vertical position. Thus, in these cases, at most (8*64) / (16*8)=4 multiplications are performed per sample.

[0340] Finally, for W×4 blocks (where W>8), let A k is a matrix that arises by discarding each row corresponding to odd entries along the horizontal axis of the downsampled block. Thus, the output size can be 32, and again, only horizontal interpolation remains to be performed. At most (8*32) / (16*4)=4 multiplications can be performed per sample.

[0341] The transposed case can be handled accordingly.

[0342] 6.6 Number of Required Parameters and Complexity Evaluation

[0343] The parameters required for all possible proposed intra-prediction modes may include matrices and offset vectors belonging to sets S0, S1, S2. All matrix coefficients and offset vectors may be stored as 10-bit values. Therefore, according to the above description, the proposed method may require a total number of 14,400 parameters, each with a precision of 10 bits. This corresponds to 0.018 megabytes of memory. It is pointed out that, currently, a CTU of size 128×128 in standard 4:2:0 chroma subsampling consists of 24,576 values, each of which is 10 bits. Therefore, the memory requirements of the proposed intra-prediction tool do not exceed the memory requirements of the current image reference tool adopted at the last meeting. Moreover, it is pointed out that due to the 4-tap interpolation filter of the PDPC tool or the angular prediction mode with fractional angular positions, the traditional intra-prediction mode requires four multiplications per sample. Therefore, in terms of operational complexity, the proposed method does not exceed the traditional intra-prediction mode.

[0344] 6.7 Signaling of the Proposed Intra-Prediction Mode

[0345] For luma blocks, for example, 35 ALWIP modes are proposed (other numbers of modes may be used). For each coding unit (CU) in intra mode, an indication is sent in the bitstream whether the ALWIP mode is applied on the corresponding prediction unit (PU). The signaling of the latter index may be coordinated with the MRL in the same way as the first CE test. If the ALWIP mode is applied, the index predmode of the ALWIP mode may be signaled using an MPM list with 3 MPMs.

[0346] Here, the derivation of MPM can be performed using the intra mode of the upper and left PUs as follows. There may be tables, such as three fixed tables map_angular_to_alwip idx , idx∈{0, 1, 2}, which can be assigned to each traditional intra prediction mode predmode Angular , ALWIP mode

[0347] predmode ALWIP =map_angular-to_alwip idx [predmode Angular ].

[0348] For each PU with width W and height H, we define and index

[0349] idx(PU)=idx(W,H)∈{0,1,2}

[0350] It indicates from which of the three sets the ALWIP parameters are obtained, as described in Section 4 above. above is available, belongs to the same CTU as the current PU and is in internal mode, if idx(PU)=idx(PU above ) and if in ALWIP mode PU above Applying ALWIP on

[0351]

[0352] If the above PU is available, it belongs to the same CTU as the current PU and is in intra mode, and if the traditional intra prediction mode is applied on the above PU Then we make

[0353]

[0354] In all other cases, we use

[0355]

[0356] This means that this mode is not available. In the same way but without the restriction that the left PU needs to belong to the same CTU as the current PU, we derive the mode

[0357]

[0358] Finally, three fixed default lists are provided: idx , idx∈{0, 1, 2}, each of the lists contains three different ALWIP modes.idx(PU) and Mode and In , we construct three different MPMs by replacing -1 with default values ​​and removing duplicates.

[0359] The embodiments described herein are not limited by the above described signaling of the proposed intra-prediction mode.According to an alternative embodiment, no MPM and / or mapping table is used for MIP (ALWIP).

[0360] 6.8 Adapted MPM List Derivation for Traditional Luma and Chroma Intra-Prediction Modes

[0361] The proposed ALWIP mode can be coordinated with the MPM-based coding of the conventional intra prediction mode as follows. The luma and chroma MPM list derivation process for the conventional intra prediction mode can use a fixed table map_lwip_to-angular idx , idx∈{0, 1, 2}, thus setting the ALWIP mode predmode on a given PU LWIP Mapping to one of the traditional intra-prediction modes

[0362] predmode Angular =map_lwip_to_angular idx(PU) [predmode LWIP ].

[0363] For luma MPM list export, whenever you encounter the ALWIP mode predmode LWIP When the adjacent luma block is , this block can be treated as if it is using the traditional intra prediction mode predmode Angular For chroma MPM list derivation, the same mapping can be used to convert ALWIP mode to legacy intra prediction mode whenever the current luma block uses LWIP mode.

[0364] Clearly, ALWIP mode can also be coordinated with traditional intra prediction mode without using MPM and / or mapping table. For example, it is possible that for chroma blocks, whenever the current luma block uses ALWIP mode, ALWIP mode is mapped to planar intra prediction mode.

[0365] 7. Implement effective implementation examples

[0366] The above examples are briefly outlined as they may form the basis for further extending the embodiments described herein below.

[0367] In order to predict a predetermined block 18 of the image 10, where a plurality of neighboring samples are used, 17a, 17c are used.

[0368] By averaging, a reduction 100 of a plurality of adjacent samples has been performed to obtain a reduced sample value set 102 which is lower in number of samples compared to a plurality of adjacent samples. This reduction is optional in the embodiments herein and results in a so-called sample value vector mentioned below. The reduced sample value set is subjected to a linear or affine linear transformation 19 to obtain a predicted value of a predetermined sample 104 for a predetermined block. This transformation is indicated later using a matrix A and an offset vector b, which have been obtained by machine learning (ML) and should be an effectively preformed implementation.

[0369] By interpolation, the predicted values ​​for the other samples 108 of the predetermined block are derived based on the predicted values ​​for the predetermined sample and a plurality of neighboring samples. It should be said that, in theory, the results of the affine / linear transformation can be associated with non-full-pixel sample positions of the block 18, so that according to an alternative embodiment, all samples of the block 18 can be obtained by interpolation. Interpolation is also not required at all.

[0370] A plurality of adjacent samples may extend in one dimension along two sides of the predetermined block, the predetermined samples being arranged in rows and columns and along at least one of the rows and columns, wherein the predetermined samples may be located at each n-th position starting from a sample (112) of the predetermined samples adjacent to the two sides of the predetermined block. Based on the plurality of adjacent samples, for each of at least one of the rows and columns, a support value for one of the plurality of adjacent positions (118) may be determined, the support value being aligned with a corresponding one of the at least one of the rows and columns, and by interpolation, a predicted value for other samples 108 of the predetermined block may be derived based on the predicted value for the predetermined sample and the support value for the adjacent sample aligned with at least one of the rows and columns. The predetermined samples may be located at each n-th position starting from a sample 112 of the predetermined samples adjacent to the two sides of the predetermined block along the rows, and the predetermined samples may be located at each m-th position starting from a sample 112 of the predetermined samples adjacent to the two sides of the predetermined block along the columns, wherein n, m>1. Possibly n=m. Along at least one of the rows and columns, the support value may be determined by averaging (122) for each support value a group 120 of neighboring samples within a plurality of neighboring samples including the neighboring sample 118 for which the corresponding support value is determined. The plurality of neighboring samples may be extended in one dimension along both sides of the predetermined block and reduced by grouping the plurality of neighboring samples into groups 110 of one or more consecutive neighboring samples and performing averaging on each of the one or more groups of neighboring samples having more than two neighboring samples.

[0371] For a predetermined block, a prediction residual may be transmitted in a data stream. The prediction residual may be derived from the data stream at the decoder, and the predetermined block may be reconstructed using the prediction residual and the predicted value for the predetermined sample. At the encoder, the prediction residual is encoded into the data stream at the encoder.

[0372] The image may be subdivided into a plurality of blocks having different block sizes, the plurality of blocks comprising a predetermined block. Subsequently, a linear or affine linear transform for block 18 may be selected depending on the width W and height H of the predetermined block, such that the linear or affine linear transform selected for the predetermined block is selected among a first set of linear or affine linear transforms, as long as the width W and height H of the predetermined block are within the first set of width / height pairs and a second set of linear or affine linear transforms, as long as the width W and height H of the predetermined block are within the second set of width / height pairs that do not intersect the first set of width / height pairs. Again, it becomes clear subsequently that the affine / linear transform is represented by means of parameters, namely the weights of C, and optionally, offset and scale parameters.

[0373] The decoder and the encoder may be configured to: subdivide the image into a plurality of blocks having different block sizes, including a predetermined block; and select a linear or affine linear transform depending on a width W and a height H of the predetermined block, such that the linear or affine linear transform selected for the predetermined block is selected among the following:

[0374] The first set of linear or affine linear transformations, as long as the width W and height H of the predetermined block are within the first set of width /

[0375] Highly paired,

[0376] a second set of linear or affine linear transformations, as long as the width W and height H of the predetermined block are within the second set of width / height pairs that do not intersect the first set of width / height pairs, and

[0377] The third group of linear or affine linear transformations is as long as the width W and height H of the predetermined block are not in the same group as the first and

[0378] The second set of width / height pairs intersects within one or more width / height pairs of the third set.

[0379] The third set of one or more width / height pairs comprises only one width / height pair W', H', and each linear or affine linear transform within the first set of linear or affine linear transforms is used to transform N' sample values ​​into W'*H' predicted values ​​for a W'×H' array of sample positions.

[0380] Each of the first and second sets of width / height pairs may include W p Not equal to H p The first width / height pair W p , Hp , and H q =W p And W q =H p The second width / height pair W q , H q .

[0381] Each of the first and second sets of width / height pairs may additionally include W p Equal to H p And H p >H q The third width / height pair W p , H p .

[0382] For a predetermined block, a setting index may be transmitted in the data stream indicating which linear or affine linear transform is selected for the block 18 among a set of predetermined linear or affine linear transforms.

[0383] The plurality of adjacent samples may be extended in one dimension along both sides of the predetermined block, and may be reduced by grouping, for a first subset of the plurality of adjacent samples adjacent to the first side of the predetermined block, a first subset into a first group 110 of one or more consecutive adjacent samples, and for a second subset of the plurality of adjacent samples adjacent to the second side of the predetermined block, grouping, for a second subset of the plurality of adjacent samples adjacent to the second side of the predetermined block, a second subset into a second group 110 of one or more consecutive adjacent samples, and performing averaging on each of the first and second groups of one or more adjacent samples having more than two adjacent samples to obtain a first sample value from the first group and a second sample value from the second group. Subsequently, a linear or affine linear transform may be selected from a set of predetermined linear or affine linear transforms depending on a setting index, such that two different states of the setting index result in selection of one of the linear or affine linear transforms in the predetermined set of linear or affine linear transforms, and in the case where the setting index takes a first state of two different states in the form of a first vector, the reduced sample value set may be subjected to a predetermined linear or affine linear transform to produce an output vector of predicted values, and the predicted values ​​of the output vector are distributed to predetermined samples of a predetermined block along a first scanning order, and in the case where the setting index takes a second state of two different states in the form of a second vector, the first and second vectors are different, such that a component filled by one of the first sample values ​​in the first vector is filled by one of the second sample values ​​in the second vector, and a component filled by one of the second sample values ​​in the first vector is filled by one of the first sample values ​​in the second vector, so as to produce an output vector of predicted values, and the predicted values ​​of the output vector are distributed to predetermined samples of a predetermined block along a second scanning order, wherein the predetermined block is transposed relative to the first scanning order.

[0384] Each linear or affine linear transform within the first set of linear or affine linear transforms can be used to transform N1 sample values ​​into w1*h1 predicted values ​​for a w1×h1 array of sample positions, and each linear or affine linear transform within the first set of linear or affine linear transforms is used to transform N2 sample values ​​into w2*h2 predicted values ​​for a w2×h2 array of sample positions, wherein for a first predetermined width / height pairing in the first set of width / height pairs, w1 can exceed the width of the first predetermined width / height pairing or h1 can exceed the height of the first predetermined width / height pairing, and for a second predetermined width / height pairing in the first set of width / height pairs, w1 cannot exceed the width of the second predetermined width / height pairing and h1 cannot exceed the height of the second predetermined width / height pairing. A plurality of adjacent samples may then be downscaled (100) by averaging to obtain a downscaled set of sample values ​​(102), such that the downscaled set of sample values ​​102 has N1 sample values ​​in the case where the predetermined block has a first predetermined width / height pair and in the case where the predetermined block has a second predetermined width / height pair, and the downscaled set of sample values ​​may be subjected to a selected linear or affine linear transformation along a width dimension when w1 exceeds a width of one width / height pair in the case where the predetermined block has a first predetermined width / height pair, or along a height dimension when h1 exceeds a height of one width / height pair by using only a first sub-portion of the selected linear or affine linear transformation associated with subsampling of the w1×h1 array of sample positions in the case where the predetermined block has a second predetermined width / height pair, and the downscaled set of sample values ​​may be completely subjected to the selected linear or affine linear transformation in the case where the predetermined block has a second predetermined width / height pair.

[0385] Each linear or affine linear transform within the first set of linear or affine linear transforms can be used to transform N1 sample values ​​into w1*h1 predicted values ​​for a w1×h1 array of sample positions where w1=h1, and each linear or affine linear transform within the second set of linear or affine linear transforms is used to transform N2 sample values ​​into w2*h2 predicted values ​​for a w2×h2 array of sample positions where w2=h2.

[0386] All of the above-described embodiments are merely illustrative, as they may form the basis for the embodiments described herein below. That is, the above concepts and details should be used to understand the following embodiments and should serve as a reservoir for possible extensions and modifications of the embodiments described herein below. Specifically, many of the details described above are optional, such as the averaging of adjacent samples, the fact that adjacent samples are used as reference samples, etc.

[0387] More generally, the embodiments described herein assume that the prediction signal for a rectangular block is generated from reconstructed samples, such as generating an intra prediction signal for a rectangular block from neighboring reconstructed samples to the left and above the block. The generation of the prediction signal is based on the following steps.

[0388] 1. Among the reference samples called boundary samples, but not excluding the transfer of the description to reference samples located elsewhere, the samples can be obtained by averaging. Here, the averaging is performed for the boundary samples on the left and above the block or only for the boundary samples on one of the two sides. If no averaging is performed on a side, the samples on this side remain unchanged.

[0389] 2. Perform a matrix-vector multiplication, optionally followed by adding an offset, where the input vector to the matrix-vector multiplication is the concatenation of the averaged boundary samples on the left side of the block with the original boundary samples above the block if averaging is applied only on the left side, or the concatenation of the original boundary samples on the left side of the block with the averaged boundary samples above the block if averaging is applied only on the above side, or the concatenation of the averaged boundary samples on the left side of the block with the averaged boundary samples above the block if averaging is applied only on both sides of the block. Again, there will be alternatives, such as not using averaging at all.

[0390] 3. The result of the matrix-vector multiplication and optional offset addition may optionally be a downscaled prediction signal for the subsampled set of samples in the original block. The prediction signals at the remaining positions may be generated by linear interpolation from the prediction signals for the subsampled set.

[0391] The computation of the matrix-vector product in step 2 should preferably be performed in integer arithmetic. Thus, if x=(x1, ..., x n ) represents the input for the matrix-vector product, i.e. x represents the concatenation of the (averaged) boundary samples to the left and above the block, then among x, only bit shifts, addition of offset vectors and integer multiplications should be used to compute the (reduced) prediction signal computed in step 2. Ideally, the prediction signal in step 2 would be given as Ax+b, where b is an offset vector which may be zero, and where A is derived by some machine learning based training algorithm. However, such training algorithms typically only produce a matrix A=A given in floating point precision float . We are thus faced with specifying integer operations in the sense given above so that the expression A float x is well approximated using these integer operations. Here, it is important to mention that these integer operations do not have to be chosen so that they approximate the expression A assuming a uniform distribution of the vector x float x, but usually considering the expression A floatx will approximate the input vector x to be the (averaged) boundary samples from a natural video signal, where we can expect the components x of x to be i Some correlations between them.

[0392] Figure 8 An embodiment of an apparatus 1000 for predicting a predetermined block 18 of an image 10 using a plurality of reference samples 17 is shown. The plurality of reference samples 17 may depend on a prediction mode used by the apparatus 1000 to predict the predetermined block 18. If the prediction mode is, for example, intra prediction, reference samples 171 adjacent to the predetermined block may be used. In other words, the plurality of reference samples 17 are arranged within the image 10, for example along the outer edge of the predetermined block 18. If the prediction mode is, for example, inter prediction, reference samples 172 in another image 10' may be used.

[0393] The apparatus 1000 is configured to form 100 a sample value vector 400 from a plurality of reference samples 17. The sample value vector may be obtained by different techniques. The sample value vector may, for example, include all reference samples 17. Optionally, the reference samples may be weighted. According to another example, Figure 7.1 to Figure 7.4 The sample value vector 400 is formed as described for the sample value vector 102 by one of the reference samples 17. In other words, the sample value vector 400 may be formed by averaging or downsampling. Thus, for example, a group of reference samples may be averaged to obtain a sample value vector 400 having a reduced set of values. In other words, the apparatus is configured, for example, to form the sample value vector 102 from the plurality of reference samples 17 by adopting, for each component of the sample value vector 400, one of the plurality of reference samples 17 as the corresponding component of the sample value vector and / or averaging two or more components of the sample value vector 400, i.e. averaging two or more reference samples of the plurality of reference samples 17 to obtain the corresponding component of the sample value vector 400.

[0394] The device 1000 is configured to derive 401 a further vector 402 from a sample value vector 400 onto which the sample value vector 400 is mapped by a predetermined reversible linear transformation 403. The further vector 402 comprises, for example, only integers and / or fixed-point values. By way of example, the reversible linear transformation 403 is selected such that the prediction of the samples of the predetermined block 18 is performed by integer arithmetic or fixed-point arithmetic.

[0395] Furthermore, the apparatus 1000 is configured to calculate a matrix-vector product 404 between the further vector 402 and a predetermined prediction matrix 405 to obtain a prediction vector 406, and to predict samples of the predetermined block 18 based on the prediction vector 406. Based on the advantageous further vector 402, the predetermined prediction matrix may be quantized to enable integer and / or fixed-point operations, wherein quantization errors only slightly affect the predicted samples of the predetermined block 18.

[0396] According to an embodiment, the apparatus 1000 is configured to use fixed point arithmetic operations to calculate the matrix-vector product 404. Alternatively, integer operations may be used.

[0397] According to an embodiment, the apparatus 1000 is configured to compute the matrix-vector product 404 without using floating point arithmetic operations.

[0398] According to an embodiment, the apparatus 1000 is configured to store a fixed-point representation of the predetermined prediction matrix 405. Additionally or alternatively, an integer representation of the predetermined prediction matrix 405 may be stored.

[0399] According to an embodiment, the apparatus 1000 is configured to use interpolation when predicting samples of the predetermined block 18 based on the prediction vector 406 to calculate at least one sample position of the predetermined block 18 based on the prediction vector 406, each component of the prediction vector 406 being associated with a corresponding position within the predetermined block 18. Figure 7.1 to Figure 7.4 The interpolation is performed as described in one of the embodiments shown in .

[0400] Fig. 9 1100 and the sample value vector 400. The sample of the predetermined block can be predicted based on a first matrix-vector product between a matrix A 1100 derived by some machine learning based training algorithm and a sample value vector 400. Optionally, an offset b 1110 can be added. In order to achieve an integer approximation or a fixed point approximation of this first matrix-vector product, the sample value vector can be subjected to a reversible linear transformation 403 to determine another vector 402. A second matrix-vector product between another matrix B 1200 and the another vector 402 can be equal to the result of the first matrix-vector product.

[0401] Due to the characteristics of the other vector 402, the second matrix-vector product can be an integer approximated by a matrix-vector product 404 between a predetermined prediction matrix C 405 and the other vector 402 plus another offset 408. The other vector 402 and the other offset 408 can be composed of integers or fixed-point values. For example, all components of the other offset are the same. The predetermined prediction matrix 405 can be a quantized matrix or a matrix to be quantized. The result of the matrix-vector product 404 between the predetermined prediction matrix 405 and the other vector 402 can be understood as a prediction vector 406.

[0402] In the following, more details on this integer approximation are provided.

[0403] Possible solution according to Example 1: Subtract and add the average value

[0404] The expression A that can be used in the above situation floatOne possible integration of the integer approximation of x is to replace the i0th component of x (i.e., the sample value vector 400) by the mean value mean(x) of the components of x (i.e., the predetermined value 1400). , i.e. the predetermined component 1500, and this average value is subtracted from all other components. In other words, the definition is as follows Fig.10a The reversible linear transformation 403 shown in FIG. 4 is such that the predetermined component 1500 of the other vector 402 becomes a, and each of the other components of the other vector 402 (except the predetermined component 1500) is equal to the corresponding component of the sample value vector minus a, where a is a predetermined value 1400, which is, for example, an average value, such as an arithmetic mean or a weighted average value, of the components of the sample value vector 400. This operation on the input is given by the reversible transformation T403, which has an obvious integer implementation, especially when the size n of x is a power of two.

[0405] Because A float =(A float T -1 )T, so if we perform this transformation on the input x, we must find an integral approximation to the matrix-vector product By, where B = (A float T -1 ) and y = Tx. Since the matrix-vector product A float x represents a prediction about a rectangular block (i.e., a predetermined block), and since x includes the (e.g., averaged) boundary samples of the block, we should expect that if all sample values ​​of x are equal, i.e., for all i, x i =mean(x), the predicted signal A float Each sample value in x should be close to mean(x) or exactly equal to mean(x). This means that we should expect the i0-th column of B (i.e., the other matrix 1200), i.e., the column corresponding to the predetermined component, to be very close to or equal to the column consisting of only ones. Therefore, if M(i0), i.e., the integer matrix 1300, is a matrix whose i0-th column consists of ones and all other columns are zero, so that By=Cy+M(i0)y and C=BM(i0), then we should expect the i0-th column of C (i.e., the predetermined prediction matrix 405) to have an effectively small entry or be zero, such as Fig.10b Furthermore, since the components of x are correlated, we can expect that for each i≠i0, the i-th component y of y i =x i -mean(x) often has a much smaller absolute value than the i-th component of x. Since the matrix M(i0) is an integer matrix, an integer approximation of By is achieved given an integer approximation of Cy, and with the above arguments, we can expect that the quantization error introduced by quantizing each term of C in a suitable way will only slightly affect the value of A.float Error in quantization of results of By response of x.

[0406] The predetermined value 1400 is not necessarily the mean value mean(x). float The integer approximation described herein of x may also be implemented using the following alternative definition of the predetermined value 1400:

[0407] In the expression A float In another possible integration of the integer approximation of x, the i0th component of x remain unchanged and subtract the same value from all other components That is, for every i≠i0, and In other words, the predetermined value 1400 may be a component of the sample value vector 400 that corresponds to the predetermined component 1500 .

[0408] Alternatively, the predetermined value 1400 is a default value or a value signaled in a data stream in which the image is encoded.

[0409] The predetermined value 1400 is equal to, for example, 2 bitdepth-1 In this case, another vector 402 can be represented by y0=2 bitdepth-1 and i =x i -x0 limit, where i>0.

[0410] Alternatively, the predetermined component 1500 becomes a constant minus the predetermined value 1400. The constant is equal to, for example, 2 bitdepth-1 According to one embodiment, the predetermined component of another vector y 402 1500 is equal to 2 bitdepth-1 Subtract the component of the sample value vector 400 corresponding to the predetermined component 1500 And all other components of the further vector 402 are equal to the corresponding components of the sample value vector 400 minus the components of the sample value vector 400 corresponding to the predetermined components 1500 .

[0411] For example, it may be advantageous for the predetermined value 1400 to have a small deviation from the predicted value of the samples of the predetermined block.

[0412] According to an embodiment, the apparatus 1000 is configured to include a plurality of reversible linear transformations 403, each of which is associated with a component of the other vector 402. In addition, the apparatus is configured, for example, to select a predetermined component 1500 from among the components of the sample value vector 400 and use the reversible linear transformation 403 associated with the predetermined component 1500 among the plurality of reversible linear transformations as the predetermined reversible linear transformation. This is, for example, due to the different positions of the i0-th row (i.e., the row of the reversible linear transformation 403 corresponding to the predetermined component), which depends on the position of the predetermined component in the other vector 402. If, for example, the first component of the other vector 402, i.e., y1, is the predetermined component, the i0-th row will replace the first row of the reversible linear transformation.

[0413] like Fig.10b As shown in , the matrix components 414 of the prediction matrix C 405 (which correspond to the predetermined components 1500 of the other vector 402) within the column 412 (i.e., the i0-th column) of the predetermined prediction matrix 405 are, for example, all zero. In this case, the device is, for example, configured to calculate the matrix-vector product 404 by performing a multiplication in a manner such that a reduced prediction matrix C′405 generated from the predetermined prediction matrix C 405 by discarding the column 412 and a further vector 410 generated from the other vector 402 by discarding the predetermined components 1500 are calculated, as shown in Fig.10c As shown in . Therefore, the prediction vector 406 can be calculated using fewer multiplications.

[0414] like Fig. 9 , Fig.10b and Fig.10c As shown in , the apparatus 1000 may be configured to calculate the sum of the corresponding component and a (i.e., the predetermined value 1400) for each component of the prediction vector 406 when predicting the sample of the predetermined block based on the prediction vector 406. The sum may be represented by the sum of the prediction vector 406 and the vector 409, wherein all components of the vector 409 are equal to the predetermined value 1400, as shown in Fig. 9 and Fig.10c Alternatively, the summation may be represented by the sum of the matrix-vector product 1310 between the prediction vector 406 and the integer matrix M 1300 and the further vector 402, as Fig.10b As shown in , the matrix components of the integer matrix 1300 are 1s within the columns of the integer matrix 1300, namely the i0-th column, which correspond to the predetermined components 1500 of the other vector 402, and all other components are, for example, zero.

[0415] The result of summing the predetermined prediction matrix 405 and the integer matrix 1300 is equal to or approximates, for example Fig. 9 Another matrix 1200 is shown in FIG.

[0416] In other words, a matrix (i.e., matrix B) generated by summing each matrix component of the predetermined prediction matrix C 405 in a column 412 (i.e., column i0) of the predetermined prediction matrix 405 with the predetermined component 1500 of the other vector 402 with one, i.e., another matrix B 1200, is multiplied by the reversible linear transformation 403, corresponding to, for example, a quantized version of the machine learning prediction matrix A 1100, as shown in FIG. Fig. 9 , Fig.10a and Fig.10b The sum of each matrix component in the i0th column 412 of the predetermined prediction matrix C 405 and one may correspond to the sum of the predetermined prediction matrix 405 and the integer matrix 1300, as shown in FIG. Fig.10b As shown in Fig. 9 As shown in , the machine learning prediction matrix A 1100 can be equal to the result of multiplying another matrix 1200 by the reversible linear transformation 403. This is because A·x=BT·yT -1 The predetermined prediction matrix 405 is, for example, a quantized matrix, an integer matrix and / or a fixed-point matrix, thereby a quantized version of the machine learning prediction matrix A 1100 can be implemented.

[0417] Matrix multiplication using only integer operations

[0418] For a low complexity implementation (in terms of the complexity of adding and multiplying scalar values, and in terms of the storage required for the terms of the matrices involved), it is desirable to perform the matrix multiplication 404 using only integer arithmetic.

[0419] To calculate the approximation z = Cy, that is,

[0420]

[0421] In the case of using only integer operations, according to one embodiment, the real value C i，j Must map to an integer value This can be done, for example, by quantizing with a uniform scalar, or by considering the value y i The integer values ​​represent, for example, fixed-point numbers, each of which can be stored in a fixed number of bits n_bit, for example, n_bit=8.

[0422] A matrix-vector product 404 with a matrix of size m×n (i.e., the predetermined prediction matrix 405) can then be performed as shown in this pseudocode, where 〈,〉 are arithmetic binary left and right shift operations, and +, -, and * only operate on integer values.

[0423]

[0424] Here, array C, ie the predetermined prediction matrix 405, stores fixed point numbers as, for example, integers. The final addition of final_offset and the right shift operation of right_shift_result reduce the precision by truncation to obtain the desired fixed point format at the output.

[0425] To allow for an increased range of real values ​​that can be represented by integers in C, two additional matrices offset can be used i，j and scale i，j ,like Fig.11 and Fig.12 The embodiment shown in FIG. 1 makes the matrix-vector product

[0426]

[0427] The y in j Each coefficient b i，j is given by

[0428]

[0429] Value offset i，j and scale i，j These integers may represent fixed-point numbers, which may each be represented by a fixed number of bits (e.g., 8 bits) or by the same number of bits n_bit (which are used to store the value). ) to store.

[0430] In other words, the apparatus 1000 is configured to use a prediction parameter (eg, an integer value) and value offset i，j and scale i，j ) represents a predetermined prediction matrix 405 and calculates a matrix-vector product 404 by performing multiplications and summing on components of another vector 402 and prediction parameters and intermediate results generated therefrom, wherein the absolute values ​​of the prediction parameters can be represented by n-bit fixed-point number representations, wherein n is equal to or lower than 14, or alternatively equal to or lower than 10, or alternatively equal to or lower than 8. For example, the components of another vector 402 are multiplied by the prediction parameters to produce products as intermediate results, which in turn are summed or form addends of the summation.

[0431] According to an embodiment, the prediction parameters include weights, each of which is associated with a corresponding matrix component of the prediction matrix. In other words, the predetermined prediction matrix is ​​replaced or represented, for example, by the prediction parameters. The weights are, for example, integers and / or fixed-point values.

[0432] According to one embodiment, the prediction parameters further include one or more scaling factors, such as the value scale i，j, each of the one or more scaling factors is an integer value corresponding to the weight used to scale the weight (e.g. ), the one or more corresponding matrix components are associated with one or more corresponding matrix components of the predetermined prediction matrix 405. Additionally or alternatively, the prediction parameters include one or more offsets, such as the value offset i，j , where each offset is related to a weight (e.g., an integer value ) is associated with one or more corresponding matrix components of the predetermined prediction matrix 405 of the offset, and the one or more corresponding matrix components are associated with one or more corresponding matrix components of the predetermined prediction matrix 405.

[0433] In order to reduce the offset i，j and scale i，j The amount of storage required, whose values ​​can be chosen to be constant for a particular set of indices i, j. For example, their entries can be constant for each column, or they can be constant for each row, or they can be constant for all i, j, as Fig.11 as shown in .

[0434] For example, in a preferred embodiment, offset i，j and scale i，j For a prediction mode all values ​​of the matrix are constant, such as Fig.12 Thus, when there are K prediction modes, where k = 0..K-1, only a single value o is required. k and a single value s k to compute the prediction for mode k.

[0435] According to one embodiment, offset i，j and scale i，j is constant, i.e. the same, for all matrix-based intra prediction modes. Additionally or alternatively, offset i，j and / or scale i，j The size may be constant, ie the same, for all blocks.

[0436] In the offset representation o k And the scaling represents s k In the case of , the calculation in (1) can be modified as follows:

[0437]

[0438]

[0439] Extended embodiments resulting from this solution

[0440] The above solutions refer to the following embodiments:

[0441] 1. A prediction method as in Section I, wherein in step 2 of Section I, the following is done for integer approximations of the matrix-vector products involved: n ), for a fixed i0 with 1≤i0≤n, calculate the vector y=(y1, ..., y n ), where for i≠i0, y i =x i -mean(x), and where , and where mean(x) denotes the mean of x. The vector y then serves as input for the matrix-vector product Cy (integer implementation of the matrix vector) so that the (downsampled) prediction signal pred from step 2 of section I is given as pred=Cy+meanpred(x). In these equations, meanpred(x) denotes the signal for each sample position in the domain of the (downsampled) prediction signal that is equal to mean(x). (See e.g. Fig.10b )

[0442] 2. The prediction method as in Section I, wherein in step 2 of Section I, the following is performed for integer approximations of the matrix-vector products involved: n ), for a fixed i0 with 1≤i0≤n, calculate the vector y=(y1, ..., y n-1 ), where for i<i0, y i =x i -mean(x) and where for i ≥ i0, y i =x i+1 -mean(x) and where mean(x) denotes the mean of x. The vector y then serves as input for the matrix-vector product Cy (integer implementation of the matrix vector) so that the (downsampled) prediction signal pred from step 2 of section I is given as pred=Cy+meanpred(x). In these equations, meanpred(x) denotes a signal that is equal to mean(x) for each sample position in the domain of the (downsampled) prediction signal. (See e.g. Fig.10c )

[0443] 3. The prediction method as in Section I, where by using the matrix-vector product z i =∑ j b i，j *y j The coefficients in Given an integer implementation of the matrix-vector product Cy. (See e.g. Fig.11 )

[0444] 4. The prediction method as in Section I, wherein step 2 uses one of K matrices so that multiple prediction modes can be calculated, each prediction mode using a different matrix k=0...K-1 where by using the matrix-vector product z i =∑ j b i，j *y j The coefficients in Given a matrix-vector product C k The integer implementation of y. (See e.g. Fig.12 )

[0445] That is, according to an embodiment of the present application, the encoder and the decoder operate as follows to predict a predetermined block 18 of the image 10, see Fig. 9 For prediction, a plurality of reference samples are used. As outlined above, embodiments of the present application will not be limited to intra-coding and therefore, reference samples will not be limited to neighboring samples, i.e. samples of neighboring blocks 18 of the image 10. In particular, the reference samples will not be limited to reference samples arranged along the outer edge of the block 18, such as samples of the outer edge of the adjacent block. However, this scenario is of course an embodiment of the present application.

[0446] In order to perform the prediction, a sample value vector 400 is formed from reference samples such as reference samples 17a and 17c. Possible formations have been described above. The formation may involve averaging, thereby reducing the number of samples 102 or the number of components of the vector 400 compared to the reference samples 17 that contributed to the formation. As described above, the formation may also depend to some extent on the size or dimensions of the block 18, such as its width and height.

[0447] This vector 400 should be affinely or linearly transformed to obtain the prediction of block 18. Different nomenclatures have been used above. Using the most recent nomenclature, it is intended to perform the prediction by applying the vector 400 to the matrix A by means of a matrix-vector product within the summation of the offset vector b. The offset vector b is optional. The affine or linear transformation determined by A or A and b can be determined by the encoder and decoder, or more precisely, for the purpose of prediction, it is determined based on the size and dimensions of the block 18, as already described above.

[0448] However, in order to achieve the computational efficiency improvements outlined above or to make the prediction more efficient in terms of implementation, the affine or linear transformation has been quantized, and the encoder and decoder or its predictor uses the above-mentioned C and T in order to represent and perform the linear or affine transformation, wherein C and T applied in the manner described above represent a quantized version of the affine transformation. In particular, instead of applying the vector 400 directly to the matrix A, the predictor in the encoder and decoder applies a vector 402, which is generated from the sample value vector 400 by mapping the sample value vector 400 via a predetermined reversible linear transformation T. It is possible that the transformation T as used here is the same as long as the vector 400 has the same size, that is, it does not depend on the size of the block, that is, the width and height, or at least is the same for different affine / linear transformations. In the above, the vector 402 has been denoted as y. The exact matrix used to perform the affine / linear transformation as determined by machine learning will be B. However, instead of performing B exactly, the prediction in the encoder and decoder is performed by means of an approximation or a quantized version thereof. In particular, the representation is done by appropriately representing C in the manner outlined above, where C+M represents a quantized version of B.

[0449] Therefore, prediction in the encoder and decoder is further performed by calculating the matrix-vector product 404 between the vector 402 and the predetermined prediction matrix C appropriately represented and stored at the encoder and decoder in the manner described above. The vector 406 generated from this matrix-vector product is then used to predict the samples 104 of the block 18. As described above, for the purpose of prediction, each component of the vector 406 can be summed with the parameter a as indicated at 408 to compensate for the corresponding definition of C. The optional summation of the vector 406 with the offset vector b can also be included in the prediction of the block 18 derived based on the vector 406. As described above, each component of the vector 406 and therefore each component of the summation of the vector 406, the vector of all a indicated at 408, and the optional vector b can directly correspond to the sample 104 of the block 18, and therefore, indicate the predicted value of the sample. It is also possible to predict only a subset of the samples 104 of the block in this way and to derive the remaining samples of the block 18, such as 108, by interpolation.

[0450] As described above, there are different embodiments for setting a. For example, it can be the arithmetic mean of the components of vector 400. For this case, see Figure 10. The reversible linear transformation T can be as indicated in Figure 10. i0 is a predetermined component of the sample value vector and vector 402, respectively, which is replaced by a. However, as also indicated above, there are other possibilities. However, with regard to the representation of C, it has also been indicated above that C can be embodied in different ways. For example, the matrix-vector product 404 can end in its actual calculation with the actual calculation of a smaller matrix-vector product with lower dimensions. Specifically, as indicated above, due to the definition of C, the entire i0-th column 412 of C can become 0, so that the actual calculation of the product 404 can be performed by a reduced version of the vector 402, which is obtained by omitting the component , that is, produced in vector 402 by multiplying this reduced vector 410 by the reduced matrix C′, which is produced in C by discarding the i0-th column 412.

[0451] The weights of C or the weights of C', i.e. the components of this matrix, can be represented and stored in a fixed-point representation. However, these weights 414 can also be stored in a manner associated with different scaling and / or offsets as described above. The scaling and offset can be defined for the entire matrix C, i.e. all weights 414 for the matrix C or the matrix C' are equal, or can be defined in a way that all weights 414 for the same row or all weights 414 for the same column of the matrix C and the matrix C' are constant or equal, respectively. In this regard, Fig.11 It is shown that the calculation of the matrix-vector product (ie the result of the product) can actually be performed slightly differently, ie for example by shifting the multiplication with the scaling towards vector 402 or 404 , thereby reducing the number of further multiplications that have to be performed. Fig.12 This illustrates the case where one scale and one offset are used for all weights 414 for C or C', such as performed in calculation (2) above.

[0452] According to an embodiment, the apparatus described herein for predicting a predetermined block of an image may be configured to use matrix-based intra-sample prediction, comprising the following features:

[0453] The apparatus is configured to form a sample value vector pTemp[x] 400 from a plurality of reference samples 17. Assuming pTemp[x] is 2*boundarySize, pTemp[x] may be filled by - for example by direct copying or by subsampling or pooling - neighboring samples redT[x] (where x=0..boundarySize-1) located at the top of the predetermined block, followed by neighboring samples redL[x] located to the left of the predetermined block (for example in the case of isTransposed=0), or vice versa in the case of transposition (for example in the case of isTransposed=1).

[0454] The input value p[x] of x=0..inSize-1 is derived, that is, the apparatus is configured to derive another vector p[x] from the sample value vector pTemp[x], to which the sample value vector pTemp[x] is mapped by a predetermined reversible linear transformation (or more specifically, a predetermined reversible affine linear transformation), as follows:

[0455] - If mipSizeId is equal to 2, the following applies:

[0456] p[x]=pTemp[x+1]-pTemp[0]

[0457] - Otherwise (mipSizeId is less than 2), the following applies:

[0458] p[0]=(1<<(BitDepth-1))-pTemp[0]

[0459] p[x]=pTemp[x]-pTemp[0], where x=1..inSize-1

[0460] Here, the variable mipSizeId indicates the size of the predetermined block. That is, according to this embodiment, the reversible transformation used to derive another vector from the sample value vector depends on the size of the predetermined block. The dependency may be given by

[0461] mipSizeId boundarySize predSize 0 2 4 1 4 4 2 4 8

[0462] Where predSize indicates the number of predicted samples within a predetermined block, 2*bondarySize indicates the size of the sample value vector, and is related to inSize (i.e. the size of the other vector) according to inSize=(2*boundarySize)-(mipSizeId==2)? 1:0. More precisely, inSize indicates the number of those components of the other vector that actually participate in the calculation. inSize is as large as the size of the sample value vector for smaller block sizes, and is a smaller component for larger block sizes. In the former case, one component, i.e. the component corresponding to the predetermined component of the other vector, can be ignored, as in the matrix-vector product calculated subsequently, the contribution of the corresponding vector component will produce zero anyway, and therefore, no calculation is actually required. In the case of an alternative embodiment, the dependence on the block size can be ignored, in which only one of the two alternatives is inevitably used, i.e., regardless of the block size (the option corresponding to mipSizeId is less than 2, or the option corresponding to mipSizeId is equal to 2).

[0463] In other words, for example, a predetermined reversible linear transformation is defined such that predetermined components of the other vector p become a, while all other components correspond to the components of the sample value vector minus a, where, for example, a=pTemp[0]. This is easily visible in the case of the first option corresponding to mipSizeId being equal to 2, and only the components of the other vector formed in a differential manner are further considered. That is, in the case of the first option, the other vector is actually {p[0...inSize]; pTemp[0]}, where pTemp[0] is a, and the actually calculated part of the matrix-vector multiplication used to produce the matrix-vector product, i.e. the result of the multiplication, is limited to the inSize components of the other vector and the corresponding columns of the matrix, since the matrix has zero columns that do not need to be calculated. In other cases, corresponding to mipSizeId less than 2, a=pTemp[0] is selected as all components of the other vector except p[0], that is, each of the other components p[x] (where x=1..inSize-1) of the other vector p except the predetermined component p[0] is equal to the corresponding component of the sample value vector pTemp[x] minus a, but p[0] is selected to be a constant minus a. The matrix-vector product is then calculated. The constant is the mean of the representable values, that is, 2 x-1(i.e. 1<<(BitDepth-1)), where x represents the bit depth of the computational representation used. It should be noted that if p[0] was instead chosen to be pTemp[0], the computed product would only deviate from the one computed using p[0] (p[0]=(1<<(BitDepth-1))-pTemp[0]) as indicated above by a constant vector, which constant vector may be taken into account when predicting an internal block based on the product, i.e. the prediction vector. Therefore, the value a is a predetermined value, e.g. pTemp[0]. The predetermined value pTemp[0] is in this case, for example, the component of the sample value vector pTemp corresponding to the predetermined component p[0]. It may be the neighboring sample closest to the upper left corner of the predetermined block at the top of the predetermined block or to the left of the predetermined block.

[0464] For the intra sample prediction process according to predModeIntra, for example, a specified intra prediction mode, the apparatus is configured to be applied to the following steps, for example, to perform at least the first step:

[0465] 1. The matrix-based intra prediction samples predMip[x][y], where x=0..predSize-1, y=0..predSize–1, are derived as follows:

[0466] -The variable modeId is set equal to predModeIntra.

[0467] - Derive the weight matrix mWeight[x][y] by calling the MIP weight matrix derivation process with mipSizeld and modeld as input, where x=0..inSize-1, y=0..predSize*predSize-1.

[0468] - The matrix-based intra prediction samples predMip[x][y] are derived as follows, where x=0..predSize-1, y=0..predSize–1:

[0469]

[0470] oW)>>6)+pTemp[0]

[0471] In other words, the device is configured to calculate a matrix-vector product, which is the product between another vector p[i] or, in the case of mipSizeId equal to 2, {p[i]; pTemp[0]}, and a predetermined prediction matrix mWeight or, in the case of mipSizeId less than 2, a prediction matrix mWeight with additional zero weight columns corresponding to the omitted components of p, to obtain a prediction vector, which has been assigned here to an array of block positions {x, y} distributed inside a predetermined block to produce an array predMip[x][y]. The prediction vector will correspond to the concatenation of the rows of predMip[x][y] or the columns of predMip[x][y], respectively.

[0472] According to one embodiment, or according to a different interpretation, only the component

[0473] is understood as a prediction vector, and the device is configured to calculate the sum of the corresponding component and a (eg pTemp[0]) for each component of the prediction vector when predicting samples of a predetermined block based on the prediction vector.

[0474] The apparatus may optionally be configured to predict samples of a predetermined block based on a prediction vector (eg predMip or Perform additional subsequent steps.

[0475] 2. Matrix-based intra-prediction samples predMip[x][y], where x = 0..predSize–1, y = 0..predSize-1

[0476] For example, it is clipped as follows:

[0477] predMip[x][y]=Clip1(predMip[x][y])

[0478] 3. When isTransposed is equal to TRUE, the predSize×predSize array predMip[x][y], where x =

[0479] 0..predSize-1, y=0..predSize–1, for example, is transposed as follows:

[0480] predTemp[y][x]=predMip[x][y]

[0481] predMip=predTemp

[0482] 4. Predicted samples predSamples[x][y], where x = 0..nTbW-1, y = 0..nTbH-1, for example

[0483] Next export:

[0484] - If nTbW specifies the transform block width greater than predSize or nTbH specifies the transform block height greater than predSize, then use the input block size predSize, matrix-based intra prediction samples predMip[x][y]

[0485] The MIP prediction upsampling process is invoked with the transform block width nTbW, the transform block height nTbH, the top reference sample refT[x] (where x=0..nTbW-1, y=0..predSize-1), the top reference sample refL[y] (where y=0..nTbH-1) as input, and the output is the predicted sample array predSamples.

[0486] Otherwise, predSamples[x][y], x=0..nTbW–1, y=0..nTbH–1, is set equal to predMip[x][y].

[0487] In other words, the device is configured to predict samples predSamples of a predetermined block based on the prediction vector predMip.

[0488] Fig.13 A method 2000 is shown for predicting a predetermined block of an image using multiple reference samples, comprising: forming 2100 a sample value vector from multiple reference samples; deriving 2200 another vector from the sample value vector, the sample value vector being mapped onto the other vector by a predetermined reversible linear transformation; calculating 2300 a matrix-vector product between the other vector and a predetermined prediction matrix to obtain a prediction vector; and predicting 2400 samples of the predetermined block based on the prediction vector.

[0489] References

[0490] [1]P.Helle et al., "Non-linear weighted intra prediction", JVET-L0199, Macao, China, October 2018.

[0491] [2] F.Bossen, J.Boyce, K.Suehring, X.Li, V.Seregin, "JVET common testconditions and software reference configurations for SDR video", JVET-K1010, Ljubljana, SI, July 2018.

[0492] Further embodiments and examples

[0493] Generally, the examples can be implemented as a computer program product with program instructions, which, when the computer program product runs on a computer, are operative for performing one of the methods described. The program instructions may, for example, be stored on a machine-readable medium.

[0494] Other examples comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0495] In other words, an example of the method is, therefore, a computer program having program instructions for performing one of the methods described herein, when the computer program runs on a computer.

[0496] Another example of the method is therefore a data carrier medium (or a digital storage medium, or a computer-readable medium) comprising, on which is recorded the computer program for performing one of the methods described herein. The data carrier medium, the digital storage medium or the recorded medium is tangible and / or non-transitory, rather than an intangible and transitory signal.

[0497] A further example of the method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.The data stream or the sequence of signals may be transmitted, for example, via a data communication connection, for example via the Internet.

[0498] Another example comprises a processing means, for example a computer or a programmable logic device, which performs one of the methods described herein.

[0499] Another example comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0500] Another example includes an apparatus or system for transferring (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. For example, the receiver may be a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, include a file server for transferring the computer program to the receiver.

[0501] In some examples, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functionality of the methods described herein. In some examples, a field programmable gate array can collaborate with a microprocessor to perform one of the methods described herein. Typically, the methods can be performed by any appropriate hardware device.

[0502] The examples described above are merely illustrative of the principles discussed above. It should be understood that modifications and variations of the configurations and details described herein will be apparent. Therefore, it is intended to be limited by the scope of the following claims rather than by the specific details presented by way of the description and explanation of the examples herein. Even if they appear in different figures, equal or equivalent (several) elements with equal or equivalent functionality are represented in the subsequent description by equal or equivalent reference numerals.

Claims

1. A method for predicting an image block, comprising: obtaining one or more sample values ​​from a plurality of adjacent samples located on a boundary of the block; determining an input value from the one or more sample values ​​based on the indication of the block size; determining a matrix-based intra-prediction sample by applying a predetermined prediction matrix to the determined input value; as well as Samples of the block are predicted based on the determined matrix-based intra-prediction samples. 2 . The method of claim 1 , wherein the plurality of adjacent samples are located on a top boundary and a left boundary of the block.

3. The method of claim 1 , wherein determining an input value (p[x]) based on the indication of the block size (mipSizeld) comprises: If mipSizeId is equal to 2, p[x] is determined from the one or more sample values ​​(pTemp[x]) according to the following: p[x] = pTemp[x+1] - pTemp[0]; as well as If mipSizeId is less than 2, p[x] is determined as follows: p[x]=pTemp[x]-pTemp[0], where x = 1, 2, …, inSize - 1, and inSize corresponds to the number of components in p[x].

4. The method of claim 1 , wherein determining a matrix-based intra prediction sample (predMip) by applying a predetermined prediction matrix (mWeight) to the determined input value (p[x]) comprises: as well as Wherein predSize indicates the number of prediction samples within the block, pTemp[0] includes the sample value of the one or more sample values ​​(pTemp[x]), and inSize corresponds to the number of components in p[x].

5. The method of claim 1 , further comprising encoding the block by: deriving a prediction signal based on the prediction samples of the block; and A prediction residual is encoded for the block to correct the prediction signal.

6. The method of claim 1 , further comprising decoding the block by: deriving a prediction signal based on the prediction samples of the block; and decoding a prediction residual for the block; and The prediction signal is corrected using the prediction residual.

7. A device for predicting an image block, the device comprising: non-transitory computer readable medium; as well as At least one processor is configured to execute instructions from the non-transitory computer-readable medium to implement the method according to any one of claims 1-6.

8. A computer program product, comprising a computer program, the computer program being configured to implement the method according to any one of claims 1 to 6 when executed by at least one processor.

9. A computer-readable storage medium having a computer program stored thereon, wherein the computer program has a program code for executing the method according to any one of claims 1 to 6 when the computer is run.

Images