Intra prediction mode concept for block-based image coding.

Abstract

To provide an apparatus supporting at least one intra-prediction mode according to which an intra-prediction signal is determined by applying a first template of samples which neighbors a current block onto a neural network.SOLUTION: An apparatus for block-wise decoding a picture from a data stream 12 and / or encoding a picture into a data stream is configured, for a current block 18 differing from a predetermined size, to: resample a second template 60 of a set 86 of samples neighboring the current block so as to conform with a first template 170 in order to obtain a resampled template; apply the resampled template of the samples onto a neural network 84 so as to obtain a preliminary intra-prediction signal; and resample the preliminary intra-prediction signal so as to conform with the current block in order to obtain an intra-prediction signal for the current block.SELECTED DRAWING: Figure 9a

Description

[Technical field]

[0001] The present application relates to the concept of improved intra-prediction modes for block-based image coding, such as may be used in video codecs such as HEVC or successors of HEVC. [Background technology]

[0002] Intra prediction modes are widely used in image and video coding. In video coding, intra prediction modes compete with other prediction modes, such as inter-prediction modes, such as motion-compensated prediction modes. In intra prediction modes, the current block is predicted based on neighboring samples, i.e., samples that have already been coded as far as the encoder side is concerned and already decoded as far as the decoder side is concerned. The neighboring sample values ​​are extrapolated to the current block to form a prediction signal for the current block, and the prediction residual is transmitted in the data stream for the current block. The better the prediction signal, the smaller the prediction residual, and therefore the fewer bits are needed to code the prediction residual.

[0003] To be effective, several aspects need to be taken into account in order to form an effective framework for intra prediction in a block-based image coding environment. For example, the more intra prediction modes supported by a codec, the more side information rate is consumed to inform the decoder of the selection. On the other hand, the set of supported intra prediction modes needs to be able to provide a good prediction signal, i.e. a prediction signal that results in a low prediction residual. Summary of the Invention [Problem to be solved by the invention]

[0004] This application seeks to provide an improved intra-prediction mode concept that allows for more efficient compression for block-based image codecs when using the intra-prediction mode concept. [Means for solving the problem]

[0005] This object is achieved by the subject matter of the independent claims of the present application.

[0006] 1. An apparatus (e.g., a decoder) for block-wise decoding of an image from a data stream, the apparatus supporting at least one intra prediction mode, in which an intra prediction signal of a block of a predetermined size of the image is determined by applying a first template of samples neighboring a current block to a neural network, the apparatus comprising: resampling a second template of samples adjacent to the current block to match the first template to obtain a resampled template; Apply the resampled template of the samples to a neural network to obtain a preliminary intra prediction, An apparatus is disclosed that is configured to resample a preliminary intra-prediction signal to match a current block to obtain an intra-prediction signal of the current block.

[0007] 1. An apparatus (e.g. an encoder) for block-wise encoding of an image into a data stream, the apparatus supporting at least one intra prediction mode, in which an intra prediction signal for a block of a predetermined size of the image is determined by applying a first template of samples neighboring the current block to a neural network, the first template being determined for a current block of a different size from the predetermined size, resampling a second template of samples adjacent to the current block to match the first template to obtain a resampled template; Apply the resampled template of the samples to a neural network to obtain a preliminary intra prediction, An apparatus is also disclosed that is configured to resample a preliminary intra-prediction signal to match a current block to obtain an intra-prediction signal of the current block.

[0008] The apparatus may be configured to resample by downsampling the second template to obtain the first template.

[0009] The apparatus may be configured to resample the preliminary intra prediction signal by upsampling the preliminary intra prediction signal.

[0010] The apparatus may be configured to transform the preliminary intra prediction signal from a spatial domain to a transform domain and to resample the preliminary intra prediction signal in the transform domain.

[0011] The apparatus may be configured to resample the transform domain preliminary intra prediction signal by scaling coefficients of the preliminary intra prediction signal.

[0012] The device is Increase the dimension of the intra prediction signal to match the dimension of the current block; zero-padding the added coefficients of the preliminary intra-prediction signal, the added coefficients being associated with higher frequency bins; The transform domain preliminary intra prediction signal may be configured to be resampled by

[0013] The apparatus may be configured to construct a transform domain preliminary intra prediction signal by an inverse quantized version of the prediction residual signal.

[0014] The apparatus may be configured to resample the preliminary intra prediction signal in the spatial domain.

[0015] The apparatus may be configured to resample the preliminary intra prediction signal by performing bilinear interpolation.

[0016] The apparatus may be configured to encode information regarding resampling and / or the use of neural networks of different dimensions in the data field.

[0017] 1. An apparatus (e.g., a decoder) for block-wise decoding an image from a data stream, comprising: An apparatus is also disclosed that supports at least one intra prediction mode in which an intra prediction signal for a current block of an image is determined by applying a first set of neighboring samples of the current block to a neural network to obtain a prediction of a set of transform coefficients of a transform of the current block.

[0018] 1. An apparatus (e.g., an encoder) for block-wise encoding of an image into a data stream, comprising: An apparatus is also disclosed that supports at least one intra prediction mode in which an intra prediction signal for a current block of an image is determined by applying a first set of neighboring samples of the current block to a neural network to obtain a prediction of a set of transform coefficients of a transform of the current block.

[0019] One of the devices can be configured to inverse transform the prediction to obtain a reconstructed signal.

[0020] One of the devices may be configured to decode an index from the data stream using a variable length code and to perform a selection using the index.

[0021] One of the devices may be configured to determine a ranking of the set of intra-prediction modes and then resample the second template.

[0022] resampling a second template of samples adjacent to the current block to obtain a resampled template conforming to the first template; applying the resampled template of samples to a neural network to obtain a preliminary intra prediction signal; resampling the preliminary intra-prediction signal to match the current block to obtain an intra-prediction signal of the current block; A method is disclosed comprising:

[0023] 1. A method for block-wise decoding of an image from a data stream, comprising the steps of: A method is disclosed that comprises applying a first set of neighboring samples of a current block to a neural network to obtain a prediction of a set of transform coefficients of a transform of the current block.

[0024] 1. A method for block-wise encoding of an image into a data stream, comprising the steps of: A method is disclosed that comprises applying a first set of neighboring samples of a current block to a neural network to obtain a prediction of a set of transform coefficients of a transform of the current block.

[0025] The above and / or following methods may use an apparatus comprising at least one of the above and / or following devices.

[0026] Also disclosed is a computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the above and / or the following methods and / or to implement the above and / or the following in at least one component of an apparatus.

[0027] Also disclosed is a data stream obtained by the above and / or the following method and / or by the above and / or the following device.

[0028] As far as the design of the above mentioned neural network is concerned, the present application provides many examples for properly determining its parameters.

[0029] Advantageous implementations of the present application are the subject of the dependent claims. Preferred examples of the present application are described below with reference to the figures. [Brief description of the drawings]

[0030] [Figure 1] 1 shows a schematic block diagram illustrating an encoder for encoding an image into a data stream, as a general example in which the examples of the present application can be implemented. [Diagram 2] FIG. 1 shows a block diagram of a more specific example of such an encoder. [Diagram 3] 2 shows a schematic block diagram of a decoder that matches the encoder of FIG. 1 and serves as an example of a decoder in which examples of the present application can be implemented. [Figure 4] 4 shows a block diagram of a more specific example of the decoder of FIG. 3, which is compatible with the encoder of FIG. 2; [Diagram 5] 1 shows a schematic diagram illustrating the operation modes of an encoder and a decoder according to an example of the present application with respect to processing blocks using intra prediction; [Figure 6] 1 shows a schematic block diagram of a decoder according to an example of the present application including several neural network based intra prediction modes. [Figure 7a]9 shows schematic diagrams illustrating operation modes of an example encoder and decoder that support neural network-based intra-prediction modes and neural network-based ordering of these modes by transmitting an index in the data stream to an ordered list of neural network-based intra-prediction modes, together with a frequency indicating whether the intra-prediction mode used is a member of a set of neural network-based intra-prediction modes. It will be appreciated that the index may be coded using variable length coding to take advantage of the different frequencies determined by decision 90. [Figure 7b] 7a in that no freq signalling is used. [Figure 7c] FIG. 7B shows a schematic diagram that differs from FIG. 7b in that the mode ordering is not controlled using a neural network. [Figure 7d] FIG. 7B shows a schematic diagram that differs from FIG. 7b in that the mode ordering is not controlled using a neural network. [Figure 8] 1 illustrates an apparatus for designing a set of neural network-based intra-prediction modes according to an example. [Figure 9a] FIG. 1 shows a schematic diagram illustrating the operation modes of an encoder and a decoder according to an example in which a neural network, whether neural network-based or not, is used to order the supported intra-prediction modes. [Figure 9b] This shows a schematic diagram that differs from FIG. 9a in that a neural network base is used to control the probability distribution estimation for the entropy decoding / encoding of indices into the set of supported intra-prediction modes. [Figure 10] 1 shows an apparatus for designing a neural network for assisting and selecting among a set of intra-prediction modes for block-based image coding according to an example. [Figure 11-1] 1 illustrates an example encoder. [Figure 11-2] 1 shows an example decoder. [Figure 12] 1 shows a schematic diagram illustrating the operation modes of an encoder and a decoder according to an example; [Figure 13a] 1 shows a schematic diagram of an example technique. [Figure 13b] 1 shows a schematic diagram of an example technique. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] In the following, various examples are described that help achieve more effective compression when using intra prediction. Some examples achieve improved compression efficiency by using a set of neural network-based intra prediction modes. The latter can be in addition to other intra prediction modes, for example heuristically designed, or can be provided exclusively. Other examples use neural networks to perform selection among multiple intra prediction modes. And even other examples take advantage of both of the specialties described herein.

[0032] To facilitate the understanding of the following examples of this application, the description begins with the presentation of possible encoders and decoders to which the examples outlined thereafter of this application can be built. Figure 1 shows an apparatus for block-wise encoding of an image 10 into a data stream 12. The apparatus is indicated using the reference number 14 and can be a still image encoder or a video encoder. In other words, the image 10 can be a current image from a video 16 if the encoder 14 is configured to encode a video 16 containing the image 10 into the data stream 12, or if the encoder 14 can exclusively encode the image 10 into the data stream 12.

[0033] As mentioned above, the encoder 14 performs the encoding in a block-by-block manner or on a block basis. To this end, the encoder 14 subdivides the image 10 into blocks, which units of the encoder 14 encode the image 10 into a data stream 12. Examples of possible subdivisions of the image 10 into blocks 18 are given in more detail below. In general, the subdivision may result in blocks 18 of a fixed size, such as an array of blocks arranged in rows and columns, or blocks 18 of different block sizes, such as by using a hierarchical multi-tree subdivision starting from the entire image area of ​​the image 10 or from a pre-partition of the image 10 into an array of tree blocks, and these examples should not be treated as excluding other possible ways of subdividing the image 10 into blocks 18.

[0034] Moreover, the encoder 14 is a predictive encoder configured to predictively encode the image 10 into the data stream 12. For a particular block 18, this means that the encoder 14 determines a prediction signal for the block 18 and encodes into the data stream 12 the prediction residual, i.e. the prediction error where the prediction signal deviates from the actual image content in the block 18.

[0035] The encoder 14 may support different prediction modes to derive a prediction signal for a particular block 18. The prediction mode of interest in the following example is an intra prediction mode, where the interior of the block 18 is spatially predicted from neighboring, already coded samples of the image 10. The coding of the image 10 into the data stream 12, and therefore the corresponding decoding procedure, may be based on a particular coding order 20 defined among the blocks 18. For example, the coding order 20 may traverse the blocks 18 in a raster scan order, such as row-by-row from top to bottom, while traversing each row from left to right. In case of a hierarchical multi-tree based subdivision, a raster scan order may be applied within each hierarchical level, and a depth-first traversal order may be applied. That is, leaf notes in a block of a particular hierarchical level precede blocks of the same hierarchical level that have the same parent block according to the coding order 20. Depending on the coding order 20, neighboring, already coded samples of the block 18 may be typically located on one or more sides of the block 18. In the example presented here, for example, the neighboring, already coded samples of block 18 are located above and to the left of block 18 .

[0036] Intra-prediction modes may not be the only modes supported by the encoder 14. For example, if the encoder 14 is a video encoder, the encoder 14 may also support intra-prediction modes in which the blocks 18 are temporally predicted from previously encoded images of the video 16. Such intra-prediction modes may be motion-compensated prediction modes in which a motion vector is signaled for such blocks 18 indicating the relative spatial offset of the portion of which the prediction signal of the block 18 is derived as a copy. Additionally or alternatively, other non-intra-prediction modes may also be available, such as inter-view prediction modes in the case where the encoder 14 is a multiview encoder, or non-prediction modes in which the interior of the block 18 is coded as is, i.e., without prediction.

[0037] Before beginning the description of this application by focusing on intra-prediction modes, we will describe a more specific example of a possible block-based encoder, i.e., a possible implementation of encoder 14 as described with respect to FIG. 2, and then present two corresponding examples of decoders compatible with FIGS. 1 and 2, respectively.

[0038] Fig. 2 shows a possible implementation of the encoder 14 of Fig. 1, i.e. the encoder is configured to use transform coding to code the prediction residual, but this is mostly by way of example and the present application is not limited to such a type of prediction residual coding. According to Fig. 2, the encoder 14 comprises a subtractor 22 configured to subtract a corresponding prediction signal 24 from an inbound signal, i.e. the image 10, or on a block basis, a current block 18, to obtain a prediction residual signal 26 which is subsequently coded into the data stream 12 by a prediction residual encoder 28. The prediction residual encoder 28 comprises a lossy coding stage 28a and a lossless coding stage 28b. The lossy stage 28a comprises a quantizer 30 which receives the prediction residual signal 26 and quantizes the samples of the prediction residual signal 26. As already mentioned above, the present example uses a transform coding of the prediction residual signal 26, therefore the lossy coding stage 28a comprises a transformation stage 32 connected between the subtractor 22 and the quantizer 30 so as to transform such prediction residual 26, spectrally decomposed, by a quantization of the quantizer 30 performed on the transformed coefficients presenting the residual signal 26. The transformation can be a DCT, a DST, an FFT, a Hadamard transform, etc. The transformed and quantized prediction residual signal 34 then undergoes lossless coding by a lossless coding stage 28b, which is an entropy coder that entropy codes the quantized prediction residual signal 34 into the data stream 12. The encoder 14 further comprises a prediction residual signal reconstruction stage 36 connected to the output of the quantizer 30 so as to reconstruct the prediction residual signal from the transformed and quantized prediction residual signal 34 in a way that is also usable by the decoder. That is to say, it is the quantizer 30 that takes into account the coding losses. To this end, the prediction residual reconstruction stage 36 comprises an inverse quantizer 38 which performs the inverse of the quantization of the quantizer 30, followed by an inverse transformer 40 which performs an inverse transformation to the transformation performed by the transformer 32, such as the inverse of the spectral decomposition, such as the inverse of any of the specific transformation examples mentioned above. The encoder 14 comprises an adder 42 which adds the reconstructed prediction residual signal output by the inverse transformer 40 with the prediction signal 24 to output a reconstructed signal, i.e. reconstructed samples.This output is provided to a predictor 44 of the encoder 14, which determines the prediction signal 24 based thereon. It is the predictor 44 that supports all prediction modes already mentioned above with respect to Figure 1. Figure 2 also shows that, if the encoder 14 is a video encoder, the encoder 14 may also include an in-loop filter 46 that filters the fully reconstructed image that, after being filtered, forms the reference image of the predictor 44 for the inter-prediction blocks.

[0039] As already mentioned above, the encoder 14 operates on a block basis. In the following description, the block basis of interest is a subdivision of the image 10 into blocks for which an intra prediction mode is selected from a set or a number of intra prediction modes supported by the predictor 44 or the encoder 14, respectively, and the selected intra prediction mode is executed individually. However, there may be other types of blocks into which the image 10 is subdivided as well. For example, the above determination regarding whether the image 10 is inter-coded or intra-coded can be made with granularity or on a block-by-block basis deviating from the block 18. For example, the inter-mode / intra-mode decision can be performed at the level of the coding blocks into which the image 10 is subdivided and each coding block is subdivided into predictive blocks. The predictive blocks having coding blocks for which it is determined that intra prediction is used are each subdivided into an intra prediction mode decision. In response, for each of these predictive blocks, it is determined which supported intra prediction mode is to be used for the respective predictive block. These predictive blocks form the block 18 of interest here. Prediction blocks in a coding block related to inter-prediction will be treated differently by the predictor 44. They will be inter-predicted from a reference image by determining a motion vector and copying the prediction signal of this block from the location in the reference image pointed to by the motion vector. Another block subdivision concerns the subdivision into transform blocks in the units where the transformation by the transformer 32 and the inverse transformer 40 is performed. The transformed blocks can be, for example, the result of further subdivision of the coding block. Naturally, the examples described here should not be treated as limiting and other examples exist. Just for the sake of completeness, it is noted that the subdivision into coding blocks can, for example, use multi-tree subdivision and, similarly, the prediction blocks and / or transform blocks can be obtained by further subdivision of the coding block using multi-tree subdivision.

[0040] A decoder or device for block-wise decoding that fits the encoder 14 of FIG. 1 is shown in FIG. 3. This decoder 54 does the opposite to the encoder 14, i.e. it decodes the image 10 from the data stream 12 block-wise and for this purpose supports several intra prediction modes. The decoder 54 may, for example, include a residual provider 156. All other possibilities discussed above with respect to FIG. 1 are also valid for the decoder 54. To this end, the decoder 54 may be a still image decoder or a video decoder, and all prediction modes and predictability are also supported by the decoder 54. The difference between the encoder 14 and the decoder 54 mainly consists in the fact that the encoder 14 selects or chooses the coding decision according to some optimization, for example to minimize some cost function that may depend on the coding rate and / or the coding distortion. One of these coding options or coding parameters may include the selection of the intra prediction mode to be used for the current block 18 from among the available or supported intra prediction modes. The selected intra-prediction mode is then signaled by the encoder 14 of the current block 18 in the data stream 12, and the decoder 54 redoes the selection using this signaling of the data stream 12 of the block 18. Similarly, the subdivision of the image 10 into blocks 18 can be the subject of optimization in the encoder 14, and corresponding subdivision information can be conveyed in the data stream 12, and the decoder 54 restores the subdivision of the image 10 into blocks 18 based on the subdivision information. To summarise the above, the decoder 54 can be a predictive decoder operating on a block basis, and in addition to the intra-prediction mode, the decoder 54 can support other prediction modes, such as, for example, an inter-prediction mode if the decoder 54 is a video decoder. In decoding, the decoder 54 can also use the coding order 20 described with reference to FIG. 1, which is followed in both the encoder 14 and the decoder 54, so that the same neighbouring samples are available for the current block 18 in both the encoder 14 and the decoder 54.Therefore, in order to avoid unnecessary repetitions, the description of the operating modes of the encoder 14 must also apply to the decoder 54 as far as the subdivision of the image 10 into blocks is concerned, for example as far as the prediction and as far as the coding of the prediction residuals is concerned. The difference lies in the fact that the encoder 14 selects in or inserts into the data stream 12, by optimization, several coding options or coding parameters and signals, which are derived from the data stream 12 by the decoder 54 in order to redo the prediction, such as the subdivision.

[0041] FIG. 4 shows a possible implementation of the decoder 54 of FIG. 3, i.e. one that matches the implementation of the encoder 14 of FIG. 1, as shown in FIG. 2. Many elements of the encoder 54 of FIG. 4 are the same as those occurring in the corresponding encoder of FIG. 2, so the same reference numbers with an apostrophe are used in FIG. 4 to indicate these elements. In particular, the adder 42', the optional in-loop filter 46' and the predictor 44' are connected in the prediction loop in the same way as they are in the encoder of FIG. 2. The reconstructed, i.e. inverse quantized and retransformed prediction residual signal applied to the adder 42' is derived by a sequence of an entropy decoder 56 that reverses the entropy coding of the entropy encoder 28b, followed by a residual signal reconstruction stage 36', composed of an inverse quantizer 38' and an inverse transformer 40' as in the case of the encoding side. The output of the decoder is a reconstruction of the image 10. The reconstruction of the image 10 may be available directly at the output of the adder 42' or alternatively at the output of the in-loop filter 46'. In order to improve the image quality, some postfilters can be placed at the output of the decoder to subject the reconstruction of the image 10 to some postfiltering, although this option is not shown in FIG.

[0042] Again, with respect to Figure 4, the explanations given above with respect to Figure 2 are valid for Figure 4, except that the encoder only performs optimization tasks and related decisions regarding coding options. However, all the explanations regarding block subdivision, prediction, inverse quantization, and retransformation are also valid for the decoder 54 of Figure 4.

[0043] Before proceeding with the description of possible examples of the present application, some remarks should be made regarding the above examples. Although not explicitly mentioned above, it is clear that the block 18 can have any shape. It can be, for example, rectangular or quadratic. Furthermore, although the above description of the operating modes of the encoder 14 and the decoder 54 often refers to the "current block" 18, it is clear that the encoder 14 and the decoder 54 act accordingly for each block for which an intra-prediction mode is selected. As mentioned above, there are other blocks possible, but the following description focuses on the block 18 for which the image 10 is subdivided and for which an intra-prediction mode is selected.

[0044] To summarize the situation of a particular block 18 for which an intra-prediction mode is selected, reference is made to FIG. 5. FIG. 5 shows a current block 18, i.e., a block currently being coded or decoded. FIG. 5 shows a set 60 of neighboring samples 62, i.e., samples 62 with spatially neighboring blocks 18. Samples 64 in the block 18 are to be predicted. The derived predicted signal is therefore a prediction of each sample 64 in the block 18. As already mentioned above, a plurality of 66 prediction modes are available for each block 18, and this plurality of 66 modes simply includes inter-prediction modes if the block 18 is intra-predicted. A selection 68 is performed on the encoder side and on the decoder side to determine one of the intra-prediction modes from the plurality 66 that is used to predict (71) the predicted signal of the block 18 based on the neighboring sample set 60. The examples further described below differ with respect to the available intra-prediction modes 66 and the operating mode for the selection 68, e.g., whether side information is set in the data stream 12 for the selection 68 for the block 18. However, the description of these examples begins with a specific description that provides mathematical details. According to this first example, the selection of a particular block 18 to be intra predicted is associated with a corresponding side information signaling 70 and data stream, and the plurality of 66 intra prediction modes include a set of neural network based intra prediction modes 72 and a set of further intra prediction modes of heuristic design 74. One of the intra prediction modes of the set 74 can be, for example, a DC prediction mode in which a certain average value is determined based on neighboring sample sets 60 and this average value is assigned to all samples 64 in the block 18. Additionally or alternatively, the set 74 can include an inter prediction mode, which may be referred to as an angular inter prediction mode, in which sample values ​​of neighboring sample sets 60 are copied to the block 18 along a particular intra prediction direction with this intra prediction direction differing between such angular intra prediction modes.FIG. 5 shows that the data stream 12 includes, in addition to optional side information 70 regarding the selection 68 of a plurality of 66 intra prediction modes, a portion 76 in which the prediction residual is coded, which coding may optionally include transform coding with quantization in the transform domain, as described above.

[0045] In particular, to facilitate understanding of the following description of a specific example of the present application, FIG. 6 illustrates a general mode of operation of an intra prediction block in an encoder and a decoder. FIG. 6 illustrates a block 18 and a set of neighboring samples 60 based on which intra prediction is performed. It should be noted that this set 60 may vary in cardinality between the intra prediction modes of the plurality of 66 intra prediction modes. That is, the number of samples in the set 60 is actually used according to the respective intra prediction mode for determining the prediction signal of the block 18. However, this is for ease of understanding and is not illustrated in FIG. 6. FIG. 6 illustrates that the encoder and the decoder use one neural network 800 to 80 for each of the neural network-based intra prediction modes of the set 72. KB 6 shows that the set 60 has a corresponding intra-prediction mode among the sets 72. The set 60 is applied to the respective neural network in order to derive a corresponding intra-prediction mode among the sets 72. In addition, FIG. 6 shows quite representatively one block 82 as providing based on the input, i.e. the set 60 of neighboring samples, one or more prediction signals of one or more intra-prediction modes of the set 74, such as a DC mode prediction signal and / or an angular intra-prediction mode prediction signal. The following description is for i=0...K. B Neural network 80 with -1 ican be advantageously determined. The particular example shown below also provides the encoder and decoder with another neural network 84 dedicated to providing a probability value for each neural network-based intra-prediction mode in the set 72 based on a set 86 of neighboring samples that may or may not match the set 60. Thus, the probability values ​​are provided when the neural network 84 assists the neural network 84 to render the side information 70 for mode selection more effectively. For example, in the example described below, a variable length code is used to indicate one of the intra-prediction modes, and at least as far as the set 72 is concerned, the probability value provided by the neural network 84 uses the variable length code in the side information 70 as an index into an ordered list of intra-prediction modes ordered according to the probability values ​​output by the neural network 84 for the neural network-based intra-prediction modes in the set 72, thereby optimizing or reducing the code rate of the side information 70. Thus, as shown in FIG. 6, the mode selection 68 is effectively performed in response to both the probability values ​​provided by the further neural network 84 and the side information 70 in the data stream 12. 1. An algorithm to train the parameters of the neural network that performs intra-prediction A block of the video frame, i.e. block 18,

number

number

number

number

number

number

number

number

number

[0046] What follows is a data-driven optimization approach to some of the possible hybrid video coding standards, i.e. set 72.

number

[0047] 1. In the optimization algorithm we implement, we want to use a good approximation of the cost function, especially including the number of bits we can expect to spend on informing the prediction residual.

[0048] 2. We want to jointly train several intra predictors to be able to handle different signal characteristics.

[0049] 3. When training intra prediction, you need to consider the number of bits required to inform which intra mode to use.

[0050] 4. Keep the set of intra predictions already defined, e.g., the HEVC intra predictions, and train our predictions as complementary predictions.

[0051] 5. Typical hybrid video coding standards usually use a specific block

number

[0052] In the next four sections we can explain how each of these requirements can be addressed. More precisely, section 1.1 explains how to deal with the first item; section 1.2 explains how to deal with items 2-3; section 1.4 explains how to take item 4 into account; and finally, section 1.5 explains how to deal with the last item. 1.1 Algorithm for training a loss function that approximates the rate function of a video codec Data-driven approaches to determining the unknown parameters used in video codecs are usually set up as optimization algorithms that try to minimize a predefined loss function on a given set of training examples. Typically, for a numerical optimization algorithm to work in practice, the latter loss function needs to satisfy some smoothness requirements.

[0053] On the other hand, video encoders like HEVC offer a high degree of rate-distortion cost.

number

number

number

number

[0054] True Function

number

number

number

[0055] More precisely, as before,

number

number

number

number

number

number

number

number

number

number

number

number

[0056] function

number

number

number

number

number

[0057] Weight

number

number

number

number

number

number

[0058] For that task, we typically use (stochastic) gradient descent. 1.2 Training fixed block shape predictions In this section, specific blocks

number

number

[0059] We assume that we are given a predefined "architecture" of our predictions, which allows us to model some fixed

number

number

number

number

number

number

[0060] The following section explains this point in detail. The function (2) is the neural network 800-800 in Figure 6. KB Define -1.

[0061] Then, the second parameter-dependent function

number

[0062] Similarly,

number

number

number

[0063] Similarly, an example using function (4) representing neural network 84 of FIG. 6 is given in Section 1.3.

[0064] function

number

[0065] This function defines, for example, the VLC code length distribution used for the side information 70, i.e. the code lengths associated with the side information 70 having more cad ponites of the set 72.

[0066] next,

number

number

[0067] For the time being,

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0068] The gradient update involves calculating the proportion of residuals and the probability distribution of the latter.

number

number

number

number

[0069] Given the loss function in (5), we can obtain

number

number

number

number

number

number

number

number

number

number

[0070]

number

number

number

number

number

number

number

number

number

number

number

number

[0071] Nonlinear activation functions

number

number

[0072] Where:

number

number

number

number

number

number

number

[0073] function

number

number

number

number

[0074] Where:

number

number

number

number

[0075] therefore,

number

number

number

number

number

number

number

number

number

number

number

number

[0076] Similarly, the function

number

number

number

number

[0077] Where:

number

number

number

number

number

number

number

[0078] That is,

number

number

number

number

number

[0079] Next, we convert the loss function from (5) to the loss function

number

[0080] For a large set of training examples, we keep the notation from the end of the previous section,

number

number

[0081] To do this, we typically first find the weights by optimization (6), then initialize with those weights to find the weights to optimize (10). 1.5 Joint training of predictions for several block shapes In this section, we have described how training the prediction takes into account the fact that in common video coding standards it is usually possible to divide blocks into smaller sub-blocks in various ways and perform intra prediction on the smaller sub-blocks.

[0082] That is, some

number

number

number

number

number

[0083]

number

number

number

number

number

number

number

number

[0084] For a given color component,

number

number

number

number

[0085] While keeping the notation of section 1.2,

number

number

number

number

number

number

number

[0086] moreover,

number

number

number

[0087] As in section 1.4,

number

number

number

[0088] next,

number

number

number

number

number

number

[0089] next,

number

[0090] next,

number

[0091] lastly,

number

number

number

number

[0092] Usually, first

number

number

number

number

number

number

number

number

number

number

number

number

number

[0093] 2. The decoder reconstructs from the bitstream a flag that is part of the side information 70, indicating whether any of the following options are true: [label=

[0094] (i)

number

[0095] (ii)

number

number

[0096] 3. If option 2 of step 2 is true, the decoder proceeds to designated block 10, just as in the underlying hybrid video coding standard.

[0097] 4. If option 1 of step 2 is true, the decoder

number

number

number

number

[0098] (i) a decoder,

number

number

number

number

number

[0099] (ii) a decoder,

number

number

number

number

number

number

[0100] The decoder then extracts from the bitstream 12 a unique

number

[0101] The latter index

number

number

number

number

[0102] 5. Option 1 of step 2 is true and the decoder indexes according to the previous step 4.

number

number

number

[0103] Integrating an intra prediction function designed based on a data-driven learning approach into an existing hybrid video codec. The description had two main parts. In the first part, a specific algorithm for offline training of the intra prediction function was described. In the second part, how the video decoder uses the latter prediction function to generate a prediction signal for a particular block was described.

[0104] Thus, what has been described above in sections 1.1 to 1.2 is, inter alia, an apparatus for block-wise decoding of an image 10 from a data stream 12. The apparatus 54 comprises at least an intra prediction signal of a current block 18 of the image 10, the intra prediction signal being fed to a neural network 80. i The apparatus 54 supports multiple intra-prediction modes, including a set 72 of intra-prediction modes determined by applying a first set 60 of neighboring samples of the current block 18 to the first set 62 of neighboring samples of the current block 18. The apparatus 54 selects (68) an intra-prediction mode for the current block 18 from the multiple intra-prediction modes 66 and performs a neural network simulation using the one intra-prediction mode, i.e., the selected corresponding neural network 80. m 5. The decoder 54 is configured to predict (71) the current block 18 using a neural network based decoder 84. Although the decoder presented in Section 2 had an intra-prediction mode 74 among the plurality 66 of supported intra-prediction modes in addition to the neural network based one in set 72, this is merely an example and need not be the case. Furthermore, the above description in Sections 1 and 2 may be modified in that the decoder 54 does not use or include a further neural network 84. With respect to the optimizations above, this may be achieved by using the findings

number

[0105] A further alternative to the description presented in section 2 above is that the decoder 54 may instead derive a ranking among the set of neural network-based intra-prediction modes 72 in response to a first portion of the data stream related to neighbors of the current block 18, to select an intra-prediction mode to be ultimately used from the ordered list of intra-prediction modes in response to a second portion other than the first portion of the data stream to obtain an ordered list of intra-prediction modes. The "first portion" may, for example, relate to coding or prediction parameters related to one or more blocks neighboring the current block 18. And the "second portion" may, for example, be an index pointing to or an index of the neural network-based intra-prediction mode set 72. When interpreted consistent with section 2 outlined above, the decoder 54 comprises a further neural network 84 that determines a probability value for each intra-prediction mode of the set of intra-prediction modes 72 by applying thereon a set of neighboring samples 86, in order to order these probability values ​​to determine a rank for each intra-prediction mode of the set 72, thereby obtaining an ordered list of intra-prediction modes. An index in the data stream 12 as part of the side information 70 is then used as an index into the ordered list, where the index is M Bcan be coded using a variable length code indicating the code length. And as explained above in section 2, in item 4i, according to a further alternative, the decoder 54 can use the above probability values ​​determined by the further neural network 84 for each neural network-based intra-prediction mode of the set 72 to efficiently perform entropy coding of the index into the set 72. In particular, the symbol alphabet of this index, which is part of the side information 70 and is used as an index into the set 72, includes the symbols or values ​​of each mode in the set 72, and the probability values ​​provided by the neural network 84 provide probability values ​​that lead to efficient entropy coding in that, for the design of the neural network 84 according to the above description, these probability values ​​closely represent the actual symbol statistics. For this entropy coding, for example, arithmetic coding or probability interval partitioning entropy (PIPE) coding can be used.

[0106] Advantageously, no additional information is required for any of the intra-prediction modes in the set 72. iderives a prediction signal for the current block 18 without any additional guidance in the data stream, when advantageously parameterized for the encoder and decoder, for example according to the above description in sections 1 and 2. As already indicated above, the presence of other intra prediction modes other than the neural network-based modes of set 72 is optional. They are indicated above by set 74. In this regard, it is noted that one possible way of selecting the set 60, i.e. the set of neighboring samples forming the input of intra prediction 71, can be such that this set 60 is the same for the intra prediction modes of set 74, i.e. heuristic. The set 60 of neural network-based intra prediction modes is large in terms of the number of neighboring samples that are included in the set 60 and that influence the intra prediction 71. In other words, the cardinality of the set 60 can be large for the neural network-based intra prediction mode 72 compared to the other modes of set 74. For example, the set 60 of any intra prediction mode of set 74 can simply include neighboring samples along a one-dimensional line extending along a side of the block 18, such as the left one and the top one. The set of neural network-based intra-prediction modes 60 extends along the just-mentioned sides of block 18, but may cover an L-shaped portion that is wider than one sample wide, such as the set of intra-prediction modes 60 of set 74. The L-shaped portion may extend further beyond the just-mentioned sides of block 18. In this manner, the neural network-based intra-prediction modes may result in better intra prediction with a correspondingly lower prediction residual.

[0107] As described above in section 2, side information 70 conveyed in data stream 12 to intra-prediction block 18 may include a flag that generally indicates whether the intra-prediction mode selected for block 18 is a member of set 72 or a member of set 74. However, this flag is merely optional, with side information 70 indicating, for example, an index into the entire plurality of intra-prediction modes 66 that includes both sets 72 and 74.

[0108] In the following, the just described alternatives are briefly described with reference to Fig. 7a to Fig. 7d. The figures define both the decoder and the encoder simultaneously, i.e. in terms of their functions with respect to the intra prediction block 18. The difference between the encoder and the decoder operation modes with respect to the intra coding block 18 is the fact that, on the one hand, the encoder runs all or at least some of the available intra prediction modes 66 and determines at 90 the best one, for example in terms of a cost function that minimizes the meaning, and the encoder forms the data stream 12, i.e. the code dates therein, and the decoder derives the data therefrom by decoding and reading, respectively. Fig. 7a shows the operation modes of the above outlined alternatives, in which a flag 70a in the side information 70 of the block 18 indicates whether the intra prediction mode determined to be the best mode for the block 18 by the encoder in step 90 is one of the sets 72, i.e. the neural network based intra prediction modes, or one of the sets 74, i.e. the non-neural network based intra prediction modes. The encoder inserts the flag 70a into the data stream 12 accordingly, while the decoder retrieves the flag 70a from there. Fig. 7a assumes that the determined intra-prediction mode 92 is in the set 72. Then, a separate neural network 84 determines a probability value for each neural network-based intra-prediction mode of the set 72, and using these probability values ​​set 72, or more precisely, the neural network-based intra-prediction modes therein are ordered according to their probability values, such as in descending order of probability values, thereby resulting in an ordered list 94 of intra-prediction modes. Then, the index 70b, which is part of the side information 70, is encoded by the encoder into the data stream 12 and decoded therefrom by the decoder. Thus, the decoder can determine which set of the sets 72 and 74. The intra-prediction mode used for the block 18 is located to perform an ordering 96 of the set 72 if the intra-prediction mode used is located in the set 72.If the determined intra-prediction mode is located in set 74, the index may also be transmitted in data stream 12. Thus, the decoder may generate a prediction signal for block 18 using the determined intra-prediction mode by controlling selection 68 accordingly.

[0109] FIG. 7b illustrates an alternative where flag 70a is not present in data stream 12. Instead, ordered list 94 would include not only intra-prediction modes of set 72, but also intra-prediction modes of set 74. The index in side information 70 is an index into this larger ordered list and indicates the determined intra-prediction mode, i.e., the one determined is the optimum 90. In the case of neural network 84 only providing probability values ​​for neural network-based intra-prediction modes in 72, the ranking among intra-prediction modes of set 72 relative to intra-prediction modes of set 74 can be determined by other means, such as necessarily placing the neural network-based intra-prediction modes of set 72 to precede the modes of set 74 in ordered list 94, or to alternate between them with each other. That is, the decoder can derive the index from data stream 12 and use index 70 as an index into ordered list 94 by deriving ordered list 94 from multiple intra-prediction modes 66 using the probability values ​​output by neural network 84. FIG. 7c illustrates a further variation. Figure 7c illustrates the case where flag 70a is not used, although a flag could be used instead. The problem addressed by Figure 7c concerns the possibility that neither the encoder nor the decoder uses neural network 84. Rather, ordering 96 is derived by other means, such as coding parameters conveyed within data stream 12 with respect to one or more neighboring blocks 18, i.e., portions 98 of data stream 12 that relate to such one or more neighboring blocks.

[0110] Figure 7d shows a further variation of Figure 7a, where index 70b is encoded using entropy coding and decoded from data stream 12 using entropy decoding, generally indicated using reference numeral 100. The sample statistics or probability distribution used for entropy coding 100 is controlled by the probability values ​​output by neural network 84 as explained above, which makes the entropy coding of index 70b very efficient.

[0111] It is true that for all examples 7a to 7d, a mode of set 74 may not exist, and therefore the respective module 82 may be missing, and flag 70a would not be needed anyway.

[0112] Moreover, although not shown in any of the figures, it is clear that the mode selection 68 at the encoder and decoder can be synchronized with each other without explicit signaling 70, i.e., without consuming side information. Rather, the selection can be derived from other means, such as by taking the first one in the ordered list 94 necessarily, or by deriving an index into the ordered list 94 based on coding parameters associated with one or more neighboring blocks. FIG. 8 shows an apparatus for designing a set of intra-prediction modes of the set 72 used for block-based image coding. The apparatus 108 includes neural networks 800 to 800. KB-1 , as well as a parameterizable network 109 that inherits or includes neural network 84. Here, in FIG. 8, we have a parameterizable version of the neural network based intra-prediction mode 1 for providing probability values ​​for neural network based intra-prediction mode 0, a parameterizable version of the neural network based intra-prediction mode 1 for providing probability values ​​for neural network based intra-prediction mode K, and a parameterizable version of the neural network based intra-prediction mode 1 for providing probability values ​​for neural network based intra-prediction mode K. B-1 84. A neural network for providing a probability value associated with KB-1 8. The parameters 111 for parameterizing the neural network 84 and the neural networks 800 to 80 KB-1are input or applied by the updater 110 to the parameter inputs of each of these neural networks. The device 108 has access to a reservoir or a number of image test blocks 114 together with corresponding adjacent sample sets 116. These pairs of blocks 114 and their associated adjacent sample sets 116 are used sequentially by the device 108. In particular, the current image test block 114 is applied to a parameterizable neural network 109, the neural network 80 providing a prediction signal 118 for each neural network-based intra-prediction mode of the set 72, each neural network 80 providing a probability value for each of these modes. For this purpose, these neural networks use the current parameters 111 and 113.

[0113] In the above description, rec is used to denote the image test block 114.

number

number

number

number

[0114] Thus, the updater 110 tries to update the parameters 111 and 113 so as to reduce the coding cost function, and these updated parameters 111 and 113 are then used by the parameterizable neural network 109 to process a number 112 of next image test blocks. As mentioned above with respect to section 1.5, there can be a mechanism to control that mainly those pairs of image test blocks 114 and their associated adjacent sample sets 116 are applied to the recursive update process in which intra prediction, in the rate-distortion sense, is performed, preferably without subdivision of the block, thereby avoiding that the parameters 111 and 113 are over-optimized based on image test blocks whose sub-block wise coding is more cost-effective anyway.

[0115] So far, the above examples mainly concern the case where the encoder and decoder have a set of neural network-based intra-prediction modes among the supported intra-prediction modes 66. According to the examples described with respect to Fig. 9a and Fig. 9b, this does not necessarily have to be the case. Fig. 9a is intended to outline the operation modes of the encoder and decoder according to the examples whose description is provided in a manner that focuses on the differences with the description presented above with respect to Fig. 7a. The plurality of 66 supported intra-prediction modes may or may not include neural network-based intra-prediction modes, and may or may not include non-neural network-based intra-prediction modes. Thus, the modules 170 of Fig. 9a, respectively configured by the encoder and decoder to provide each of the supported modes 66, may be neural network-based or heuristically motivated, and may or may not include a prediction signal based on the corresponding prediction signal. As already indicated above, such intra-prediction modes may be neural network-based or heuristically motivated, and may or may not calculate a prediction signal based on a DC intra-prediction mode or an angular intra-prediction mode or other. These modules 170 may therefore be represented as prediction signal computers. However, the example encoder and decoder of Fig. 9a include a neural network 84. The neural network 84 calculates probability values ​​for supported intra-prediction modes 66 based on a neighboring sample set 86, and can then turn the multiple intra-prediction modes 66 into an ordered list 94. An index 70 in the data stream 12 of block 18 points to this ordered list 94. The neural network 84 thus helps to reduce the side information rate spent on signaling the intra-prediction modes.

[0116] FIG. 9b shows an alternative to FIG. 9a in that instead of ordering, entropy decoding / encoding 100 of index 70 is used to control the probability or its simple statistics, i.e., to control the entropy probability distribution of the entropy decoding / encoding in the encoder / decoder according to the probability values ​​determined for the neural network 84 for each mode of the plurality 66.

[0117] FIG. 10 shows an apparatus for designing or parameterizing a neural network 84. It is thus an apparatus 108 for designing a neural network for assisting in selecting among a set 66 of intra-prediction modes. Here, for each mode of the set 66, a corresponding neural network block together forms the neural network 84, and the parameterizable neural network 109 of the apparatus 108 is merely parameterizable with respect to these blocks. For each mode, there is also a predicted signal computer 170, which does not have to be parameterizable according to FIG. 10. Thus, the apparatus 108 of FIG. 10 calculates a cost estimate for each mode based on a predicted signal 118 calculated by the corresponding predicted signal computer 170 and, if necessary, on a corresponding probability value determined by the corresponding neural network block for this mode. Based on the resulting cost estimate 124, a minimum cost selector 126 selects the mode of the minimum cost estimate, and an updater 110 updates the parameters 111 of the neural network 84.

[0118] With respect to the description of Figures 7a to 7d and Figures 9a and 9b, the following should be noted. A common feature of the examples of Figures 9a and 9b, which are also used by some of the examples of Figures 7a to 7d, was the fact that the neural network values ​​probability values ​​to improve or reduce the overhead associated with side information 70 to inform the decoder of the mode determined at the encoder side in the optimization process 90. However, as shown above with respect to the examples of Figures 7a to 7d, it should be clear that the examples of Figures 9a and 9b can be modified to the extent that no side information 70 is spent in the data stream 12 with respect to mode selection. Rather, the probability values ​​output by the neural network 84 for each mode can be used to necessarily synchronize the mode selection between the encoder and the decoder. In that case, there would be no optimization decision 90 at the encoder side with respect to mode selection. Rather, the mode to be used among the sets 66 would be determined in the same way at the encoder side and the decoder side. Similar statements hold with respect to the corresponding examples of Figures 7a to 7d when modified to not use any side information 70 in the data stream 12. However, going back to the example of Figures 9a and 9b, it is interesting that the selection process 68 at the decoder side depends on the probability values ​​output by the neural network, in that the ordering or probability distribution estimation dependence on the probability values ​​changes the interpretation of the side information as far as the encoder is concerned; the dependence on the probability values ​​not only affects the encoding of the side information 70 into the data stream 12, for example using a respective variable length coding of the indexes into an ordered list, or using entropy coding / decoding with a probability distribution estimation depending on the probability values ​​of the neural network, but also the optimization step 90: here the code rate for transmitting the side information 70 can be taken into account and therefore affects the decision 90. Example in Figure 11-1 Fig. 11-1 shows a possible implementation of the encoder 14-1, i.e. the encoder is configured to use transform coding to code the prediction residual, but this is mostly an example and the present application is not limited to such a prediction residual coding. According to Fig. 11-1, the encoder 14-1 comprises a spatial domain subtractor 22 configured to subtract a corresponding prediction signal 24-1 from an inbound signal, i.e. the image 10, or from a current block 18 on a block basis, to obtain a spatial domain prediction residual signal 26, which is subsequently coded into the data stream 12 by a prediction residual encoder 28. The prediction residual encoder 28 comprises a lossy coding stage 28a and a lossless coding stage 28b. The lossy coding stage 28a comprises a quantizer 30 that receives the prediction residual signal 26 and quantizes samples of the prediction residual signal 26. The present example uses transform coding of the prediction residual signal 26, and therefore the lossy coding stage 28a includes a transform stage 32 connected between the subtractor 22 and the quantizer 30 so as to transform such a spectrally decomposed prediction residual 27 by a quantization of the quantizer 30 performed on the transformed coefficients representing the residual signal 26. The transform can be a DCT, a DST, an FFT, a Hadamard transform, etc. The transformed and transform domain quantized prediction residual signal 34 then undergoes lossless coding by a lossless coding stage 28b, which is an entropy coder that entropy codes the quantized prediction residual signal 34 into the data stream 12.

[0119] The encoder 14-1 further comprises a transform domain prediction residual signal reconstruction stage 36-1 connected to the transform domain output of the quantizer 30 so as to reconstruct the prediction residual signal from the transformed and quantized prediction residual signal 34 (in the transform domain) in a way that is also usable by the decoder, i.e. taking into account the coding loss of the quantizer 30. For this purpose, the prediction residual reconstruction stage 36-1 comprises an inverse quantizer 38-1 that performs the inverse of the quantization of the quantizer 30 to obtain an inverse quantized version 39-1 of the prediction residual signal 34, followed by an inverse transformer 40-1 that performs an inverse transformation with respect to the transformation performed by the transformer 32, such as the inverse of a spectral decomposition, such as the inverse of any of the specific transformation examples mentioned above. Downstream of the inverse transformer 40-1 it has a spatial domain output 60 that can contain a template that serves to obtain the prediction signal 24-1. In particular, the predictor 44-1 may provide a transform domain output 45-1 which, when inverse transformed in an inverse transformer 51-1, provides the predicted signal 24-1 in the spatial domain (the predicted signal 24-1 is subtracted from the inbound signal 10 to obtain the prediction residual 26 in the time domain). In interframe mode, an in-loop filter 46-1 may also filter the fully reconstructed image 60 and, after filtering, form a reference image 47-1 for the predictor 44-1 with respect to the inter-prediction block (hence, in these cases, adder 57-1 inputs from elements 44-1 and 36-1 are required, but there is no need for an inverse transformer 51-1 to provide the predicted signal 24-1 to the subtractor 22, as indicated by dashed line 53-1).

[0120] However, unlike the encoder 14 of Fig. 2, the encoder 14-1 (in the prediction residual reconstruction stage 36-1) comprises a transform domain adder 42-1 arranged between the inverse quantizer 38-1 and the inverse transformer 40-1. The transform domain adder 42-1 provides the inverse transformer 40-1 with a sum 43-1 (in the transform domain) of the inverse quantized version 39-1 of the prediction residual signal 34 (provided by the inverse quantizer 38-1) using a transform domain prediction signal 45-1 as provided by a transform predictor 44-1. The predictor 44-1 may take the output from the inverse transformer 40-1 as a feedback input.

[0121] Thus, the spatial domain prediction signal 24-1 is obtained from the transform domain prediction signal 45-1. The transform domain predictor 44-1, which may also operate on a neural network according to the above example, is input by a spatial domain signal but outputs a transform domain signal. Example in Figure 11-2 FIG. 11-2 shows a possible implementation of a decoder 54-2, i.e. one that matches the implementation of the encoder 14-1. Many elements of the encoder 54-2 are the same as those occurring in the corresponding encoder of FIG. 11-1, so the same references with the addition of "-2" are used in FIG. 11-2 to indicate these elements. In particular, the adder 42-2, the optional in-loop filter 46-2, and the predictor 44-2 are connected to the prediction loop in the same way as in the encoder of FIG. 11-1. The reconstructed, i.e. inversely quantized and retransformed prediction residual signal 24-2 (e.g., 60) is derived by a sequence of an entropy decoder 56 that reverses the entropy coding of the entropy encoder 28b, followed by a residual signal reconstruction stage 36-2 that is composed of an inverse quantizer 38-2 and an inverse transformer 40-2 as in the case of the encoding side. The output of the decoder is a reconstruction of the image 10. Several post filters 46-2 can be placed at the output of the decoder in order to subject the reconstruction of the image 10 to some post filtering to improve the image quality. Similarly, the explanation given above with respect to Fig. 11-1 is also valid for Fig. 11-2, except that the encoder only performs optimization tasks and related decisions regarding coding options. However, all the explanations regarding block subdivision, prediction, inverse quantization and retransformation are also valid for the decoder 54 of Fig. 11-2. The reconstructed signal 24-2 is provided to a predictor 44-2, which may operate with a neural network according to the examples of Figs. 5 to 10. The predictor 44-2 may provide a transform domain prediction 45-2.

[0122] Contrary to the example of Fig. 4, but similar to the example of Fig. 11-1, the inverse quantizer 38-2 provides an inverse quantized version 39-2 of the prediction residual signal 34 (in the transform domain) that is not provided directly to the inverse transformer 40-2. Instead, the inverse quantized version 39-2 of the prediction residual signal 34 is input to the adder 42-2 and constituted by the transform domain prediction value 45-2. Thus, a transform domain reconstructed signal 43-2 is obtained, which, when subsequently inverse transformed by the inverse transformer 40-2, becomes the reconstructed signal 24-2 in the spatial domain used to display the image 10. Example in Figure 12 Reference is now made to Fig. 12, both the decoder and the encoder simultaneously, i.e. in terms of their functioning with respect to the intra prediction block 18. The difference between the encoder and the decoder operation modes with respect to the intra coding block 18 is the fact that, on the one hand, the encoder runs all or at least some of the available intra prediction modes 66 and determines at 90 the best one, for example in terms of a cost function that minimizes the significance, and the encoder forms the data stream 12, i.e. the code dates thereon, and the decoder derives the data therefrom by decoding and reading, respectively. Fig. 12 shows the operation mode of the alternative outlined above, in which a flag 70a in the side information 70 of the block 18 indicates whether the intra prediction mode determined by the encoder in step 90 to be the best mode for the block 18 is one of the sets 72, i.e. the neural network based intra prediction modes, or one of the sets 74, i.e. the non-neural network based intra prediction modes. The encoder inserts the flag 70a accordingly into the data stream 12, while the decoder retrieves the flag 70a therefrom. FIG. 12 assumes that the determined intra-prediction mode 92 is in the set 72. Then, a separate neural network 84 determines a probability value for each neural network-based intra-prediction mode of the set 72, and using these probability value sets 72, or more precisely, the neural network-based intra-prediction modes therein are ordered according to their probability values, such as in descending order of probability values, thereby resulting in an ordered list 94 of intra-prediction modes. Then, the index 70b, which is part of the side information 70, is encoded by the encoder into the data stream 12 and decoded therefrom by the decoder. Thus, the decoder can determine which set of the sets 72 and 74. The intra-prediction mode used for the block 18 is located to perform an ordering 96 of the set 72 if the intra-prediction mode used is located in the set 72. If the determined intra-prediction mode is located in the set 74, the index can also be transmitted in the data stream 12.The decoder may therefore generate a prediction signal for block 18 using the determined intra-prediction mode by controlling selection 68 accordingly.

[0123] As can be seen from Fig. 12, a prediction residual signal 34 (in the transform domain) is coded into the data stream 12. Inverse quantizers 38-1, 38-2 derive inverse quantized prediction residual signals 39-1, 39-2 in the transform domain. From predictors 44-1, 44-2, transform domain prediction signals 45-1, 45-2 are obtained. Then, adder 42-1 sums the values ​​39-1 and 45-1 together (or adder 42-2 sums the values ​​39-2 and 45-2) to obtain a transform domain reconstruction signal 43-1 (or 43-2). Downstream of the inverse transformers 40-1, 40-2, spatial domain prediction signals 24-1, 24-2 (e.g. templates 60) are obtained and can be used to reconstruct a block 18 (which can be displayed, for example).

[0124] All of the variations in FIGS. 7b-7d can be used to implement the examples in FIGS. 11-1, 11-2, and 12. Discussion A method for generating an intra prediction signal via a neural network is defined and how this method can be included in a video or still image codec is explained. In these examples, instead of predicting into the spatial domain, the predictors 44-1, 44-2 can predict into the transform domain of a predefined image transformation that may already be available in the underlying codec, such as the discrete cosine transform. Secondly, each intra prediction mode defined for an image on a block of a particular shape induces an intra prediction mode for an image on a larger block.

[0125] Let B be a block of pixels with M rows and N columns in which image im resides. Let B be the neighbors of B (block 18) for which an already reconstructed image rec is available. recAssume that a template (template 60 or 86) exists. Next, in the example of Fig. 5 to Fig. 10, new intra prediction modes defined by a neural network are introduced. Each of these intra prediction modes is similarly calculated using the reconstructed samples rec(24-1, 24-2) as rec The signal generating unit 45 generates a predicted signal pred(45-1, 45-2) which is an image of the signal 45-1.

[0126] T, B rec Let the image transform defined by the image above (for example the prediction residual signal 34 output by element 30) and let S be the inverse transform of T (for example 43-1 or 43-2). Then the prediction signal pred(45-1, 45-2) is considered as the prediction of T(im). This means that at the reconstruction stage, after the calculation of pred(45-1, 45-2), it is necessary to calculate the image S(pred)(24-1, 24-2) to obtain the actual prediction of the image im(10).

[0127] It should be noted that the transform T we work with has some energy compaction properties for natural images, which are exploited as follows: for each of the intra-modes defined by the neural network, by a predefined rule, the value of pred(45-1, 45-2) at a certain position in the transform domain is set to zero, independent of the input rec(24-1, 24-2). This reduces the computational complexity of obtaining the predicted signal pred(45-1, 45-2) in the transform domain.

[0128] (With reference to FIGS. 5-10, assume that a transform T (32) and an inverse transform S (40) are used in the transform residual coding of the underlying codec. In the reconstructed signal (24, 24') of B, the prediction residual res (34) is inverse transformed by the inverse transform S (40) to obtain S(res), which is added to the underlying prediction signal (24) to obtain the final reconstructed signal (24).) In contrast, Figures 11 and 12 refer to the following procedure: if the predicted signal pred (45-1, 45-2) is generated by the neural network intra-prediction method as described above, the final reconstructed signal (24-1, 24-2) is obtained by the inverse transform (40-1, 40-2) of pred+res (pred is 45-1 or 45-2, res is 39-1 or 39-2), the sum of which is 43-1 or 43-2, which is the transform domain version of the final reconstructed signal 24-1, 24-2.

[0129] Finally, it should be noted that the above modifications of the intra prediction performed by the neural network as described above are arbitrary and unnecessarily correlated with each other. This means that for a given transform T (32) with an inverse transform S (40-1, 40-2) and one of the intra prediction modes defined by the neural network described above, it may be extracted from the bitstream or from a predefined setting whether the mode is considered as predictive to the transform domain corresponding to T. Figures 13a and 13b Referring to Figures 13a and 13b, a strategy is shown that may be applied, for example, to spatial domain-based methods (e.g., Figures 11a and 11b) and / or transform domain-based methods (e.g., Figures 1-4).

[0130] In some cases, a neural network adapted to blocks of a particular size can be used at will (e.g., M×N, where M is the number of rows and N is the number of columns), but the actual blocks 18 of the image to be reconstructed have a different size (e.g., M1×N1). It should be noted that it is possible to perform operations that allow for the use of neural networks adapted to a particular size (e.g., M×N) without the need to use ad-hoc trained neural networks.

[0131] In particular, the device 14 or 54 may enable block-wise decoding of an image (e.g., 10) from a data stream (e.g., 12). The device 14, 54 natively supports at least one intra prediction mode, according to which an intra prediction signal for a block (e.g., 136, 172) of a predetermined size (e.g., M×N) of the image is determined by applying a first template (e.g., 130, 170) of samples neighboring the current block (e.g., 136, 176) on a neural network (e.g., 80). The device may be configured as follows for a current block (e.g., 18) of a different size (e.g., M1×N1): - resampling (e.g. D, 134, 166) a second template (e.g. 60) of samples adjacent to the current block (e.g. 18) to obtain a resampled template (e.g. 130, 170) in accordance with the first template (e.g. 130, 170); applying a resampled template (e.g., 130, 170) of samples on a neural network (e.g., 80) to obtain a preliminary intra-prediction signal (e.g., 138); -Resample (eg U, V, 182) the preliminary intra-prediction signal (138) to match the current block (18, B1) to obtain an intra-prediction signal for the current block.

[0132] Figure 13a shows an example in the spatial domain. The spatial domain block 18 (also denoted as B1) can be an M1xN1 block from which the image im1 is reconstructed (even though the image im1 is not yet available at the moment). 1,rec (e.g., set 60) already has a reconstructed image rec1, where rec1 is adjacent to im1 (and B 1,rec Note that B1 is adjacent to B2. Block 18 and template 60 (the “second template”) can form element 132.

[0133] The dimensionality of B1 raises the possibility that we do not have a neural network at our disposal to reconstruct B1. However, if we have a neural network at our disposal with a block of different dimensions (such as the "first template"), we can take the following steps:

[0134] A transformation operation (here shown as D or 134) can be applied, for example, to element 130. However, since B1 is still unknown, we cannot apply the transformation D (130) to B 1,rec Note that it is easily possible to apply the transform 130 only to the template 130. The transform 130 can provide an element 136 formed from the transformed (resampled) template 130 and a block 138.

[0135] For example, an M1 x N1 block B1 (18) (with unknown coefficients) could theoretically be transformed into an M x N block B (138) (also with unknown coefficients), but because the coefficients of block B (138) are unknown, there is no need to actually perform the transformation.

[0136] Similarly, transformation D(134) is 1,rec (60) is divided into different templates B rec (130). The template 130 has a vertical thickness L (i.e., L columns of vertical portions) and a horizontal thickness K (i.e., K rows of horizontal portions), and B rec =D(B 1,rec It can be appreciated that the template 130 can include the following: -B rec K × N blocks on (130), -B rec The M×L block to the left of (130), and -B rec (130) above, and B rec The K×L block to the left of the K×N block on the M×L block to the left of (130).

[0137] In some cases, the transformation operation D (134) can be a downsampling operation, where M>M and N>N (particularly when M is a multiple of M and N is a multiple of N). For example, when M=2M and N=2N, the transformation operation D can be based on hiding some bins in a chess-like manner (e.g., B 1,rec Delete the diagonal from 60 and add B rec (Get value of 130).

[0138] At this point, B rec (B rec =D(rec1)) is the MxN reconstructed image. In path 138a, the device 14, 54 can use the necessary neural network (e.g., in the predictor 44, 44') natively trained for the MxN block (e.g., by operating as in Figs. 5-10). By applying the above path (138a), the image im1 of block B is obtained. (In some examples, path 138a does not use a neural network, but uses other techniques known in the art.)

[0139] At this point, image im1 in block B (138) has size M×N, but the size of the image to be displayed needs to be M1×N1. Note, however, that it is simply possible to perform a transformation (e.g., U) 140 that transforms image im1 in block B (138) to M1×N1.

[0140] Note that if D performed in 134 is a downsampling operation, U in 140 may be an upsampling operation. Thus, U(140) can be obtained by introducing coefficients in an M1xN1 block in addition to the coefficients in the MxN block 138 obtained in operation 138a using a neural network.

[0141] For example, if M1=2M and N1=2N, it is easily possible to perform an interpolation (e.g., bilinear interpolation) to approximate ("guess") the coefficients of im1 that were discarded by transform D. Thus, the M1×N1 image im1 can be obtained as element 142 and used to display the block image as part of image 10.

[0142] In particular, it is theoretically possible to obtain block 144, which is nevertheless identical to template 60 (except for the errors due to transformations D and U). Therefore, advantageously, we can use block 144, which is already at our disposal as template 60. 1,rec To get a new version of B rec There is no need to convert.

[0143] The operations shown in Fig. 13a can be performed, for example, in predictor 44 or 44'. Thus, the M1 x N1 image im1 (142) can be understood as the predicted signal 24 (Fig. 2) or 24' (Fig. 4) that is summed with the predicted residual signal output by inverse transformer 40 or 40' to obtain a reconstructed signal.

[0144] Figure 13b shows an example in the transform domain (e.g., in the examples of Figures 11-1, 11-2). An element 162 is represented as being formed by a spatial domain template 60 (already decoded) and a spatial domain block 18 (with unknown coefficients). The block 18 may have a size M1xN1 and may have unknown coefficients, which are to be determined, for example, in a predictor 44-1 or 44-2.

[0145] While we are free to use any neural network of determined size M×N, we may not have a neural network that directly operates on an M1×N1 block in the transform domain.

[0146] However, it should be noted that in predictors 44-1, 44-2, it is possible to use a transform D (166) applied to template 60 ("second template") to obtain a spatial domain template 170 having a different dimension (e.g., a reduced dimension). Template 170 ("first template") may have, for example, an L-shaped shape, such as the shape of template 130 (see above).

[0147] At this point, in path 170a, the neural network (e.g., 800-80 N ) can be applied according to any of the examples above (see FIGS. 5-10). Thus, at the end of passage 170a, known coefficients of version 172 of block 18 can be obtained.

[0148] However, it should be noted that the dimensions MxN of 172 do not fit the dimensions M1xN1 of the block 18 that must be visualized. Therefore, a transformation to the transform domain (e.g., in 180) can be manipulated. For example, an MxN transform domain block T (176) can be obtained. To increase the number of rows and columns to M1 and N1, respectively, a technique called zero padding can be used, for example, by introducing values ​​"0" corresponding to frequency values ​​associated with frequencies that are not present in the MxN transform T (176). Therefore, a zero padding region 178 can be used (e.g., can have an L-shape). In particular, the zero padding region 178 includes a number of bins (all zeros) that are inserted in the block 176 to obtain a block 182. This can be obtained by a transformation V from T (transformed from 172) to T1 (182). The dimensions of T (176) do not match those of block 18, but the dimensions of T1 (182) do in fact match those of block 18 due to the insertion of zero-padding regions 178. Furthermore, the zero-padding is obtained by inserting higher frequency bins (with zero values), which results in a result similar to interpolation.

[0149] Therefore, in adders 42-1 and 42-2, a transform T1 (182) can be added which is a version of 45-1 and 45-2. -1 to obtain reconstructed values ​​60 in the spatial domain that are used to visualize the image 10.

[0150] The encoder may encode information about resampling (and the use of neural networks for blocks of a size different from that of block 18) into the data stream 12 so that the decoder has that knowledge. Discussion Let B1 (e.g., 18) be a block with M1 rows and N1 columns, and assume that M1 ≥ M and N1 ≥ N. Let B1,rec be a neighbor of B1 (e.g., the neighboring template 60), and let B 1,rec Region B considered as a subset of rec Let im1 (e.g., 138) be the image of B1, and rec1 (e.g., 1,rec (coefficient of B 1,rec The above solution is based on a predefined downsampling operation D (e.g., 134, 166) that maps the image of B1,rec to the image of B1. For example, if M1=2M and N1=2N, then B rec If B1,rec consists of K rows on B, L columns on the left side of B, and a top-left corner of B with size K×L, and B1,rec consists of 2K rows on B1, 2L columns on the left side of B, and a top-left corner of B with size 2K×2L, then D can be a smoothing filter followed by a 2x downsampling operation in each direction. Thus, D(rec1) is the sum of B rec Using the neural network-based intra prediction mode described above, we can form a prediction signal pred(45-1), which is the image on B, from D(rec1).

[0151] Here, we distinguish two cases: First, assume that in B, neural network-based intra prediction predicts to the sample (spatial) domain, as in Fig. 2, Fig. 4, and Fig. 13a. Let U(140) be a fixed upsampling filter that maps the image of B (e.g., 138) to the image of B1 (e.g., 142). For example, when M1=2M and N1=2N, U can be a bilinear interpolation operation. Then, U(pred) can be formed to obtain the image on B1 (e.g., 45-1) that is considered as the predicted signal of im1 (e.g., 10).

[0152] Secondly, as in Fig. 11-1, Fig. 11-2 and Fig. 13b, assume that in B, the prediction signal pred (e.g. 45-2) should be considered as a prediction signal in the transform domain for an image transform T on B using an inverse transform S. Let T1 be an image transform on B1 using an inverse transform S1. Assume that a predefined mapping V is given that maps an image from the transform domain of T to the transform domain of T1. For example, if T is a discrete cosine transform of an M×N block using an inverse transform S and T1 is a discrete cosine transform of an M1×N1 block using an inverse transform S1, then a block of transform coefficients of B can be mapped to a block of transform coefficients of B1 by zero padding and scaling (see, e.g. 178). This means that if the position in frequency space is greater than M or N in the horizontal response vertical direction, all transform coefficients of B1 are set to zero and the appropriately scaled transform coefficients of B are copied to the remaining M*N transform coefficients of B1. Then, V(pred) can be formed to obtain an element in the transform domain of T1, which can be considered as the predicted signal of T1(im1). The signal V(pred) can be further processed as described above.

[0153] As described above with respect to Figures 1-10, we have also described how a neural network-based operation is used to rank several intra-prediction modes at a particular block B by generating a conditional probability distribution between these modes, and how this ranking can be used to inform which intra-prediction mode to apply at the current block. Using a downsampling operation (e.g., 166) at the input of the neural network that generates the latter ranking in the same way as the actual prediction modes produces a ranking for extending prediction modes to a larger block B1 than just described, and thus is used to inform which extension mode to use at block B1. Whether to generate a prediction signal using a neural network-based intra-prediction mode from a smaller block B on a given block B1 can be predefined or signaled as side information of the underlying video codec. More Examples Generally speaking, such a decoder may comprise such an encoder, and / or vice versa. For example, the encoder 14 may be or include the decoder 54 (or vice versa). The encoder 14-1 may be the decoder 54-2 (or vice versa), etc. Furthermore, the encoder 14 or 14-1 may also be understood to include a decoder itself, since the quantized prediction residual signal 34 forms a stream that is decoded to obtain the prediction signal 24 or 24-1.

[0154] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, with a block or apparatus corresponding to a method step or feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or function of a corresponding apparatus. Some or all of the method steps can be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some examples, one or more of the most important method steps can be performed by such an apparatus.

[0155] The encoded data stream of the present invention can be stored on a digital storage medium or can be transmitted over a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

[0156] Depending on specific implementation requirements, examples of the present invention can be implemented in hardware or software. Implementation can be done using digital storage media such as floppy disks, DVDs, Blu-rays, CDs, ROMs, PROMs, EPROMs, EEPROMs, flash memories, etc., on which electronically readable control signals are stored and which cooperate (or can cooperate) with a programmable computer system to execute the respective methods. Thus, the digital storage media can be computer readable.

[0157] Some examples according to the present invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to cause one of the methods described herein to be performed.

[0158] In general, examples of the invention can be implemented as a computer program product comprising program code which operates to perform one of the methods when the computer program product is run on a computer. The program code may for example be stored on a machine readable carrier.

[0159] Other examples comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0160] Thus, an example of the inventive methods is a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0161] A further example of the inventive methods is therefore a data carrier (or digital storage medium, or computer readable medium) having recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recorded medium is typically tangible and / or non-transitory.

[0162] A further example of the inventive methods is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, the data stream or the sequence of signals may be adapted to be transferred via a data communication connection, such as for example the Internet.

[0163] Further examples include a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0164] A further example comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0165] Further examples according to the invention include a device or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The device or system may include, for example, a file server for transferring the computer program to the receiver.

[0166] In some examples, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some examples, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0167] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0168] The apparatus described herein, or any components of the apparatus described herein, may be implemented at least in part in hardware and / or software.

[0169] The methods described herein can be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0170] Any of the methods described herein, or any of the components of the apparatus described herein, may be at least partially implemented by hardware and / or software.

[0171] The above examples merely illustrate the principles of the present invention. It is understood that modifications and variations of the configurations and details described herein will be apparent to those skilled in the art. It is therefore intended to be limited only by the scope of the appended claims and not by the specific details presented as examples and illustrations herein.

Claims

1. 1. A method for decoding an image from a data stream, comprising the steps of: The method comprises: determining a first intra prediction signal for a block of a first size using the prediction mode and a first set of samples neighboring the block; downsampling a second set of samples adjacent to the current block of a second size to obtain a downsampled set of samples having dimensions of the first set of samples; generating a preliminary intra prediction signal using the prediction mode and the set of downsampled samples; upsampling the preliminary intra prediction signal to obtain a second intra prediction signal of the second size; predicting the current block using the second intra prediction signal; and A method comprising:

2. the first size is different from the second size; the first set and the second set of samples include previously reconstructed samples; the image includes the block and the current block; The method of claim 1.

3. the first size corresponds to a first dimension and a second dimension; the second size corresponds to a third dimension and a fourth dimension; the first dimension is different from the third dimension; The method of claim 1.

4. determining, for the current block, the prediction mode to be used for determining the first intra prediction signal for generating the preliminary intra prediction signal; The method of claim 1 further comprising:

5. selecting a first set of prediction modes or a second set of prediction modes for the block, the first set of prediction modes being adapted for different size blocks and the second set of prediction modes including a DC mode and multiple angle modes; selecting the prediction mode from the first set of prediction modes based on blocks of the first size; applying the selected prediction mode to the first set of samples to determine the first intra prediction signal; The method of claim 1 further comprising:

6. obtaining a prediction mode corresponding to a prediction mode selected from a set of prediction modes based in part on the first size; applying the prediction mode to the first set of samples to determine the first intra prediction signal; The method of claim 1 further comprising:

7. transforming the preliminary intra prediction signal from a spatial domain to a transform domain; upsampling the preliminary intra prediction signal in the transform domain; The method of claim 1 further comprising:

8. 1. An electronic device for decoding an image from a data stream, the electronic device comprising: a processor; The processor, determining a first intra prediction signal for a block of a first size using the prediction mode and a first set of samples neighboring the block; downsampling a second set of samples adjacent to the current block of a second size to obtain a downsampled set of samples having dimensions of the first set of samples; generating a preliminary intra prediction signal using the prediction mode and the set of downsampled samples; upsampling the preliminary intra prediction signal to obtain a second intra prediction signal of the second size; predicting the current block using the second intra prediction signal; The electronic device is configured to:

9. the first size is different from the second size; the first set and the second set of samples include previously reconstructed samples; the image includes the block and the current block; 9. The electronic device of claim 8.

10. the first size corresponds to a first dimension and a second dimension; the second size corresponds to a third dimension and a fourth dimension; the first dimension is different from the third dimension; 9. The electronic device of claim 8.

11. The processor, determining, for the current block, the prediction mode to be used for determining the first intra prediction signal for generating the preliminary intra prediction signal; The electronic device of claim 8 further configured to:

12. The processor, selecting a first set of prediction modes or a second set of prediction modes for the block, the first set of prediction modes being adapted for different size blocks and the second set of prediction modes including a DC mode and a plurality of angle modes; selecting the prediction mode from the first set of prediction modes based on a block of the first size; applying the selected prediction mode to the first set of samples to determine the first intra prediction signal. The electronic device of claim 8 further configured to:

13. The processor, obtaining a prediction mode corresponding to a prediction mode selected from a set of prediction modes based in part on the first size; applying the prediction mode to the first set of samples to determine the first intra prediction signal. The electronic device of claim 8 further configured to:

14. The processor, Transforming the preliminary intra prediction signal from a spatial domain to a transform domain; upsampling the preliminary intra prediction signal in the transform domain; The electronic device of claim 8 further configured to:

15. A non-transitory computer-readable medium, comprising: When executed, the method causes at least one processor to determining a first intra prediction signal for a block of a first size using the prediction mode and a first set of samples neighboring the block; downsampling a second set of samples adjacent to the current block of a second size to obtain a downsampled set of samples having dimensions of the first set of samples; generating a preliminary intra prediction signal using the prediction mode and the set of downsampled samples; upsampling the preliminary intra prediction signal to obtain a second intra prediction signal of the second size; predicting the current block using the second intra prediction signal; A non-transitory computer readable medium comprising instructions for causing

16. the first size is different from the second size; the first set and the second set of samples include previously reconstructed samples; the block and the current block are contained within an image; 16. The non-transitory computer-readable medium of claim 15.

17. the first size corresponds to a first dimension and a second dimension; the second size corresponds to a third dimension and a fourth dimension; the first dimension is different from the third dimension; 16. The non-transitory computer-readable medium of claim 15.

18. When executed, the method causes at least one processor to determining, for the current block, the prediction mode to be used for determining the first intra prediction signal for generating the preliminary intra prediction signal; 20. The non-transitory computer-readable medium of claim 15, further comprising instructions to cause

19. When executed, the method causes at least one processor to selecting a first set of prediction modes or a second set of prediction modes for the block, the first set of prediction modes being adapted for different size blocks and the second set of prediction modes including a DC mode and a plurality of angle modes; selecting the prediction mode from the first set of prediction modes based on a block of the first size; applying the selected prediction mode to the first set of samples to determine the first intra prediction signal.

20. The non-transitory computer-readable medium of claim 15, further comprising instructions to cause

20. When executed, the method causes at least one processor to obtaining a prediction mode corresponding to a prediction mode selected from a set of prediction modes based in part on the first size; applying the prediction mode to the first set of samples to determine the first intra prediction signal.

20. The non-transitory computer-readable medium of claim 15, further comprising instructions to cause

Images