Concept of intra-predictive mode for block-based image coding

Abstract

The present invention provides an image decoding method and apparatus that includes an intra-predictive mode enabling more efficient compression of block-based image codecs. [Solution] The image decoding method resamples a second template of a sample adjacent to the current block to match a first template in order to obtain a resampled template, applies the resampled template of the sample to a neural network to obtain a preliminary intra-prediction signal, and resamples the preliminary intra-prediction signal to match the current block in order to obtain an intra-prediction signal for the current block.

Description

[Technical Field]

[0001] This application relates to the concept of an improved intra-predictive mode for block-based image coding, which can be used with video codecs such as HEVC or its successors. [Background technology]

[0002] Intra-predictive mode is widely used in image and video encoding. In video encoding, intra-predictive mode competes with other prediction modes such as mutual prediction modes, including motion-compensated prediction mode. In intra-predictive mode, the current block is predicted based on adjacent samples, i.e., samples that have already been encoded as far as the encoder side is concerned and already decoded as far as the decoder side is concerned. The adjacent sample values ​​are extrapolated to the current block to form the predicted signal for the current block, and the predicted residual is transmitted in the data stream of the current block. The better the predicted signal, the smaller the predicted residual, and therefore the fewer bits are needed to encode the predicted residual.

[0003] To be effective, several aspects must be considered in order to form an effective framework for intra-prediction in a block-based image coding environment. For example, the more intra-prediction modes supported by the codec, the greater the consumption of side information rate to notify the decoder of the selection. On the other hand, the set of supported intra-prediction modes must be able to provide good prediction signals, i.e., prediction signals with low prediction residuals. [Overview of the project] [Problems that the invention aims to solve]

[0004] This application seeks to provide an intra-predictive mode concept that enables more efficient compression of block-based image codecs when using an improved intra-predictive mode concept. [Means for solving the problem]

[0005] This objective is achieved by the subject matter of the independent claims of this application.

[0006] A device (e.g., a decoder) for decoding an image from a data stream in block units, which supports at least one intra-prediction mode in which an intra-prediction signal of a block of a predetermined size of the image is determined by applying a first template of samples adjacent to the current block to a neural network, wherein for the current block which is different from the predetermined size, To obtain a resampled template, resample the second template of the samples adjacent to the current block to match the first template. To obtain preliminary intra-predictions, a resampled template of the samples is applied to the neural network. An apparatus is disclosed which is configured to resample a preliminary intra-prediction signal to match the current block in order to obtain an intra-prediction signal for the current block.

[0007] A device (e.g., an encoder) for encoding an image into a data stream in blocks, which supports at least one intra-prediction mode in which an intra-prediction signal of a block of a predetermined size of the image is determined by applying a first template of samples adjacent to the current block to a neural network, wherein for the current block which is different from the predetermined size, To obtain a resampled template, resample the second template of the samples adjacent to the current block to match the first template. To obtain preliminary intra-predictions, a resampled template of the samples is applied to the neural network. An apparatus is also disclosed that is configured to resample a preliminary intra prediction signal to match a current block in order to obtain an intra prediction signal for the current block.

[0008] The apparatus can be configured to resample by downsampling a second template to obtain a first template.

[0009] The apparatus can be configured to resample the preliminary intra prediction signal by upsampling the preliminary intra prediction signal.

[0010] The apparatus can be configured to convert the preliminary intra prediction signal from a spatial domain to a transform domain and resample the preliminary intra prediction signal in the transform domain.

[0011] The apparatus can be configured to resample the transform domain preliminary intra prediction signal by scaling the coefficients of the preliminary intra prediction signal.

[0012] The apparatus increases the dimension of the intra prediction signal to match the dimension of the current block, and zero-pads the coefficients of the added coefficients of the preliminary intra prediction signal that are related to higher frequency bins to resample the transform domain preliminary intra prediction signal.

[0013] The apparatus can be configured to construct the transform domain preliminary intra prediction signal with an inverse quantization version of the prediction residual signal.

[0014] The apparatus can be configured to resample the preliminary intra prediction signal in the spatial domain.

[0015] The device can be configured to resample the preliminary intra-prediction signal by performing bilinear interpolation.

[0016] The device can be configured to encode information about resampling and / or the use of neural networks of different dimensions into a data field.

[0017] A device (e.g., a decoder) for decoding an image from a data stream in block units, Also disclosed is an apparatus that supports at least one intra-prediction mode in which an intra-prediction signal of the current block of an image is determined by applying a first set of adjacent samples of the current block to a neural network to obtain predictions of a set of transformation coefficients for the transformation of the current block.

[0018] A device (e.g., an encoder) for encoding an image into a data stream in block units, Also disclosed is an apparatus that supports at least one intra-prediction mode in which an intra-prediction signal of the current block of an image is determined by applying a first set of adjacent samples of the current block to a neural network to obtain predictions of a set of transformation coefficients for the transformation of the current block.

[0019] One of the devices can be configured to inversely transform the prediction to obtain the reconstructed signal.

[0020] One of the devices can be configured to decode an index from a data stream using a variable-length code and then use the index to perform a selection.

[0021] One of the devices can be configured to determine the ranking of a set of intra-prediction modes and then resample a second template.

[0022] This involves resampling the second template of a sample adjacent to the current block to obtain a resampled template that conforms to the first template, and Applying a resampled template of the sample to a neural network to obtain a preliminary intra-predicted signal, The preliminary intra-prediction signal is resampled to match the current block, and the intra-prediction signal for the current block is obtained. A method comprising the above is disclosed.

[0023] A method for decoding an image from a data stream in block units, A method is disclosed that comprises applying a first set of adjacent samples of the current block to a neural network to obtain predictions of a set of transformation coefficients for the transformation of the current block.

[0024] A method for encoding an image in blocks into a data stream, A method is disclosed that comprises applying a first set of adjacent samples of the current block to a neural network to obtain predictions of a set of transformation coefficients for the transformation of the current block.

[0025] The above and / or below methods may be used with equipment comprising at least one of the above and / or below devices.

[0026] Also disclosed is a computer-readable storage medium that, when executed by a computer, contains instructions causing the computer to perform the methods described above and / or below, and / or to implement the methods described above and / or below in at least one component of the device.

[0027] Data streams acquired by the above and / or the following methods and / or by the above and / or the following devices are also disclosed.

[0028] As far as the design of the neural network described above is concerned, this application provides many examples for appropriately determining its parameters.

[0029] A favorable implementation of this application is the subject of the dependent claims. A preferred example of this application is described below with respect to the figures. [Brief explanation of the drawing]

[0030] [Figure 1] As a typical example of how the examples in this application can be implemented, a schematic block diagram showing an encoder for encoding an image into a data stream is provided. [Figure 2] A block diagram of a more specific example of the encoder shown in Figure 1 is presented. [Figure 3] A schematic block diagram is shown illustrating a decoder that fits the encoder in Figure 1 and functions as an example of a decoder in which the example of this application can be implemented. [Figure 4] A block diagram of a more specific example of the decoder shown in Figure 3, which is compatible with the encoder in Figure 2, is shown. [Figure 5] Regarding the processing of blocks using intra-prediction, a schematic diagram illustrating the operating modes of an encoder and decoder according to an example of this application is provided. [Figure 6] A schematic block diagram is shown illustrating a decoder according to an example of this application, which includes several neural network-based intra-prediction modes. [Figure 7a]A schematic diagram shows the operating modes of an encoder and decoder for an example that supports neural network-based ordering of neural network-based intra-prediction modes by sending an index in a data stream to an ordered list of neural network-based intra-prediction modes, along with a flex indicating whether the intra-prediction mode used is a member of a set of neural network-based intra-prediction modes. It goes without saying that the index may be encoded using variable-length coding to take advantage of the different frequencies determined by decision 90. [Figure 7b] This schematic diagram differs from Figure 7a in that flex signaling is not used. [Figure 7c] This schematic diagram differs from Figure 7b in that mode ordering is not controlled using a neural network. [Figure 7d] This schematic diagram differs from Figure 7b in that mode ordering is not controlled using a neural network. [Figure 8] The example shows a setup for designing a set of neural network-based intra-predictive modes. [Figure 9a] A schematic diagram shows the operating modes of encoders and decoders in examples where neural networks are used to order supported intra-prediction modes, regardless of whether they are neural network-based or not. [Figure 9b] Figure 9a shows a different schematic diagram in that a neural network base is used to control the probability distribution estimation for entropy decoding / coding of indices to a set of supported intra-prediction modes. [Figure 10] The example shows a device for designing a neural network to assist and select from a set of intra-predictive modes for block-based image coding. [Figure 11-1] The encoder used in the example is shown. [Figure 11-2] The decoder used in the example is shown. [Figure 12] A schematic diagram illustrating the operating modes of the encoder and decoder in the example is shown. [Figure 13a] This shows a schematic diagram of the technology involved in the example. [Figure 13b] This shows a schematic diagram of the technology involved in the example. [Modes for carrying out the invention]

[0031] The following describes various examples that can help achieve more effective compression when using intra-prediction. Some examples improve compression efficiency by using a set of neural network-based intra-prediction modes. The latter may be added to, for example, other heuristically designed intra-prediction modes, or provided exclusively. Other examples use neural networks to perform selection from among multiple intra-prediction modes. And even these other examples utilize both of the specializations described here.

[0032] To facilitate understanding of the following examples of this application, the description begins with a presentation of possible encoders and decoders that can be adapted to construct the examples outlined later in this application. Figure 1 shows a device for encoding an image 10 into a data stream 12 in blocks. The device is indicated by reference numeral 14 and may be a still image encoder or a video encoder. In other words, the image 10 may be the current image from a video 16 if the encoder 14 is configured to encode a video 16 containing the image 10 into a data stream 12, or if the encoder 14 can exclusively encode the image 10 into the data stream 12.

[0033] As described above, the encoder 14 performs encoding in a block-by-block manner or on a block basis. For this purpose, the encoder 14 subdivides the image 10 into blocks, and the units of the encoder 14 encode the image 10 into a data stream 12. Examples of possible subdivisions of the image 10 into blocks 18 are shown in more detail below. In general, the subdivision may end up in blocks 18 of a fixed size, such as an array of blocks arranged in rows and columns, or in blocks 18 of different block sizes, such as by using hierarchical multitree subdivision, which starts from the entire image area of ​​the image 10 or from a pre-partition of the image 10 into an array of tree blocks, and these examples should not be treated as excluding other possible ways of subdividing the image 10 into blocks 18.

[0034] Furthermore, the encoder 14 is a predictive encoder configured to predictively encode the image 10 into the data stream 12. For a particular block 18, this means that the encoder 14 determines the predictive signal for block 18 and encodes the predictive residual, i.e., the prediction error by which the predictive signal deviates from the actual image content within block 18, into the data stream 12.

[0035] The encoder 14 can support different prediction modes to derive the prediction signal for a particular block 18. The prediction mode important in the following example is the intra-prediction mode, in which the interior of block 18 is spatially predicted from adjacent, already encoded samples of image 10. The encoding of image 10 into the data stream 12, and therefore the corresponding decoding procedure, can be based on a specific encoding order 20 defined among the blocks 18. For example, the encoding order 20 can traverse the blocks 18 in a raster scan order such as top to bottom row by row, traversing each row from left to right. In the case of hierarchical multi-tree based subdivision, the raster scan order can be applied within each hierarchical level, and a depth-first traversal order can be applied. That is, leaf notes in a block at a particular hierarchical level precede blocks at the same hierarchical level that have the same parent block according to the encoding order 20. Depending on the encoding order 20, adjacent, already encoded samples of block 18 can typically be placed on one or more sides of block 18. In the examples presented herein, for example, the already encoded samples adjacent to block 18 are located at the top and left of block 18.

[0036] The encoder 14 does not necessarily support only intra-predictive modes. For example, if encoder 14 is a video encoder, encoder 14 can also support an intra-predictive mode in which block 18 is temporarily predicted from images of previously encoded video 16. Such an intra-predictive mode may be a motion-compensated predictive mode in which a motion vector is signaled to such block 18, indicating the relative spatial offset of the portion from which the predicted signal of block 18 is derived as a copy. Additionally or alternatively, other non-intra-predictive modes may also be available, such as an inter-view predictive mode when encoder 14 is a multi-view encoder, or a non-predictive mode in which the inside of block 18 is encoded as is, i.e., without prediction.

[0037] Before beginning the description of this application with a focus on the intra-predictive mode, we will describe a more specific example of a possible block-based encoder, namely, a possible implementation of encoder 14 such that two corresponding examples of decoders, described with respect to Figure 2, are then presented, which are adapted to Figures 1 and 2, respectively.

[0038] Figure 2 shows a possible implementation of the encoder 14 of Figure 1, i.e., one in which the encoder is configured to use transform coding to encode the predictive residuals, but this is merely an example and the present application is not limited to this type of predictive residual coding. According to Figure 2, the encoder 14 includes a subtractor 22 configured to subtract the corresponding predictive signal 24 from an inbound signal, i.e., an image 10, or block-based from the current block 18, to obtain a predictive residual signal 26 which is later encoded into a data stream 12 by the predictive residual encoder 28. The predictive residual encoder 28 consists of an irreversible coding stage 28a and a reversible coding stage 28b. The irreversible stage 28a includes a quantizer 30 which receives the predictive residual signal 26 and quantizes samples of the predictive residual signal 26. As already mentioned above, this example uses transform coding of the predicted residual signal 26, and therefore the irreversible coding stage 28a includes a transform stage 32 connected between the subtractor 22 and the quantizer 30 to transform such predicted residuals 26 spectrally decomposed by the quantizer 30 performing quantization on the transformed coefficients that present the residual signal 26. The transform can be DCT, DST, FFT, Hadamard transform, etc. Next, the transformed and quantized predicted residual signal 34 undergoes reversible coding by a reversible coding stage 28b, which is an entropy coder that entropy codes the quantized predicted residual signal 34 into a data stream 12. The encoder 14 further includes a predicted residual signal reconstruction stage 36 connected to the output of the quantizer 30 to reconstruct the predicted residual signal from the transformed and quantized predicted residual signal 34 in a manner also available to the decoder. That is, it is the quantizer 30 that considers coding loss. For this purpose, the prediction residual reconstruction stage 36 comprises an inverse quantizer 38 that performs the inverse of the quantization of the quantizer 30, and a subsequent inverse converter 40 that performs the inverse transform of the transform performed by the converter 32, such as the inverse of spectral decomposition, such as the inverse of any of the specific transform examples described above. The encoder 14 comprises an adder 42 that adds the reconstructed prediction residual signal output by the inverse converter 40 and the prediction signal 24 to output a reconstructed signal, i.e., a reconstructed sample.This output is supplied to the predictor 44 of the encoder 14, which then determines the prediction signal 24 based on it. The predictor 44 supports all the prediction modes already described in relation to Figure 1. Figure 2 also shows that if the encoder 14 is a video encoder, the encoder 14 can also include an in-loop filter 46 that filters a fully reconstructed image that, after being filtered, forms a reference image of the predictor 44 with respect to the mutual prediction block.

[0039] As already mentioned above, the encoder 14 operates on a block basis. In the following description, the block base in question is an subdivided block of the image 10, for which an intra-prediction mode is selected from a set or multiple intra-prediction modes supported by the predictor 44 or encoder 14, respectively, and the selected intra-prediction modes are executed individually. However, other types of blocks may also exist in which the image 10 is subdivided. For example, the above determination of whether the image 10 is intercoded or intracoded can be made at a granular level or at the unit of blocks that deviate from block 18. For example, inter-mode / intra-mode determination can be performed at the level of coded blocks in which the image 10 is subdivided and each coded block is subdivided into prediction blocks. Each prediction block having a coded block in which it has been determined that intra-prediction will be used is subdivided into intra-prediction mode determinations. Then, for each of these prediction blocks, it is determined which supported intra-prediction mode to use for each prediction block. These prediction blocks form block 18 of interest here. Prediction blocks within a coded block related to mutual prediction will be handled differently by the predictor 44. They will be mutually predicted from the reference image by determining the motion vector and copying the prediction signal of this block from the position in the reference image indicated by the motion vector. Another block subdivision relates to the subdivision into transformed blocks in the unit where the transformation by the converter 32 and inverse converter 40 is performed. The transformed blocks may be, for example, the result of further subdivision of the coded block. Of course, the examples described here should not be treated as limiting, and other examples exist. For completeness only, it should be noted that subdivision into coded blocks can use, for example, multi-tree subdivision, and similarly, prediction blocks and / or transformed blocks can be obtained by further subdivision of the coded block using multi-tree subdivision.

[0040] Figure 3 shows a decoder or device for block-level decoding that is compatible with the encoder 14 in Figure 1. This decoder 54 does the opposite of the encoder 14; that is, it decodes the image 10 from the data stream 12 in blocks, and for this purpose supports multiple intra-prediction modes. The decoder 54 may include, for example, a residual provider 156. All the other possibilities described above with respect to Figure 1 are also valid for the decoder 54. For this reason, the decoder 54 can be a still image decoder or a video decoder, and all prediction modes and predictability are also supported by the decoder 54. The difference between the encoder 14 and the decoder 54 is mainly the fact that the encoder 14 selects or chooses coding decisions according to some optimization, such as minimizing some cost function which may depend on coding speed and / or coding distortion. One of these coding options or coding parameters may include the selection of an intra-prediction mode to be used for the current block 18 from among the available or supported intra-prediction modes. Next, the selected intra-prediction mode is signaled by the encoder 14 of the current block 18 in the data stream 12, and the decoder 54 uses this signaling of the data stream 12 of block 18 to redo the selection. Similarly, the subdivision of image 10 into blocks 18 can be subject to optimization within the encoder 14, the corresponding subdivision information can be transmitted in the data stream 12, and the decoder 54 recovers the subdivision of image 10 into blocks 18 based on the subdivision information. To summarize the above, the decoder 54 can be a block-based predictive decoder, and in addition to the intra-prediction mode, the decoder 54 can support other predictive modes, such as mutual predictive mode, if the decoder 54 is a video decoder. In decoding, the decoder 54 can also use the coding order 20 described with respect to Figure 1, which is followed by both the encoder 14 and the decoder 54 so that the same neighboring samples are available for the current block 18 in both the encoder 14 and the decoder 54.Therefore, in order to avoid unnecessary repetition, the description of the operating modes of the encoder 14 must also apply to the decoder 54, for example, as far as prediction is concerned and as far as coding of prediction residuals is concerned. The difference lies in the fact that the encoder 14, by optimization, selects or inserts several coding options or coding parameters and signals into the data stream 12, which are then derived from the data stream 12 by the decoder 54 for re-predicting, such as re-subdivision.

[0041] Figure 4 shows a possible implementation of the decoder 54 in Figure 3, i.e., one that fits the implementation of the encoder 14 in Figure 1, as shown in Figure 2. Since many elements of the encoder 54 in Figure 4 are the same as those occurring in the corresponding encoder in Figure 2, the same reference code with an apostrophe is used in Figure 4 to indicate these elements. In particular, the adder 42', the optional in-loop filter 46', and the predictor 44' are connected to the predictor loop in the same way as they are in the encoder in Figure 2. The reconstructed, i.e., inversely quantized and retransformed predictor residual signal applied to the added 42' is derived by a sequence of entropy decoders 56 that reverse the entropy coding of the entropy encoder 28b, followed by a residual signal reconstruction stage 36' consisting of an inverse quantizer 38' and an inverse converter 40', just as in the case of the coding side. The output of the decoder is the reconstruction of image 10. The reconstruction of image 10 may be available directly at the output of the adder 42' or at the output of the in-loop filter 46'. To improve image quality, several post-filters can be placed at the decoder output to apply some post-filtering to the reconstruction of image 10, but this option is not shown in Figure 4.

[0042] To reiterate, the explanation given above for Figure 2 is also valid for Figure 4, except that the encoder simply performs the relevant decisions regarding the optimization task and coding options. However, all explanations regarding block subdivision, prediction, inverse quantization, and retransformation are also valid for the decoder 54 in Figure 4.

[0043] Before proceeding to the description of possible examples of this application, some notes must be made regarding the above examples. Although not explicitly mentioned above, it is clear that block 18 can have any shape. It could be, for example, a rectangle or a quadratic shape. Furthermore, although the above description of the operating modes of encoder 14 and decoder 54 often refers to the “current block” 18, it is clear that encoder 14 and decoder 54 operate accordingly for each block in which the intra-predictive mode is selected. As mentioned above, other blocks are also possible, but the following description focuses on block 18 in which image 10 is subdivided and the intra-predictive mode is selected.

[0044] Refer to Figure 5 to summarize the circumstances of a particular block 18 in which an intra-prediction mode is selected. Figure 5 shows the current block 18, i.e., the block currently being encoded or decoded. Figure 5 shows a set 60 of adjacent samples 62, i.e., samples 62 with spatially adjacent blocks 18. The samples 64 in block 18 are to be predicted. Therefore, the derived predicted signal is the prediction for each sample 64 in block 18. As already mentioned above, a plurality of prediction modes 66 are available for each block 18, and when block 18 is intra-predicted, this plurality of modes 66 simply includes a mutual prediction mode. A selection 68 is performed on the encoder and decoder sides to determine one of the plurality of 66 intra-prediction modes used to predict (71) the predicted signal of block 18 based on the adjacent set of samples 60. The examples described further below differ in their operating modes regarding the available intra-prediction modes 66 and selection 68, for example, in terms of whether side information regarding the selection 68 for block 18 is set in the data stream 12. However, the description of these examples begins with a concrete description that provides mathematical details. According to this first example, the selection of a particular block 18 to be intra-predicted is associated with a corresponding side information signaling 70 and data stream, and the multiple intra-prediction modes 66 include a set 72 of neural network-based intra-prediction modes and a further set 74 of heuristic-designed intra-prediction modes. One of the intra-prediction modes in set 74 could be a DC prediction mode in which, for example, a certain mean is determined based on an adjacent sample set 60, and this mean is assigned to all samples 64 in block 18. Additionally or alternatively, set 74 could include a mutual prediction mode, which may be called an angular mutual prediction mode in which the sample values ​​of an adjacent sample set 60 are copied to block 18 along a specific predictive direction in which such angular intra-prediction modes differ.Figure 5 shows that the data stream 12 includes, as described above, a portion 76 of the prediction residuals encoded, in addition to side information 70 which may be present as needed regarding a selection 68 from a plurality of 66 intra-prediction modes, the encoding may include transform coding with quantization in the transform domain as needed.

[0045] In particular, to facilitate understanding of the following description of a specific example of this application, Figure 6 shows a general operating mode of the intra-prediction block in the encoder and decoder. Figure 6 shows block 18 and adjacent samples 60 set based on which intra-prediction is performed. Note that this set 60 may vary with respect to cardinality among multiple intra-prediction modes of 66 intra-prediction modes. That is, the number of samples in set 60 is actually used according to each intra-prediction mode for determining the predicted signal of block 18. However, this is for the sake of clarity and is not shown in Figure 6. Figure 6 shows the encoder and decoder using one neural network 800 to 80 for each of the set 72 neural network-based intra-prediction modes. KB This indicates that it has -1. Set 60 is applied to each neural network to derive the corresponding intra-prediction mode among Set 72. In addition, Figure 6 fairly representatively shows one block 82 as what it provides based on the input, i.e., the set 60 of adjacent samples, e.g., the DC mode prediction signal and / or the angular intra-prediction mode prediction signal, one or more prediction signals of one or more intra-prediction modes in Set 74. The following description assumes i=0···K B Neural network 80 having -1 iThis shows how the parameters can be determined favorably. The specific example shown below also provides the encoder and decoder with another neural network 84 dedicated to providing probability values ​​for each neural network-based intra-prediction mode in set 72 based on a set of adjacent samples 86 which may or may not match set 60. Thus, the probability values ​​are provided so that the neural network 84 can more effectively render the side information 70 for mode selection. For example, in the example described below, a variable-length code is used to point to one of the intra-prediction modes, and the probability value provided by the neural network 84, at least as far as set 72 is concerned, uses the variable-length code in the side information 70 as an index to an ordered list of intra-prediction modes ordered according to the probability values ​​output by the neural network 84 for the neural network-based intra-prediction modes in set 72, thereby optimizing or reducing the coding rate of the side information 70. For this reason, as shown in Figure 6, mode selection 68 is effectively performed depending on both the probability values ​​provided by the additional neural network 84 and the side information 70 in the data stream 12. 1. An algorithm for training the parameters of a neural network that performs intraprediction. A block of video frames, namely block 18

number

number

number

number

number

number

number

number

number

[0046] Next, we will explain some of the things that can happen with a typical hybrid video coding standard, namely Set 72, through a data-driven optimization approach.

number

[0047] 1. In the optimization algorithm we implement, we want to use a suitable approximation of the cost function, including the number of bits we expect to be able to spend to inform the prediction residuals.

[0048] 2. We would like to jointly train several intra-prediction models to be able to handle various signal characteristics.

[0049] 3. When training intra-prediction, you need to consider the number of bits required to indicate which intra-mode to use.

[0050] 4. We maintain a set of already defined intra-predictions, e.g., HEVC intra-predictions, and train our predictions as complementary predictions.

[0051] 5. Typical hybrid video coding standards usually define a specific block.

number

[0052] The following four sections will explain how each of these requirements can be addressed. More precisely, Section 1.1 will explain how to handle the first item. Section 1.2 will explain how to handle items 2 and 3. Section 1.4 will explain how to take item 4 into consideration. Finally, Section 1.5 will explain how to handle the last item. 1.1 Algorithm for training a loss function that approximates the rate function of a video codec A data-driven approach to determining unknown parameters used in a video codec is typically set up as an optimization algorithm that attempts to minimize a predefined loss function over a specific set of training examples. Typically, for a numerical optimization algorithm to actually work, the latter loss function must satisfy several smoothness requirements.

[0053] On the other hand, video encoders like HEVC have a rate distortion cost.

number

number

number

number

[0054] True function

number

number

number

[0055] More precisely, as before,

number

number

number

number

number

number

number

number

number

number

number

number

[0056] function

number

number

number

number

number

[0057] weight

number

number

number

number

number

number

[0058] For that task, the (stochastic) gradient descent method is typically used. 1.2 Training for predicting fixed block shapes This section will discuss specific blocks.

number

number

[0059] Let's assume we are given a predefined "architecture" of our predictions. This gives us some fixed

number

number

number

number

number

number

[0060] The following section will explain this point in detail. Function (2) is the neural network 800-80 in Figure 6. KB -1 is defined.

[0061] Next, the second parameter-dependent function

number

[0062] Similarly,

number

number

number

[0063] Similarly, an example using function (4) representing the neural network 84 in Figure 6 is shown in Section 1.3.

[0064] function

number

[0065] This function defines, for example, the VLC code length distribution used for side information 70, i.e., the code length associated with side information 70 having more sets 72 cad ponytes.

[0066] next,

number

number

[0067] For the time being,

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0068] Gradient updates involve the proportion of residuals and the probability distribution of the latter.

number

number

number

number

[0069] Given the loss function in (5), data-driven optimization can be performed

number

number

number

number

number

number

number

number

number

number

[0070]

number

number

number

number

number

number

number

number

number

number

number

number

[0071] Nonlinear activation function

number

number

[0072] Here,

number

number

number

Number

Number

Number

Number

[0073] function

Number

Number

Number

Number

[0074] Here,

Number

Number

Number

Number

[0075] Therefore,

Number

Number

Number

Number

Number

Number

Number

Number

Number

number

number

number

[0076] Similarly, function

number

number

number

number

[0077] Here,

number

number

number

number

number

number

number

[0078] In other words,

number

number

number

number

number

[0079] Next, the loss function is obtained from (5)

number

[0080] For large sets of training examples, maintain the notation from the end of the previous section.

number

number

[0081] To do this, typically, the weights are first found through optimization (6), then initialized with those weights to find the weights to optimize (10). 1.5 Collaborative training for predicting several block shapes This section explains how, in training predictions, we can take into account that common video coding standards typically allow for the division of blocks into smaller subblocks in various ways, and that it is usually possible to perform intra-predictions on these smaller subblocks.

[0082] In other words, several

number

number

number

number

number

[0083]

number

number

number

number

number

number

number

number

[0084] Regarding the given color components,

number

number

number

number

[0085] While maintaining the notation in Section 1.2,

number

number

number

number

number

number

number

[0086] moreover,

number

number

number

[0087] Similar to Section 1.4,

number

number

number

[0088] next,

number

number

number

number

number

number

[0089] next,

number

[0090] next,

number

[0091] lastly,

number

number

number

number

[0092] Usually, first

number

number

number

number

number

number

number

number

number

number

number

number

number

[0093] 2. The decoder reconstructs the flags, which are part of the side information 70, from the bitstream and indicates whether one of the following options is true: [label=)

[0094] (i)

number

[0095] (ii)

number

number

[0096] 3. If option 2 in step 2 is true, the decoder proceeds to the specified block 10, as in the case of the underlying hybrid video coding standard.

[0097] 4. If option 1 in step 2 is true, the decoder is defined according to (4).

number

number

number

number

[0098] (i) The decoder is

number

number

number

number

number

[0099] (ii) The decoder is

number

number

number

number

number

number

[0100] Next, the decoder extracts a unique bitstream from the bitstream 12, which is also part of the datastream 12.

number

[0101] The latter index

number

number

number

number

[0102] 5. If option 1 in step 2 is true and the decoder indexes according to the previous step 4

number

number

number

[0103] Integration into an existing hybrid video codec of an intra prediction function designed based on a data-driven learning approach. The description had two main parts. In the first part, a specific algorithm for the offline training of the intra prediction function was described. In the second part, how the video decoder uses the latter prediction function to generate a prediction signal for a specific block was described.

[0104] Therefore, what was described in Sections 1.1 to 2 above is, among other things, a device for decoding the image 10 from the data stream 12 in block units. The device 54 supports a plurality of intra prediction modes including a set 72 of intra prediction modes in which the intra prediction signal of the current block 18 of the image 10 is determined by applying a first set 60 of adjacent samples of the current block 18 to the neural network 80 i The device 54 selects (68) one intra prediction mode for the current block 18 from the plurality of intra prediction modes 66 and uses one intra prediction mode, that is, the corresponding neural network 80 that is selected m To predict (71) the current block 18 using. The decoder presented in Section 2 had an intra prediction mode 74 within a plurality of 66 of the supported plurality of intra prediction modes in addition to those based on the neural network of the set 72, but this is just an example and it doesn't have to be so. Furthermore, the above description of Sections 1 and 2 may be changed in that the decoder 54 does not use and does not include a further neural network 84. Regarding the above optimization, this is the finding

Number

[0105] A further alternative to the explanation presented in Section 2 above is that the decoder 54 may instead derive a ranking among a set of neural network-based intra-prediction modes 72, depending on the first part of the data stream relating to the adjacencies of the current block 18, in order to select the intra-prediction mode to be ultimately used from an ordered list of intra-prediction modes, depending on the second part of the data stream other than the first part, in order to obtain an ordered list of intra-prediction modes. The “first part” may relate, for example, to coding or prediction parameters relating to one or more blocks adjacent to the current block 18. The “second part” may be, for example, an index pointing to or being the index of the set of neural network-based intra-prediction modes 72. Interpreted in consistency with Section 2 outlined above, the decoder 54 comprises a further neural network 84 that determines the probability values ​​by applying a set of adjacency samples 86 to each intra-prediction mode in the set of intra-prediction modes 72, in order to order these probability values ​​to determine the rank of each intra-prediction mode in the set 72, thereby obtaining an ordered list of intra-prediction modes. Next, the index in the data stream 12 as part of the side information 70 is used as an index to the ordered list. Here, this index is M BThis can be encoded using a variable-length code that indicates the code length. Then, as described above in Section 2, in item 4i, according to a further alternative example, the decoder 54 can use the above probability values ​​determined by a further neural network 84 for each neural network-based intra-prediction mode of set 72 in order to efficiently perform entropy coding of the index to set 72. In particular, the symbolic alphabet of this index, which is part of the side information 70 and used as the index to set 72, contains the symbol or value of each mode in set 72, and the probability values ​​provided by the neural network 84 provide probability values ​​that lead to efficient entropy coding in the case of the design of the neural network 84 as described above, in that these probability values ​​strictly represent the actual symbolic statistics. For this entropy coding, for example, arithmetic coding or probability interval partitioning entropy (PIPE) coding can be used.

[0106] To be advantageous, no additional information is needed for any of the 72 intra-prediction modes in each neural network 80. iFor example, when parameterized favorably for encoders and decoders according to the above descriptions in Sections 1 and 2, it derives the predicted signal for the current block 18 without additional guidance to the data stream. As already shown above, the presence of other intra-prediction modes other than the neural network-based modes of set 72 is optional. They are shown above by set 74. In this regard, it should be noted that one possible way of selecting set 60, i.e., the set of adjacent samples that form the input to the prediction 71, may be heuristic, such that this set 60 is the same as for the intra-prediction modes of set 74. The set 60 of the neural network-based intra-prediction modes is larger in terms of the number of adjacent samples included in set 60 and that influence the intra-prediction 71. In other words, the cardinality of set 60 can be greater for the neural network-based intra-prediction mode 72 compared to the other modes of set 74. For example, set 60 of any intra-prediction mode of set 74 may simply include adjacent samples along a one-dimensional line extending along the sides of block 18, such as the left one and the top one. The neural network-based intra-prediction mode set 60 extends along the side just mentioned in block 18, but can cover an L-shaped portion wider than one sample width, like the intra-prediction mode set 60 in set 74. The L-shaped portion can extend further beyond the side just mentioned in block 18. In this way, the neural network-based intra-prediction mode can yield better intra-predictions with correspondingly lower prediction residuals.

[0107] As described in Section 2 above, the side information 70 transmitted to the intra-prediction block 18 in the data stream 12 may include a fleck that generally indicates whether the intra-prediction mode selected for block 18 is a member of set 72 or set 74. However, this fleck is merely an option, accompanied by side information 70 that indicates an index to a total of multiple intra-prediction modes 66, including both sets 72 and 74.

[0108] Below, the alternative described above is briefly outlined with respect to Figures 7a to 7d. The figures define both the decoder and encoder simultaneously, i.e., in terms of their functions with respect to the intra-prediction block 18. The difference between the encoder operating mode and the decoder operating mode with respect to the intra-coded block 18 is, on the one hand, that the encoder performs all or at least some of the available intra-prediction modes 66 and determines in step 90 which is optimal in terms of a cost function that minimizes meaning, for example, the encoder forms the data stream 12, i.e., the code fills it with dates, and the decoder derives data from it by decoding and reading, respectively. Figure 7a shows the operating mode of the alternative outlined above, indicating whether the flag 70a in the side information 70 of block 18 is the intra-prediction mode determined by the encoder in step 90 to be the best mode for block 18, either in set 72, i.e., a neural network-based intra-prediction mode, or in set 74, i.e., one of the non-neural network-based intra-prediction modes. The encoder inserts the flag 70a into the data stream 12 accordingly, while the decoder retrieves the flag 70a from there. Figure 7a assumes that the determined intra-prediction mode 92 is located in set 72. A separate neural network 84 then determines the probability value of each neural network-based intra-prediction mode in set 72, and using these probability values ​​set 72, or more precisely, ordering the neural network-based intra-prediction modes within it according to their probability values, such as in descending order of probability values, thereby yielding an ordered list 94 of intra-prediction modes. Next, the index 70b, which is part of the side information 70, is encoded into the data stream 12 by the encoder and decoded from there by the decoder. Thus, the decoder can determine which set of sets 72 and 74 to use. The intra-prediction mode to be used in block 18 is located in set 72, so as to perform the ordering 96 of set 72.If the determined intra-prediction mode is located in set 74, the index can also be transmitted in the data stream 12. Thus, the decoder can generate the prediction signal for block 18 using the determined intra-prediction mode by controlling selection 68 accordingly.

[0109] Figure 7b shows an alternative where flag 70a is not present in data stream 12. Instead, the ordered list 94 would include not only the intra-prediction modes of set 72 but also the intra-prediction modes of set 74. The index in side information 70 is an index to this larger ordered list, indicating the determined intra-prediction mode, i.e., the one determined is the optimization 90. For a neural network 84 that provides probability values ​​for neural network-based intra-prediction modes only within 72, the ranking of the intra-prediction modes of set 72 relative to the intra-prediction modes of set 74 can be determined by other means, such as necessarily arranging the neural network-based intra-prediction modes of set 72 to precede the modes of set 74 in the ordered list 94, or to alternate them with each other. That is, the decoder can derive an index from data stream 12 and use index 70 as an index to the ordered list 94 by deriving the ordered list 94 from multiple intra-prediction modes 66 using the probability values ​​output by the neural network 84. Figure 7c shows further variations. Figure 7c shows the case where flag 70a is not used, but the flag can be used instead. The issue addressed in Figure 7c relates to the possibility that neither the encoder nor the decoder uses the neural network 84. Rather, the ordering 96 is derived by other means, such as coding parameters transmitted within the data stream 12 with respect to one or more adjacent blocks 18, i.e., the portion of the data stream 12 98 relating to such one or more adjacent blocks.

[0110] Figure 7d shows a further variation of Figure 7a, namely, that index 70b is encoded using entropy coding and decoded from the data stream 12 using entropy decoding, generally indicated by the reference code 100. The sample statistics or probability distribution used for entropy coding 100 is controlled by the probability values ​​output by the neural network 84 as described above, which makes the entropy coding of index 70b very efficient.

[0111] For all examples 7a through 7d, it is true that the mode of set 74 may not exist. Therefore, each module 82 may be missing, and flag 70a is unnecessary anyway.

[0112] Furthermore, although not shown in any of the figures, it is clear that mode selection 68 in the encoder and decoder can be synchronized with each other without explicit signaling 70, i.e., without consuming side information. Rather, selection can be derived by other means, such as by necessarily taking the first one in the ordered list 94, or by deriving an index in the ordered list 94 based on coding parameters related to one or more adjacent blocks. Figure 8 shows an apparatus for designing a set of intra-predictive modes of set 72 used for block-based image coding. Apparatus 108 uses neural networks 800 to 80 KB-1 A parameterizable version of the neural network 84, as well as a parameterizable network 109 that inherits from or includes the neural network 84. Here, in Figure 8, as individual units, namely from the neural network 840 to provide the probability values ​​of the neural network-based intra-prediction mode 0, the neural network-based intra-prediction mode K B-1 A neural network 84 to provide related probability values ​​within KB-1 This is shown up to the parameter 111 for parameterizing neural network 84 and neural network 800 to 80 KB-1The parameters 113 for parameterizing are input or applied by the updater 110 to the respective parameter inputs of these neural networks. The device 108 has access to a reservoir or multiple image test blocks 114, along with corresponding adjacent sample sets 116. These pairs of blocks 114 and their associated adjacent sample sets 116 are used sequentially by the device 108. In particular, the current image test block 114 is applied to the parameterizable neural network 109, and the neural network 80 provides prediction signals 118 to each neural network-based intra-prediction mode of set 72, with each neural network 80 providing probability values ​​for each of these modes. For this purpose, these neural networks use the current parameters 111 and 113.

[0113] In the above explanation, rec is used to indicate image test block 114.

number

number

number

number

[0114] Therefore, the updater 110 attempts to update parameters 111 and 113 to reduce the coding cost function, and these updated parameters 111 and 113 are then used by a parameterizable neural network 109 to process the next multiple image test blocks 112. As described above with respect to Section 1.5, there may be a mechanism to control that primarily those pairs of image test blocks 114 and their associated adjacent sample sets 116 are applied to the recursive update process in which intra-predictions are made, in terms of rate distortion, preferably without subdividing the blocks, thereby avoiding that parameters 111 and 113 are over-optimized based on image test blocks for which coding in units of their sub-blocks is more cost-effective, in any case.

[0115] Up to this point, the above examples have primarily concerned the case where the encoder and decoder have a set of neural network-based intra-prediction modes within the supported intra-prediction modes 66. As shown in the examples with respect to Figures 9a and 9b, this is not necessarily the case. Figure 9a attempts to outline the operating modes of the encoder and decoder according to an example in which its explanation is provided in a way that focuses on the differences from the explanation presented above with respect to Figure 7a. The multiple supported intra-prediction modes 66 may or may not include neural network-based intra-prediction modes, and may or may not include non-neural network-based intra-prediction modes. Thus, the modules 170 in Figure 9a, each composed of an encoder and a decoder to provide each of the supported modes 66, do not necessarily have a neural network as the corresponding prediction signal. As already shown above, such intra-prediction modes can be neural network-based or heuristically motivated and can compute prediction signals based on either DC intra-prediction modes or angular intra-prediction modes or others. Thus, these modules 170 can be represented as prediction signal computers. However, the encoder and decoder in the example of Figure 9a include a neural network 84. The neural network 84 calculates probability values ​​for supported intra-prediction modes 66 based on adjacent sample sets 86, and as a result can transform multiple intra-prediction modes 66 into an ordered list 94. Index 70 in the data stream 12 of block 18 points to this ordered list 94. Thus, the neural network 84 helps reduce the side information rate spent on signaling the intra-prediction modes.

[0116] Figure 9b shows an alternative to Figure 9a, in that instead of ordering, the entropy decoding / coding 100 at index 70 is used to control its probability or simple statistics, i.e., to control the entropy probability distribution of the entropy decoding / coding in the encoder / decoder according to the probability values ​​determined for each of the multiple 66 modes for the neural network 84.

[0117] Figure 10 shows a device for designing or parameterizing a neural network 84. Thus, it is a device 108 for designing a neural network to assist in selecting from a set of intra-prediction modes 66. Here, for each mode in the set 66, corresponding neural network blocks come together to form the neural network 84, and the parameterizable neural network 109 of the device 108 is simply parameterizable with respect to these blocks. For each mode, there is also a prediction signal computer 170, which does not need to be parameterizable according to Figure 10. Thus, the device 108 in Figure 10 calculates a cost estimate for each mode based on the prediction signal 118 calculated by the corresponding prediction signal computer 170, and, if necessary, based on the corresponding probability value determined by the corresponding neural network block for that mode. Based on the resulting cost estimate 124, the minimum cost selector 126 selects the mode with the minimum cost estimate, and the updater 110 updates the parameters 111 of the neural network 84.

[0118] Regarding the explanation of Figures 7a to 7d and Figures 9a and 9b, please note the following. A common feature of the examples in Figures 9a and 9b, which are also used in some examples of Figures 7a to 7d, is the fact that they are probability values ​​of neural network values ​​to improve or reduce the overhead associated with side information 70 for notifying the decoder of the mode determined on the encoder side in the optimization process 90. However, as shown above with respect to the examples in Figures 7a to 7d, it should be clear that the examples in Figures 9a and 9b can be modified to the extent that no side information 70 is spent on the data stream 12 with respect to mode selection. Rather, the mode selection can necessarily be synchronized between the encoder and decoder using the probability values ​​output by the neural network 84 for each mode. In that case, there would be no optimization decision 90 on the encoder side with respect to mode selection. Rather, the modes used between sets 66 would be determined in the same way on both the encoder and decoder sides. If modified so that no secondary information 70 in the data stream 12 is used, similar statements apply to the corresponding examples in Figures 7a to 7d. However, returning to the examples in Figures 9a and 9b, it is interesting that the selection process 68 on the decoder side depends on the probability values ​​output by the neural network, in that the dependency on ordering or probability distribution estimation to the probability values ​​changes the interpretation of the side information as far as the encoder is concerned. This dependency on probability values ​​not only affects the encoding of the side information 70 to the data stream 12, for example, using variable-length coding of each index in an ordered list, or using entropy coding / decoding with probability distribution estimation depending on the probability values ​​of the neural network, but also the optimization step 90: here we can take into account the coding rate for transmitting the side information 70, and thus affect the decision 90. Example in Figure 11-1 Figure 11-1 shows a possible implementation of encoder 14-1, i.e., configured such that the encoder uses transform coding to encode predictive residuals, but this is merely an example and the present application is not limited to this type of predictive residual coding. According to Figure 11-1, encoder 14-1 includes a spatial domain subtractor 22 configured to subtract the corresponding predictive signal 24-1 from an inbound signal, i.e., image 10, or block-based from the current block 18, to obtain a spatial domain predictive residual signal 26, which is later encoded into a data stream 12 by predictive residual encoder 28. Predictive residual encoder 28 includes an irreversible coding stage 28a and a reversible coding stage 28b. The irreversible coding stage 28a includes a quantizer 30 that receives the predictive residual signal 26 and quantizes samples of the predictive residual signal 26. This example uses transform coding of the predicted residual signal 26, and therefore the irreversible coding stage 28a includes a transform stage 32 connected between the subtractor 22 and the quantizer 30 to transform such predicted residuals 27 spectrally decomposed by the quantizer 30 performing quantization of the transformed coefficients that present the residual signal 26. The transform can be DCT, DST, FFT, Hadamard transform, etc. The transformed and transformed domain-quantized predicted residual signal 34 then undergoes reversible coding by a reversible coding stage 28b, which is an entropy coder that entropy codes the quantized predicted residual signal 34 into a data stream 12.

[0119] Encoder 14-1 further comprises a transform-domain predictive residual signal reconstruction stage 36-1 connected to the transform-domain output of quantizer 30 to reconstruct the predictive residual signal from the transformed and quantized predictive residual signal 34 (in the transform domain) in a manner also usable by the decoder, i.e., to account for the coding loss of quantizer 30. For this purpose, predictive residual reconstruction stage 36-1 comprises an inverse quantizer 38-1 that performs the inverse of quantization of quantizer 30 to obtain an inverse quantized version 39-1 of the predictive residual signal 34, and a subsequent inverse converter 40-1 that performs the inverse transform to the transform performed by converter 32, such as the inverse of spectral decomposition, such as the inverse of any of the particular transform examples described above. Downstream of the inverse converter 40-1 is a spatial domain output 60 that can contain a template useful for obtaining the predictive signal 24-1. In particular, the predictor 44-1 may have a transformed domain output 45-1, which, when inverted by the inverse transformer 51-1, provides a predicted signal 24-1 in the spatial domain (the predicted signal 24-1 is subtracted from the inbound signal 10 to obtain the predicted residual 26 in the time domain). In interframe mode, an in-loop filter 46-1 may also filter the fully reconstructed image 60, and after filtering, form a reference image 47-1 of the predictor 44-1 with respect to the mutual predictive block (thus in these cases an adder 57-1 input from elements 44-1 and 36-1 is required, but the inverse transformer 51-1 is not needed to provide the predicted signal 24-1 to the subtractor 22, as shown by the dashed line 53-1).

[0120] However, unlike encoder 14 in Figure 2, encoder 14-1 (in the prediction residual reconstruction stage 36-1) includes a transformation domain adder 42-1 positioned between the inverse quantizer 38-1 and the inverse transformer 40-1. The transformation domain adder 42-1 provides the inverse transformer 40-1 with a total of 43-1 (in the transformation domain) of the inverse quantized version 39-1 of the prediction residual signal 34 (provided by the inverse quantizer 38-1), using a transformation domain prediction signal 45-1, such as that provided by the transformation predictor 44-1. The predictor 44-1 can take the output from the inverse transformer 40-1 as a feedback input.

[0121] Therefore, the spatial domain prediction signal 24-1 is obtained from the transformation domain prediction signal 45-1. Furthermore, the transformation domain predictor 44-1, which can operate in a neural network according to the above example, is input by the spatial domain signal but outputs the transformation domain signal. Example in Figure 11-2 Figure 11-2 shows a possible implementation of decoder 54-2, i.e., one that fits the implementation of encoder 14-1. Since many elements of encoder 54-2 are the same as those occurring in the corresponding encoder in Figure 11-1, the same reference designation with "-2" is used in Figure 11-2 to indicate these elements. In particular, the adder 42-2, an arbitrary in-loop filter 46-2, and the predictor 44-2 are connected to the predictive loop in the same way as the encoder in Figure 11-1. The reconstructed, i.e., inversely quantized and retransformed predictive residual signal 24-2 (e.g., 60) is derived by a sequence of entropy decoders 56 that reverse the entropy coding of entropy encoder 28b, followed by a residual signal reconstruction stage 36-2 consisting of an inverse quantizer 38-2 and an inverse converter 40-2, just as in the case of the coding side. The output of the decoder is the reconstruction of image 10. Several post-filters 46-2 can be placed at the output of the decoder to subject the reconstruction of image 10 to some post-filtering to improve image quality. Similarly, the explanation given above for Figure 11-1 is also valid for Figure 11-2, except that the encoder simply performs the relevant decisions regarding the optimization task and coding options. However, all explanations regarding block subdivision, prediction, inverse quantization, and retransformation are also valid for the decoder 54 in Figure 11-2. The reconstructed signal 24-2 is provided to the predictor 44-2, which can operate in a neural network according to the examples in Figures 5 to 10. The predictor 44-2 can provide the transformed domain predicted values ​​45-2.

[0122] Contrary to the example in Figure 4, but similar to the example in Figure 11-1, the inverse quantizer 38-2 provides an inverse quantized version 39-2 of the predicted residual signal 34 (within the transformation domain) which is not directly provided to the inverse converter 40-2. Instead, the inverse quantized version 39-2 of the predicted residual signal 34 is input to the adder 42-2 and composed of the transformation domain predicted values ​​45-2. Thus, the transformation domain reconstructed signal 43-2 is obtained, which, when inversely transformed by the inverse converter 40-2, becomes the reconstructed signal 24-2 in the spatial domain used to display the image 10. Example in Figure 12 Refer to Figure 12 here, both the decoder and encoder simultaneously, i.e., in terms of their functions with respect to the intra-prediction block 18. The difference between the encoder operating mode and the decoder operating mode with respect to the intra-coded block 18 is, on the one hand, that the encoder performs all or at least some of the available intra-prediction modes 66 and determines in step 90 the best one in terms of a cost function that minimizes meaning, for example, that the encoder forms the data stream 12, i.e., the code fills it with dates, and the decoder derives data from it by decoding and reading, respectively. Figure 12 shows the alternative operating modes outlined above, indicating whether the flag 70a in the side information 70 of block 18 is the intra-prediction mode determined by the encoder in step 90 to be the best mode for block 18, either in set 72, i.e., a neural network-based intra-prediction mode, or in set 74, i.e., one of the non-neural network-based intra-prediction modes. The encoder inserts the flag 70a into the data stream 12 accordingly, while the decoder retrieves the flag 70a from there. Figure 12 assumes that the determined intra-prediction mode 92 is located in set 72. Next, a separate neural network 84 determines the probability value of each neural network-based intra-prediction mode in set 72, and using these probability values ​​set 72, or more precisely, the neural network-based intra-prediction modes therein are ordered according to their probability values, such as in descending order of probability values, thereby yielding an ordered list 94 of intra-prediction modes. Next, the index 70b, which is part of the side information 70, is encoded into the data stream 12 by the encoder and decoded from there by the decoder. Thus, the decoder can determine which set 72 and 74 to use. The intra-prediction mode to be used in block 18 is located in set 72, in order to perform the ordering 96 of set 72. If the determined intra-prediction mode is located in set 74, the index can also be transmitted in the data stream 12.Therefore, the decoder can generate the prediction signal for block 18 using the determined intra-prediction mode by controlling selection 68 accordingly.

[0123] As can be seen from Figure 12, the predicted residual signal 34 (in the transformation domain) is encoded in the data stream 12. The inverse quantizers 38-1 and 38-2 derive the inverse quantized predicted residual signals 39-1 and 39-2 of the transformation domain. From the predictors 44-1 and 44-2, the transformation domain predicted signals 45-1 and 45-2 are obtained. Next, the adder 42-1 sums the values ​​39-1 and 45-1 together (or the adder 42-2 sums the values ​​39-2 and 45-2) to obtain the transformation domain reconstructed signal 43-1 (or 43-2). Downstream of the inverse converters 40-1 and 40-2, the spatial domain predicted signals 24-1 and 24-2 (e.g., template 60) are obtained and can be used to reconstruct block 18 (e.g., which can be shown).

[0124] All the modifications in Figures 7b to 7d can be used to illustrate the examples in Figures 11-1, 11-2, and 12. Discussion Methods for generating intra-prediction signals via neural networks are defined, and how these methods may be incorporated into video or still image codecs is described. In these examples, instead of predicting in the spatial domain, predictors 44-1, 44-2 may predict in the transform domain of predefined image transforms that may already be available in the underlying codec, such as the discrete cosine transform. Secondly, each intra-prediction mode defined for an image on a block of a particular shape induces an intra-prediction mode for an image on a larger block.

[0125] Let B be a block of pixels in the M row and N column where image im exists. The adjacent B of B (block 18) where the already reconstructed image rec is available. recAssume that templates (60 or 86) exist. Next, in the examples in Figures 5-10, a new intra-prediction mode defined by the neural network is introduced. Each of these intra-prediction modes uses the reconstructed sample rec(24-1, 24-2) and similarly B rec The predicted signal pred(45-1, 45-2), which is an image, is generated.

[0126] T, B rec Let the image transformation defined in the image above be (e.g., the predicted residual signal 34 output by element 30), and let S be the inverse transformation of T (e.g., 43-1 or 43-2). Then, the predicted signal pred(45-1, 45-2) is considered a prediction of T(im). This means that in the reconstruction stage, after calculating pred(45-1, 45-2), we need to calculate the image S(pred)(24-1, 24-2) to obtain the actual prediction of image im(10).

[0127] It should be noted that the transformation T we are working with has some energy compression properties relative to the natural image. This is exploited as follows: For each of the intra-modes defined by the neural network, a predefined rule sets the value of pred(45-1, 45-2) at a specific location in the transformation domain to zero, independently of the input rec(24-1, 24-2). This reduces the computational complexity required to obtain the predicted signal pred(45-1, 45-2) in the transformation domain.

[0128] (Referring to Figures 5-10, assume that the transform T(32) and inverse transform S(40) are used in the transform residual coding of the underlying codec. In the reconstructed signal (24, 24') of B, the predicted residual res(34) is inversely transformed by the inverse transform S(40) to obtain S(res), and S(res) is added to the underlying predicted signal (24) to obtain the final reconstructed signal (24).) In contrast, Figures 11 and 12 refer to the following procedure: If the predicted signal pred(45-1, 45-2) is generated by the neural network intraprediction method as described above, the final reconstructed signal (24-1, 24-2) is obtained by the inverse transform (40-1, 40-2) of pred + res (pred is 45-1 or 45-2, res is 39-1 or 39-2), the sum of which is 43-1 or 43-2, which is the transformed domain version of the final reconstructed signal 24-1, 24-2.

[0129] Finally, note that the above modifications to the intra-prediction performed by the neural network are arbitrary and unnecessarily interrelated with one another. This means that for a particular transform T(32) using the inverse transform S(40-1, 40-2) and one of the intra-prediction modes defined by the neural network above, it may be extracted from the bitstream or from a predefined configuration whether the mode is considered a prediction to the transform domain corresponding to T. Figures 13a and 13b Referring to Figures 13a and 13b, strategies that may be applied to, for example, spatial domain-based methods (e.g., Figures 11a and 11b) and / or transformation domain-based methods (e.g., Figures 1-4) are shown.

[0130] In some cases, a neural network fitted to a specific size block can be freely used (e.g., M×N, where M is the number of rows and N is the number of columns), but the actual blocks of the image being reconstructed have different sizes (e.g., M1×N1). Note that it is possible to perform operations that allow the use of a neural network fitted to a specific size (e.g., M×N) without having to use an ad-hoc trained neural network.

[0131] In particular, the device 14 or 54 can enable the decoding of an image (e.g., 10) from a data stream (e.g., 12) in block units. The devices 14, 54 natively support at least one intra-prediction mode, in which the intra-prediction signal for a block (e.g., 136, 172) of a given size (e.g., M × N) of the image is determined by applying a first template (e.g., 130, 170) of adjacent samples to the current block (e.g., 136, 176) on a neural network (e.g., 80). For a current block (e.g., 18) of a different size (e.g., M1 × N1), the device can be configured as follows: - To obtain a resampled template (e.g., 130, 170) conforming to the first template (e.g., 130, 170), resample the second template (e.g., 60) of the samples adjacent to the current block (e.g., 18) (e.g., D, 134, 166), - To obtain a preliminary intra-predicted signal (e.g., 138), apply a resampled template (e.g., 130, 170) of samples on a neural network (e.g., 80), - To obtain the intra-prediction signal for the current block, the preliminary intra-prediction signal (138) is resampled (e.g., U, V, 182) to match the current block (18, B1).

[0132] Figure 13a shows an example in the spatial domain. The spatial domain block 18 (also shown as B1) can be an M1xN1 block on which image im1 is reconstructed (even if image im1 is not yet available at this time). Template B 1,rec (For example, set 60) has an already reconstructed image rec1, where rec1 is adjacent to im1 (and B 1,rec Note that it is adjacent to B1. Block 18 and template 60 ("second template") can form element 132.

[0133] The dimension of B1 means that there may not be a neural network available to freely use to reconstruct B1. However, if a neural network is available for a block of a different dimension (such as the "first template"), the following steps can be taken.

[0134] The transformation operation (shown here as D or 134) can be applied to element 130, for example. However, since B1 is still unknown, the transformation D(130) is applied to B 1,rec It should be noted that this can be easily applied only to [this]. The transformation 130 can provide elements 136 formed from the transformed (resampled) template 130 and block 138.

[0135] For example, the M1xN1 block B1(18) (with unknown coefficients) can theoretically be transformed into the M×N block B(138) (with even more unknown coefficients). However, since the coefficients of block B(138) are unknown, it is not necessary to actually perform the transformation.

[0136] Similarly, transformation D(134) is template B 1,rec (60) Different templates B having different dimensions rec Convert to (130). Template 130 has a vertical thickness L (i.e., L columns of the vertical portion) and a horizontal thickness K (i.e., K rows of the horizontal portion), B rec =D(B 1,rec It can be an L-shape having ). It can be understood that template 130 may include the following: -B rec (130) The K x N block above, -B rec (130) The M x L ​​block on the left, and -B rec (130) Above, and B rec A K×L block to the left of a K×N block on an M×L block to the left of (130).

[0137] In some cases, the transformation operation D(134) can be a downsampling operation if M1 > M and N1 > N (especially if M is a multiple of M1 and N is a multiple of N1). For example, if M1 = 2M and N1 = 2N, the transformation operation D can be based on hiding some bins in a chess-like manner (e.g., B 1,rec Remove the diagonal from 60, B rec (Get a value of 130).

[0138] At this point, B rec (B rec =D(rec1)) is the image reconstructed in M×N. In passage 138a, devices 14, 54 can use the necessary neural network natively trained for the MxN block (e.g., in predictors 44, 44') (e.g., by operating as shown in Figures 5-10). By applying the above passage (138a), the image im1 of block B is obtained. (In some examples, passage 138a does not use a neural network but uses other techniques known in the art).

[0139] At this point, the size of image im1 in block B(138) is M×N, but the size of the displayed image needs to be M1×N1. However, it should be noted that it is simply possible to perform a transformation (e.g., U)140 to convert image im1 in block B(138) to M1xN1.

[0140] Note that if D performed in 134 is a downsampling operation, then U in 140 may be an upsampling operation. Therefore, U(140) can be obtained by introducing coefficients into the M1xN1 block in addition to the coefficients of the M×N block 138 obtained in the neural network operation 138a.

[0141] For example, if M1=2M and N1=2N, it is readily possible to perform interpolation (e.g., bilinear interpolation) to approximate ("guess") the coefficients of im1 that were discarded by the transformation D. Thus, the M1xN1 image im1 is obtained as element 142 and can be used to display the block image as part of image 10.

[0142] In particular, it is theoretically possible to obtain block 144, and nevertheless, it is the same as template 60 (except for errors due to transformations D and U). Therefore, advantageously, B, which is already freely available as template 60, can be used. 1,rec To obtain a new version of B rec There is no need to convert it.

[0143] The operation shown in Figure 13a can be performed, for example, in predictor 44 or 44'. Thus, the M1xN1 image im1(142) can be understood as the predicted signal 24 (Figure 2) or 24' (Figure 4) which is summed with the predicted residual signal output by the inverse transformer 40 or 40' to obtain the reconstructed signal.

[0144] Figure 13b shows an example in the transformation domain (for example, in the examples in Figures 11-1 and 11-2). Element 162 is represented as being formed by a spatial domain template 60 (already decoded) and a spatial domain block 18 (with unknown coefficients). Block 18 may have a size M1xN1 and may have unknown coefficients, which should be determined, for example, by predictor 44-1 or 44-2.

[0145] While a predetermined M×N-sized neural network can be freely used, there may be no neural network available to directly manipulate the M1×N1 blocks within the transformation domain.

[0146] However, it should be noted that in predictors 44-1 and 44-2, it is possible to obtain a spatial domain template 170 having a different dimension (e.g., a reduced dimension) by using the transformation D(166) applied to template 60 ("second template"). Template 170 ("first template") may have an L-shaped form, such as the shape of template 130 (see above).

[0147] At this point, in passage 170a, a neural network (for example, 800-80) is used. N ) can be applied according to any of the examples above (see Figures 5 to 10). Thus, at the end of passage 170a, the known coefficients of version 172 of block 18 can be obtained.

[0148] However, it should be noted that the dimension MxN of 172 does not fit the dimension M1xN1 of block 18 which must be visualized. Therefore, we can manipulate the transformation to the transformation domain (e.g., in 180). For example, an MxN transformation domain block T(176) can be obtained. To increase the number of rows and columns to M1 and N1 respectively, we can use a technique called zero-padding, for example, by introducing a value "0" corresponding to a frequency value associated with a frequency that does not exist in the M×N transformation T(176). Thus, a zero-padding region 178 can be used (e.g., it may have an L-shape). In particular, the zero-padding region 178 contains multiple bins (all zero) that are inserted into block 176 to obtain block 182. This can be obtained by the transformation V from T(transformed from 172) to T1(182). The dimension of T(176) does not match the dimension of block 18, but the dimension of T1(182) actually matches the dimension of block 18 due to the insertion of the zero-padding region 178. Furthermore, zero-padding is achieved by inserting higher frequency bins (which have zero values), which yields a result similar to interpolation.

[0149] Therefore, in adders 42-1 and 42-2, we can add the transformation T1(182), which is the version of 45-1 and 45-2. Then, the inverse transformation T -1 By executing this, you can obtain the reconstructed value 60 in the spatial domain used to visualize image 10.

[0150] The encoder can encode information about resampling (and the use of a neural network for blocks of a different size than block 18) into the data stream 12, and as a result, the decoder has that knowledge. Discussion Let B1 (for example, 18) be a block with M1 rows and N1 columns, and assume M1 ≥ M and N1 ≥ N. Let B1,rec be adjacent to B1 (for example, adjacent template 60), B 1,rec Region B is considered a subset of rec Let's assume (for example, 130). Let im1 (for example, 138) be the image of B1, and rec1 (for example, B 1,rec The coefficient of B 1,rec Assume the image is already reconstructed. The above solution is based on a predefined downsampling operation D (e.g., 134, 166) that maps the image of B1,rec to the image of B1. For example, if M1=2M and N1=2N, then B rec If B consists of K rows above B, L columns to the left of B, and a K×L corner in the upper left of B, and B1,rec consists of 2K rows above B1, 2L columns to the left of B, and a 2K×2L corner in the upper left of B1, then D can be an operation that applies a smoothing filter and then performs a downsampling operation of 2x in each direction. Therefore, D(rec1) is B rec This can be considered as a reconstructed image. Using the neural network-based intra-prediction mode described above, a prediction signal pred(45-1), which is the image on B, can be formed from D(rec1).

[0151] Here, we distinguish between two cases: Firstly, we assume that in B, a neural network-based intra-prediction makes predictions in the sample (spatial) domain, as shown in Figures 2, 4, and 13a. Let U(140) be a fixed upsampling filter that maps the image in B (e.g., 138) to the image in B1 (e.g., 142). For example, if M1=2M and N1=2N, U can be a bilinear interpolation operation. Next, we can form U(pred) to obtain the image on B1 (e.g., 45-1) which we consider to be the predicted signal of im1 (e.g., 10).

[0152] Secondly, as shown in Figures 11-1, 11-2, and 13b, assume that in B, the predicted signal pred (e.g., 45-2) should be considered as the predicted signal in the transformation domain with respect to the image transformation T on B using the inverse transform S. Let T1 be the image transformation on B1 using the inverse transform S1. Assume that a predefined mapping V is given that maps the image from the transformation domain of T to the transformation domain of T1. For example, if T is an M×N block discrete cosine transform using the inverse transform S, and T1 is an M1×N1 discrete cosine transform using the inverse transform S1, then the block of transformation coefficients of B can be mapped to the block of transformation coefficients of B1 by zero-padding and scaling (see, e.g., 178). This means setting all transformation coefficients of B1 to zero if the position in frequency space is greater than M or N in the vertical direction of the horizontal response, and copying the appropriately scaled transformation coefficients of B to the remaining M*N transformation coefficients of B1. Next, we can form V(pred) and obtain the elements of the transformation domain of T1, which are considered to be the predicted signal of T1(im1). The signal V(pred) can be further processed as described above.

[0153] As explained above with respect to Figures 1 to 10, we also described how to rank several intra-prediction modes in a particular block B by generating a conditional probability distribution between these modes using neural network-based operations, and how this ranking can be used to indicate which intra-prediction modes to apply in the current block. Using a downsampling operation (e.g., 166) at the input of the neural network that generates the latter ranking in the same way as the actual prediction modes will produce a ranking for extending the prediction modes to a larger block B1 than just described, and thus can be used to indicate which extended modes to use in block B1. Whether to generate a prediction signal on a given block B1 using neural network-based intra-prediction modes from a smaller block B can be predefined or signaled as side information in the underlying video codec. Other examples Generally speaking, a decoder as described above may have an encoder as described above, and / or vice versa. For example, encoder 14 may be decoder 54 or include decoder 54 (or vice versa). Encoder 14-1 may be decoder 54-2 (or vice versa), and so on. Furthermore, encoder 14 or 14-1 can also be understood as including a decoder itself, since the quantized prediction residual signal 34 forms a stream that is decoded to obtain the prediction signal 24 or 24-1.

[0154] While some embodiments have been described in the context of apparatus, it is clear that these embodiments also represent descriptions of corresponding methods, where blocks or apparatus correspond to method steps or features of method steps. Similarly, embodiments described in the context of method steps also represent descriptions of corresponding blocks, items, or functions of the corresponding apparatus. Some or all of the method steps can be performed by (or using) hardware devices such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some examples, one or more of the most important method steps can be performed by such devices.

[0155] The encoded data stream of the present invention can be stored in a digital storage medium or transmitted over a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

[0156] Depending on specific implementation requirements, examples of the present invention can be implemented in hardware or software. Implementation can be carried out using digital storage media such as floppy disks, DVDs, Blu-rays, CDs, ROMs, PROMs, EPROMs, EEPROMs, and flash memory, which store electronically readable control signals and cooperate (or can cooperate) with a programmable computer system to execute each method. Thus, the digital storage media can be computer-readable.

[0157] Some examples of the present invention include a data carrier having an electronically readable control signal that can cooperate with a programmable computer system so that one of the methods described herein can be performed.

[0158] Generally, examples of the present invention can be implemented as a computer program product comprising program code, which functions to perform one of the methods when the computer program product is executed on a computer. The program code may be stored, for example, in a machine-readable carrier.

[0159] Other examples include a computer program stored in a machine-readable carrier for performing one of the methods described herein.

[0160] Therefore, an example of the method of the present invention is a computer program having program code for performing one of the methods of the present invention when the computer program is executed on a computer.

[0161] Accordingly, a further example of the method of the present invention is a data carrier (or digital storage medium, or computer-readable medium) on which a computer program for performing one of the methods described herein is recorded. The data carrier, digital storage medium, or recorded medium is typically tangible and / or non-temporary.

[0162] Therefore, a further example of the method of the present invention is a data stream or sequence of signals representing a computer program for performing one of the methods described herein. The data stream or sequence of signals may be configured to be transmitted over a data communication connection, such as the Internet.

[0163] Further examples include processing means configured or adapted to perform one of the methods described herein, such as a computer or a programmable logic device.

[0164] Further examples include a computer on which a computer program for performing one of the methods described herein is installed.

[0165] Further examples of the present invention include an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0166] In some examples, a programmable logic device (e.g., a field-programmable gate array) can be used to perform some or all of the functions of the method herein. In some examples, a field-programmable gate array can work in conjunction with a microprocessor to perform one of the methods herein. Generally, the method is preferably performed by any hardware device.

[0167] The apparatus described herein may be implemented using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0168] The apparatus described herein, or any component of the apparatus described herein, may be implemented at least in part in hardware and / or software.

[0169] The methods described herein can be performed using hardware devices, using a computer, or using a combination of hardware devices and a computer.

[0170] Any method or apparatus described herein can be implemented at least partially by hardware and / or software.

[0171] The embodiments described above are merely illustrative of the principles of the present invention. It will be understood that modifications and variations of the configurations and details described herein will be obvious to those skilled in the art. Therefore, it is intended that the invention is limited only by the imminent claims and not by the descriptions of the examples and specific details presented herein.

Claims

1. A method for decoding an image from a data stream, From the reconstructed sample, identify a set of reference samples adjacent to the current block of the image, To obtain the transformed samples, a first transformation is performed on the set of reference samples, Based on the converted sample, an intermediate prediction sample is generated, In order to obtain a prediction sample corresponding to the current block, a second transformation is performed on the intermediate prediction sample, Reconstructing the current block using the residual signal and the predicted sample, Methods that include...

2. The converted samples have a smaller dimension than the set of reference samples. The predicted sample has a dimension corresponding to the current block, The method according to claim 1.

3. Performing the first transformation includes downsampling the set of reference samples, Performing the second transformation includes upsampling the intermediate prediction samples. The method according to claim 1.

4. The generation of the aforementioned intermediate prediction sample is To obtain a prediction mode adapted to a block having a size different from the current block size, Using the prediction mode, generate the intermediate prediction sample based on the converted sample, The method according to claim 1, including the method described in claim 1.

5. The block has a first width and a first height, The aforementioned current block has a second width and a second height, The first width is smaller than the second width. The first height is equal to the second height. The method according to claim 4.

6. The aforementioned set of reference samples includes samples arranged in an L-shaped template, The current block is not transformed before generating the intermediate prediction sample. The method according to claim 1.

7. An electronic device for decoding an image from a data stream, comprising at least one processor, The aforementioned at least one processor is From the reconstructed sample, identify the set of reference samples adjacent to the current block of the image, To obtain the converted samples, perform a first conversion on the set of reference samples, Based on the converted sample, an intermediate prediction sample is generated. In order to obtain a prediction sample corresponding to the current block, a second transformation is performed on the intermediate prediction sample. The current block is reconstructed using the residual signal and the predicted sample. An electronic device configured in such a way.

8. The converted samples have a smaller dimension than the set of reference samples. The predicted sample has a dimension corresponding to the current block, The electronic device according to claim 7.

9. To perform the first conversion, the at least one processor is further configured to downsample the set of reference samples, To perform the second transformation, the at least one processor is further configured to upsample the intermediate prediction samples. The electronic device according to claim 7.

10. To generate the aforementioned intermediate prediction samples, the at least one processor, A prediction mode is obtained that is adapted to a block having a size different from the current block size, Using the prediction mode, the intermediate prediction sample is generated based on the converted sample. The electronic device according to claim 7, configured as described above.

11. The block has a first width and a first height, The aforementioned current block has a second width and a second height, The first width is smaller than the second width. The first height is equal to the second height. The electronic device according to claim 10.

12. The aforementioned set of reference samples includes samples arranged in an L-shaped template, The current block is not transformed before generating the intermediate prediction sample. The electronic device according to claim 7.

13. When executed, at least one processor, From the reconstructed samples, identify the set of reference samples adjacent to the current block of the image. To obtain the converted samples, perform a first conversion on the set of reference samples. Based on the converted sample, an intermediate prediction sample is generated. In order to obtain a prediction sample corresponding to the current block, a second transformation is performed on the intermediate prediction sample. The current block is reconstructed using the residual signal and the predicted sample. A non-temporary computer-readable medium containing instructions.

14. When executed, at least one processor, From the reconstructed samples, identify the set of reference samples adjacent to the current block of the image. To obtain the converted samples, perform a first conversion on the set of reference samples. Based on the converted sample, an intermediate prediction sample is generated. In order to obtain a prediction sample corresponding to the current block, a second transformation is performed on the intermediate prediction sample. The current block is reconstructed using the residual signal and the predicted sample. A program that includes instructions.

Images