Operation of the neural network with adjusted weights

Abstract

The present disclosure relates to a method for operating a neural network based on adjusted weights. The method includes defining integer lower and upper thresholds for integer values ​​included in data entries of input data for a neural network layer. If the integer value included in the data entry of the input data is less than the integer lower threshold, the integer value included in the data entry of the input data is clipped to the lower threshold, and if the integer value included in the data entry of the input data is greater than the upper threshold, the integer value included in the data entry of the input data is clipped to the upper threshold, and the integer value weights are determined based on the lower threshold, the upper threshold, and a predefined accumulator register size to avoid integer overflow of an accumulator register.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the present disclosure relate generally to the field of encoding and decoding data based on neural network architectures. In particular, some embodiments relate to methods and apparatus for so encoding and decoding images and / or videos from a bitstream using multiple processing layers. [Background technology]

[0002] Hybrid image and video codecs have been used for decades to compress image and video data. In such codecs, the signal is typically encoded block-wise by predicting the block and then coding only the difference between the original block and its prediction. In particular, such coding involves transforming, quantizing, and generating a bitstream, which usually may include some entropy coding. Typically, the three components of a hybrid coding method - transforming, quantizing, and entropy coding - are optimized separately. Modern video compression standards such as High Efficiency Video Coding (HEVC), Generic Video Coding (VVC), and Essential Video Coding (EVC) also code the predicted residual signal using a transformed representation.

[0003] Recently, neural network architectures have been applied to image and / or video coding. In general, these neural network (NN)-based approaches can be applied in a variety of different ways to image and video coding. For example, several end-to-end optimized image or video coding frameworks have been discussed. Furthermore, deep learning has been used to determine or optimize parts of the end-to-end coding framework, such as prediction parameter selection or compression. Furthermore, several neural network-based approaches have been discussed for use in hybrid image and video coding frameworks, for example, for implementation as pre-trained deep learning models for intra- or inter-prediction in image or video coding.

[0004] The end-to-end optimized image or video coding applications mentioned above have in common that they generate some feature map data that is communicated between the encoder and the decoder.

[0005] A neural network is a machine learning model that uses one or more layers of nonlinear units, based on which it is possible to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. A corresponding feature map can be provided as an output of each hidden layer. Such a corresponding feature map of each hidden layer can be used as an input to the next layer in the network, i.e. the subsequent hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of the respective set of parameters. In a neural network that is partitioned between different devices, for example between an encoder and a decoder, or between a device and a cloud, the feature map at the output of the site involved in the partition (e.g., a first device) is compressed and transmitted to the remaining layers of the neural network (e.g., a second device).

[0006] Further improvements in encoding and decoding using trained network architectures would be desirable. Summary of the Invention

[0007] The present disclosure provides methods and apparatus for improving device interoperability of devices / platforms of different architectures, including neural networks. By device interoperability, it means that performing the same procedure on the same input data in two devices / platforms gives the same results in both devices / platforms. Particularly in the context of entropy model based coding and / or compression and decompression of data, such as image data, it is a non-trivial issue to provide substantially bit-exact processing results on the encoding and decoding sides, respectively, to provide the same or complementary technical effect.

[0008] These and other objects are achieved by the subject matter of the independent claims. Further implementations are evident from the dependent claims, the description and the drawings.

[0009] Particular implementations are outlined in the accompanying independent claims, while other implementations are defined in the dependent claims.

[0010] According to a first aspect, the present disclosure relates to a method of operating a neural network including a neural network layer including or connected to an accumulator register for buffering summation results and having a predefined accumulator register size. The method includes defining a lower integer threshold A and an upper integer threshold B for integer values ​​included in data entries (e.g., numbers, vectors, or tensors) of input data for the neural network layer. The method further includes clipping the integer values ​​included in the data entries of the input data to the predefined lower integer threshold if the integer values ​​included in the data entries of the input data are smaller than the predefined lower integer threshold, and clipping the integer values ​​included in the data entries of the input data to the predefined upper integer threshold if the integer values ​​included in the data entries of the input data are greater than the predefined upper integer threshold. Further, the method includes determining integer-valued weights (i.e., determined weights that include integers, e.g., only integers) of the neural network layers based on a predefined integer lower threshold, a predefined integer upper threshold, and a predefined accumulator register size such that integer overflow of the accumulator registers can be avoided.

[0011] According to the method of the first aspect, the integer weights of the neural network layers of the neural network are adjusted in such a way that integer overflow of the accumulator registers can be avoided, assuming that the input data is clipped to a predefined threshold. This allows for a significant improvement in interoperability between different platforms / devices compared to the prior art. The adjustment of the weights allows for a bit-exact reproduction of important numerical operations at the encoder and decoder side, respectively, so that the technical effects obtained by these numerical operations are the same or complementary to each other. For example, a region of an image (a still image or a frame of a video sequence) can be (entropy) coded at the encoder side and reconstructed uncorrupted at the decoder side if integer overflow of the accumulator registers can be avoided on both sides. Furthermore, by providing adjusted weights determined according to the method of the first aspect, the internal operation (state) of the associated device is defined in substantially the same way. The integer weights can be determined in such a way that integer overflow of the accumulator registers can be avoided in various ways, as described below, although the present disclosure is not limited to one of these specific ways.

[0012] The method of the first aspect works for any kind of neural network layer, including fully connected neural network layers and convolutional neural network layers. The method of the first aspect can also be advantageously implemented in a transformer architecture, where the neural network layer includes some kind of attention mechanism (see also the detailed description below). The accumulator register size is n bits, where n is a positive integer value, for example n=32 bits or n=16 bits. The stored value is -2 n-1 or 2 n-1 Within the range of -1 or 0 to 2 n-1 The accumulator register size can be a fixed size or can be dynamically allocated.

[0013] According to the implementation, the lower integer threshold is less than or equal to 0 and the upper integer threshold is greater than or equal to 0. Thus, a restriction to non-negative integer values ​​of the input data is possible, or possible or negative integer input values ​​may be allowed if deemed appropriate for the practical application.

[0014] According to the implementation, the integer lower threshold A is -2 k-1 and the upper integer threshold B is given by 2 k-1 −1, where k denotes a predefined bit depth of the layer input data, which is typically known for the particular application and configuration used and can therefore be readily used to define the clipping thresholds to be applied to the input data.

[0015] The adjusted weights are then used in the summation process typically performed by a neural network layer. Depending on the implementation, the neural network layer may perform the summation (accumulated in the so-called accumulator registers mentioned above)

[0016]

number

[0017] A specific concrete formulation for determining the integer weights is considered in this application so that integer overflow of the accumulator register can be avoided. According to the implementation, the integer weights of the neural network layers are determined by the following formula: i} is the condition

[0018]

number

[0019] Assuming that the input data values ​​are constrained to be between specified lower and upper thresholds A and B, integer overflow of the accumulator register can be reliably avoided if these conditions are met. n-1 -1 and 2 n-1 Note that is given only as an example for the accumulator register size limit and may be replaced by any other suitable limit defining the accumulator register size.

[0020] Another implementation uses integer weights {w i} is the condition

[0021]

number

[0022] Another implementation uses integer weights {w i} is the condition

[0023]

number

[0024] Addition in these implementations

[0025]

number

[0026]

number

[0027]

number

[0028]

number

[0029] In particular, user-dependent application weights of the neural network layers of the neural network may be provided (e.g. as real-valued weights, i.e. the weights contain real numbers, e.g. only real numbers). Also in the case of real-valued weights the method of the first aspect and its implementations can be advantageously used. In this case, according to the implementation, the provided real-valued weights can be scaled by a first scaling factor to obtain scaled weights, and the scaled weights can be rounded to their nearest integer value to obtain integer-valued weights. The scaling factor can be selected with high flexibility. 2 sj It seems appropriate to have a first scaling factor given by j indicates the number of bits to represent the fractional part of the real-valued weight.

[0030] In particular, according to an implementation of the method of the first aspect, a first scaling factor s for the j-th output channel of at least one neural network layer is j satisfies the following conditions:

[0031]

number

[0032] According to another implementation, the second scaling factor s for the j-th output channel of at least one neural network layer is j is given by:

[0033]

number

[0034]

number

[0035] All these conditions ensure that integer overflow will not occur.

[0036] Similarly, according to the implementation, the method includes scaling data entries of the input data (having real numbers or integers) by the second scaling factor to obtain scaled values ​​of the data entries. The scaled values ​​of the data entities may be rounded to their nearest integer values ​​to obtain integer values ​​of the data entities. The rounding may be performed by a floor function or a ceiling function.

[0037] As already mentioned, the method of the first aspect may advantageously be applied to data coding (encoding and decoding) with the same or similar advantages as above. Thus, according to a second aspect, a method of coding data is provided, comprising the steps of the method of operating a neural network according to the first aspect or any implementation thereof. In an implementation of the method of the second aspect, encoding the data comprises providing an entropy model with a neural network and entropy coding the data based on the provided entropy model, and providing the entropy model comprises performing the steps of the method of operating a neural network according to the first aspect or any implementation thereof. In the context of entropy coding, interoperability between different platforms is essential for a reliable reconstruction of the (compressed) data, with regard to a substantially bit-exact reproduction of the entropy model used both on the encoder side and on the decoder side.

[0038] The entropy model provides statistical (probabilistic) properties of the symbols being coded or decoded, such as the mean, variance, (cross)correlation, etc., and it a) Hyperprior of variational autoencoder, b) The autoregressive prior of the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder The method of the second aspect may therefore advantageously be implemented in a (variational) autoencoding system (see also the detailed description below).

[0039] According to a third aspect (and complementary to the second aspect), a method of decoding encoded data is provided, comprising the method steps of operating a neural network according to the first aspect or any implementation thereof. According to an implementation of the third aspect, decoding the data comprises providing an entropy model using a neural network and entropy decoding the data based on the provided entropy model, wherein entropy decoding the data comprises the method steps of operating a neural network according to the first aspect or any implementation thereof. Also, on the decoder side, according to an implementation of the third aspect, the entropy model is a) Variational autoencoder hyperprior, b) Autoregressive prior of the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder It is provided using one of the following:

[0040] The data processed by the method of the second or third aspect may be image data, for example representing a still image or a frame of a video sequence.

[0041] According to a fourth aspect, there is provided a method of encoding at least a portion of an image, the method comprising: Converting the tensors representing the components of the image into latent tensors; Providing an entropy model; and processing the latent tensors using a neural network based on the provided entropy model to generate a bitstream; and the step of providing the entropy model comprises performing the steps of the method according to the first aspect or an implementation thereof (having the same or similar advantages as above). According to an implementation of the fourth aspect, the entropy model comprises: a) Variational autoencoder hyperprior, b) Autoregressive prior of the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder It is provided using one of the following:

[0042] According to a fifth aspect, there is provided a method of reconstructing at least a portion of an image, the method comprising: Providing an entropy model; processing the bitstream using a neural network based on the provided entropy model to obtain latent tensors representing components of the image; and processing the latent tensors to obtain tensors representing components of the image; and the step of providing the entropy model comprises performing the steps of the method according to the first aspect or an implementation thereof (having the same or similar advantages as above). Again, the entropy model is a) Variational autoencoder hyperprior, b) Autoregressive prior of the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder may be provided using one of

[0043] The image components mentioned above may be Y, U or V components, or alternatively R, G or B components.

[0044] According to a sixth aspect, the present disclosure relates to a method of operating a neural network including at least one neural network layer including or connected to an accumulator register for buffering summation results. The method includes defining a lower integer threshold and an upper integer threshold for integer values ​​included in data entries of input data for the at least one neural network layer. The method further includes clipping the integer values ​​included in the data entries of the input data to the predefined lower integer threshold if the integer values ​​included in the data entries of the input data are smaller than the predefined lower integer threshold, and clipping the integer values ​​included in the data entries of the input data to the predefined upper integer threshold if the integer values ​​included in the data entries of the input data are larger than the predefined upper integer threshold, such that integer overflow of the accumulator register is avoided.

[0045] According to the method of the sixth aspect, due to clipping of integer-valued data (including integer values) input to the neural network layer, interoperability between different platforms / devices can be significantly improved compared to the past. Clipping allows bit-exact reproduction of important numerical operations at the encoder and decoder side, so that the technical effects obtained by these numerical operations are the same or complementary to each other. For example, an area of ​​an image (a still image or a frame of a video sequence) can be (entropy) coded at the encoder side and reconstructed without corruption at the decoder side, if integer overflows that may lead to undefined behavior on different platforms can be reliably avoided. In general, there is no standardized handling of overflow situations, and it is hardly possible to imagine a procedure that would be commonly followed in this regard. Integer overflow occurs when an arithmetic operation attempts to create a number that is outside the range that can be represented with a given number of digits - either larger than the maximum representable value or smaller than the minimum representable value. This situation may be handled differently by different compilers, devices (CPU, GPU), etc. In order to achieve bit-exact results of integer arithmetic on different platforms, integer overflow should be avoided. According to the method of the sixth aspect, integer overflow of accumulator registers can be avoided both on the encoder side and on the decoder side. Furthermore, by clipping input data values ​​according to the method of the sixth aspect, the internal operation (state) of the involved devices is defined in a substantially similar manner.

[0046] According to implementation, the method of the sixth aspect further comprises scaling the data entities of the input data having real values ​​by the first scaling factor (i.e. the data entities of the input data are multiplied by the first scaling factor) to obtain scaled values ​​of the data entities of the input data. The scaling may be performed to improve the processing of the input data values ​​in different devices. In particular, the scaling may be supplemented by rounding the scaled values ​​of the data entities of the input data to the respective nearest integer values ​​to obtain integer values ​​of the data entities of the input data. This allows input data containing real values, which may be inherently provided, to be processed by the disclosed method of operating a neural network.

[0047] The output data may be obtained by processing the input data by at least one neural network layer, and if a first scaling factor is applied, according to the implementation, the output data entities included in the output data are divided by the first scaling factor to obtain de-scaled results. The de-scaling may be performed directly on the output of the at least one neural network layer or after processing by the activation function. Thus, according to the implementation, the input data are processed by the at least one neural network layer to obtain output data including output data entities, the output data entities are processed by the activation function to obtain an output of the activation function, and the output of the activation function is divided by the first scaling factor.

[0048] According to another implementation, the method of the sixth aspect further comprises: processing the input data by at least one neural network layer to obtain output data including output data entities; factoring a first scaling factor into a first part and a second part; dividing the output data entities by a first part of the factorized first scaling factor to obtain partially un-scaled output data entities; processing the partially unscaled output data entities through an activation function to output an output of the activation function; and dividing the output of the activation function by the second part of the factorized first scaling factor. Since all descaling options can be implemented, the descaling process can be performed with a high degree of flexibility.

[0049] The integer lower threshold used in the method of the sixth aspect may be less than or equal to 0, and the integer upper threshold may be greater than or equal to 0. Suitable possible choices for the thresholds used are -2 for the integer lower threshold and k-1 and for the upper integer threshold, 2 k-1 -1, where k denotes the predefined bit depth (bit size) of the input data.

[0050] The method of the sixth aspect may be applied to any kind of neural network and neural network layers. According to the implementation, at least one neural network layer is or includes one of a fully connected neural network layer and a convolutional neural network layer. Furthermore, at least one neural network layer may include an attention mechanism (see detailed description below).

[0051] Depending on the implementation, at least one neural network layer contains integer-valued weights, which may further improve platform / device interoperability and further reduce the risk of integer overflow of accumulator register sizes (see also discussion below).

[0052] According to an implementation, real-valued weights (weights that include real numbers) are originally provided and scaled by a second scaling factor to obtain scaled weights, and the scaled weights are rounded to their nearest integer value to obtain integer-valued weights of at least one neural network, thereby allowing the method of the sixth aspect to be used for real-valued weights that are originally provided (e.g., depending on a user).

[0053] The second scaling factor is the number of bits to represent the fractional part of the real-valued weights. sj In particular, depending on the implementation, the second scaling factor s for the j-th output channel of at least one neural network layer can be given by j satisfies the following conditions:

[0054]

number

[0055] According to another implementation, the second scaling factor s for the j-th output channel of at least one neural network layer is j is given by:

[0056]

number

[0057]

number

[0058] All these conditions ensure that integer overflow will not occur.

[0059] As already mentioned, the method of the sixth aspect may advantageously be applied to data coding (encoding and decoding) with the same or similar advantages as above. In particular, the coding and decoding of data based on entropy coding represents a delicate process, in which at least some interoperability of procedures performed on different devices should give the same results in numerical precision. Thus, according to the seventh aspect, a method of coding data (with the same advantages as above) is provided, comprising steps of the method of operating a neural network according to the sixth aspect or any implementation thereof. In an implementation of the method of the seventh aspect, encoding the data comprises providing an entropy model using a neural network and entropy coding the data based on the provided entropy model, and providing the entropy model comprises performing steps of the method of operating a neural network according to the sixth aspect or any implementation thereof. The entropy coding may comprise entropy coding using an arithmetic encoder.

[0060] In the context of entropy coding, interoperability between different platforms with regard to virtually bit-exact reproduction of the entropy model used both on the encoder and decoder side is essential for reliable reconstruction of the (compressed) data.

[0061] The entropy model provides statistical (probabilistic) properties of the symbols being coded or decoded, such as the mean, variance, (cross)correlation, etc., and it a) Variational autoencoder hyperprior, b) Autoregressive prior of the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder The method of the seventh aspect may therefore advantageously be implemented in a (variational) autoencoding system (see also the detailed description below).

[0062] The method for encoding data as described above may further include signaling the defined lower threshold and the defined upper threshold to a decoder side, which can easily provide the decoder side with the necessary information for the same processing for clipping the input data as performed at the encoder side. Similarly, at least one of the first scaling factor, the second scaling factor and the third scaling factor may be signaled to the decoder side in the bitstream.

[0063] The method may further include signaling the difference relative to the predefined lower threshold and the difference relative to the predefined upper threshold to the decoder side. The method may further include signaling at least one of the difference relative to the predefined first scaling factor and the difference relative to the predefined second scaling factor to the decoder side in the bitstream. Exponential-Golomb coding may be used for the signaling.

[0064] According to an eighth aspect (and complementary to the seventh aspect), a method of decoding encoded data is provided, comprising the steps of the method of operating a neural network according to the sixth aspect or any implementation thereof (having the same advantages as above). According to the eighth implementation, decoding the data comprises providing an entropy model using a neural network and entropy decoding the data based on the provided entropy model, and entropy decoding the data comprises the steps of the method of operating a neural network according to the sixth aspect or any implementation thereof. The entropy decoding may comprise entropy decoding using an arithmetic decoder. Also, on the decoder side, according to an implementation of the eighth aspect, the entropy model is a) Variational autoencoder hyperprior, b) Autoregressive prior of the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder It is provided using one of the following:

[0065] Information regarding the defined lower threshold and the defined upper threshold may be received at the decoder side in a bitstream from the encoder side. Similarly, information regarding at least one of the first scaling factor, the second scaling factor, and the third scaling factor may be received at the decoder side in a bitstream from the encoder side.

[0066] Further, the method may include receiving, at the decoder side, information regarding the difference relative to the predefined lower threshold and the difference relative to the predefined upper threshold.Further, the method may include receiving, at the decoder side, information regarding at least one of the difference relative to the predefined first scaling factor and the difference relative to the predefined second scaling factor in the bitstream.

[0067] The data processed by the method of the second or eighth aspect may be image data, for example a still image or a frame of a video sequence.

[0068] According to a ninth aspect, there is provided a method of encoding at least a portion of an image, the method comprising: Converting the tensors representing the components of the image into latent tensors; Providing an entropy model; and processing the latent tensors using a neural network based on the provided entropy model to generate a bitstream; and providing the entropy model comprises performing steps of a method according to the sixth aspect or an implementation thereof (having the same or similar advantages as above). The latent tensor may be processed using an arithmetic encoder.

[0069] The entropy model is a) Variational autoencoder hyperprior, b) Autoregressive prior of the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder may be provided using one of

[0070] According to implementation, the method of the ninth aspect may further comprise signaling the defined lower threshold and the defined upper threshold to the decoder side in the bitstream.

[0071] Furthermore, at least one of the first scaling factor, the second scaling factor, and the third scaling factor may be signaled to the decoder side in the bitstream.

[0072] The method may further include signaling the difference relative to the predefined lower threshold and the difference relative to the predefined upper threshold to the decoder side. The method may further include signaling at least one of the difference relative to the predefined first scaling factor and the difference relative to the predefined second scaling factor to the decoder side in the bitstream. Exponential-Golomb coding may be used for the signaling.

[0073] According to a tenth aspect, there is provided a method of reconstructing at least a portion of an image, the method comprising: Providing an entropy model; processing the bitstream using a neural network based on the provided entropy model to obtain latent tensors representing components of the image; and processing the latent tensors to obtain tensors representing components of the image; and the step of providing the entropy model comprises performing the steps of the method of operating a neural network according to the sixth aspect or any implementation thereof (having the same advantages as above). The processing of the bitstream may be performed using an arithmetic decoder. The entropy model may be a) Variational autoencoder hyperprior, b) Autoregressive prior of the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder may be provided using one of

[0074] According to an implementation, the method of the tenth aspect includes reading information about the defined lower threshold and the defined upper threshold from the bitstream. Further, at least one of the first scaling factor, the second scaling factor, and the third scaling factor may be read from the bitstream. Further, the method may include receiving, at the decoder side, information regarding the difference relative to the predefined lower threshold and the difference relative to the predefined upper threshold.Further, the method may include receiving, at the decoder side, information regarding at least one of the difference relative to the predefined first scaling factor and the difference relative to the predefined second scaling factor in the bitstream.

[0075] The components of an image represented by a tensor according to the methods of the third and ninth aspects may be Y, U or V components, or alternatively R, G or B components.

[0076] According to an eleventh aspect, there is provided a neural network including a neural network layer configured to process input data to obtain output data, an activation function configured to process the output data to obtain activation function output data, a first unit for scaling, rounding and clipping the input data input to the neural network layer, and a second unit for de-scaling at least one of the output data and the activation function output data, where the first unit is divided into sub-units, each sub-unit configured for different operations such as scaling, rounding and clipping the input data, and the on and off operations of the first unit and each sub-unit are respectively switchable.

[0077] The first unit may be configured to perform the steps of the methods according to the first to sixth aspects and their implementations.

[0078] According to implementations, the neural network of the eleventh aspect is configured to perform the steps of the methods of the first to sixth aspects and their implementations.

[0079] According to a twelfth aspect, there is provided an apparatus for encoding at least a portion of an image, the apparatus including a neural network according to the twelfth aspect or an implementation thereof. a) Variational autoencoder hyperprior, b) Autoregressive prior of the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder wherein each prior comprises a neural network of the twelfth aspect or an implementation thereof.

[0080] According to a thirteenth aspect, there is provided an apparatus for decoding at least a portion of an encoded image, the apparatus comprising a neural network according to the twelfth aspect or an implementation thereof. This may be an apparatus for decoding at least a portion of an image, and a) Variational autoencoder hyperprior, b) Autoregressive prior of the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder wherein each prior comprises a neural network of the twelfth aspect or an implementation thereof.

[0081] To ensure device interoperability across neural networks, bit-exact reproducibility of activation functions on different platforms / devices is desirable. For linear activation functions and relatively simple non-linear activation functions, e.g., the ReLU function, which basically defines a clipping procedure, this requirement can be met relatively easily. For more complex non-linear ones, especially those involving exponential functions, e.g., Softmax:

[0082]

number

[0083] Thus, according to a fourteenth aspect, there is provided a neural network comprising at least one neural network layer and an activation function connected to an output of the at least one neural network layer, the activation function being implemented as an approximation function of a mathematically defined real-valued non-linear activation function (taking real numbers as arguments and outputting real numbers), the approximation function allowing processing only integers in a fixed-point representation of input values ​​of the approximation function.

[0084] In fixed-point representation / arithmetic, real numbers are Mantissa*Radix 指数 where both the base and the exponent are fixed, and thus a fixed-point representation of a fraction is essentially an integer (see also the detailed explanation below).

[0085] The approximation function approximates a mathematically defined real-valued nonlinear activation function used in the art (which often involves one or more exponential functions (with base e) and operates on floating-point representations of the input values) so that only integers in the fixed-point representation of the input values ​​can be processed (as is the case for example with the ReLU function).

[0086] Integer-only processing allows fixed-point neural networks in that bit-exact behavior on different platforms may be obtained. This allows for a significant improvement in interoperability between different platforms / devices compared to the prior art. Providing such approximation activation functions allows for bit-exact reproduction of important numerical operations at the encoder and decoder side, respectively, so that the technical effects obtained by these numerical operations are identical or complementary to each other. For example, if it is possible to obtain bit-exact behavior on both sides, a region of an image (a still image or a frame of a video sequence) can be (entropy) coded at the encoder side and reconstructed uncorrupted at the decoder side. Furthermore, by providing a neural network including an approximated activation function according to the method of the fourteenth aspect for different devices communicating with each other, the internal operation (state) of the associated devices is defined in substantially the same way. The approximation activation function implemented in the neural network of the fourteenth aspect may be selected in different ways, as explained below, and the disclosure is not limited to one of these specific ways.

[0087] According to implementations, the approximation function comprises at least one of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function. According to further embodiments of the neural network of the fourteenth aspect, the mathematically defined non-linear activation function (i.e., the one approximated by the approximation activation function) is selected from the group consisting of a Softmax function, a sigmoid function, a hyperbolic tangent function, a swish function, a Gaussian error linear unit function, and a scaling exponential linear unit function. For all of these mathematically defined non-linear activation functions that are considered suitable for practical applications, it is possible to find approximation functions that allow processing only integers of a fixed-point representation of the input values.

[0088] According to the implementation, the approximation function (approximate activation function) is a) an approximation function or expected input values ​​of the data input to at least one neural network layer, b) the size of the accumulator register for buffering the summation results formed by the neural network, and c) Approximation functions The method includes determining a finite number of Taylor series sums based on at least one of:

[0089] According to another implementation, the approximation function is a polynomial function whose maximum degree is a) an approximation function or expected input values ​​of the data input to at least one neural network layer, b) the size of the accumulator register for buffering the summation results formed by the neural network, and c) Approximation functions is determined based on at least one of the following:

[0090] According to another embodiment, the approximation function is a rational function, and the maximum degree of the polynomials in the numerator and denominator of the rational function is: a) an approximation function or expected input values ​​of the data input to at least one neural network layer, b) the size of the accumulator register for buffering the summation results formed by the neural network, and c) Approximation functions is determined based on at least one of the following:

[0091] For a mathematically defined nonlinear activation function f(x), the Taylor series

[0092]

number

[0093]

number

[0094] 2 n-1 -1 and -2 n-1 Note that is only an example regarding the accumulator register size restrictions and may be replaced by other restrictions that properly define the accumulator register sizes in the conditions.

[0095] The activation function that is particularly important in many applications is Softmax.

[0096]

number

[0097] According to the implementation of the neural network of the fourteenth aspect, the mathematically defined non-linear activation function is a Softmax function, and the approximation function (approximate activation function) is defined as follows:

[0098]

number

[0099] According to another implementation, the approximate activation function is defined as follows:

[0100]

number

[0101] The positive constant ε can be easily selected, for example, according to the expected values ​​of the components of the input vector x. The positive constant ε is, for example, 10 -15 or 10 -11 The range may be:

[0102] Numerical experiments prove that this kind of approximation function to the Softmax function based on the ReLU function can ensure the bit-exact behavior of the application of the activation function on different platforms.

[0103] The implementation of the approximate activation function can advantageously be realised in any neural network, in particular the neural network may be a convolutional or fully connected neural network or may include convolutional or fully connected neural network layers.

[0104] According to a fifteenth aspect, there is provided a method of operating a neural network (e.g. a convolutional or fully connected neural network) comprising at least one neural network layer (having the same advantages as those provided by the neural network of the fourteenth aspect), the method comprising implementing an approximation function of a mathematically defined real-valued non-linear activation function as an activation function of the at least one neural network layer, the approximation function allowing processing only integers in a fixed-point representation of input values ​​of the approximation function.

[0105] According to an implementation, the approximation function includes at least one of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function. According to a further implementation, the mathematically defined non-linear activation function is selected from the group consisting of a Softmax function, a sigmoid function, a hyperbolic tangent function, a swish function, a Gaussian error linear unit function, and a scaling exponential linear unit function.

[0106] The approximation function may include at least one neural network layer or a finite number of Taylor series sums determined based on expected input values ​​of the data input to the approximation function.

[0107] Taylor series of a mathematically defined nonlinear activation function f(x)

[0108]

number

[0109]

number

[0110] The mathematically defined non-linear activation function may be a Softmax function, in which case the approximation function implemented according to the fifteenth aspect or method of implementation thereof is given as:

[0111]

number

[0112] Alternatively, the approximation function can be defined as follows:

[0113]

number

[0114]

number

[0115] The approximate activation function implemented in the neural network of the 14th aspect and the method of the 15th aspect may advantageously be applied to data coding (encoding and decoding) with the same or similar advantages as those mentioned above. According to the 16th aspect, then, a method of encoding data is provided (with the same advantages as above) including the steps of the method of operating a neural network according to the 14th aspect or any implementation thereof. In an implementation of the method of the 16th aspect, encoding the data includes providing an entropy model using the neural network and entropy encoding the data based on the provided entropy model, and providing the entropy model includes performing the steps of the method of operating a neural network according to the 15th aspect or any implementation thereof. The entropy encoding may include entropy encoding using an arithmetic encoder.

[0116] In the context of entropy coding, interoperability between different platforms with regard to virtually bit-exact reproduction of the entropy model used both on the encoder and decoder side is essential for reliable reconstruction of the (compressed) data.

[0117] The entropy model provides statistical (probabilistic) properties of the symbols being coded or decoded, such as the mean, variance, (cross)correlation, etc., and it a) Variational autoencoder hyperprior, b) Autoregressive prior of the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder The method of the sixteenth aspect may therefore advantageously be implemented in a (variational) autoencoding system (see also the detailed description below).

[0118] According to a seventeenth aspect (and complementary to the sixteenth aspect), a method of decoding encoded data is provided, comprising the steps of the method of operating a neural network according to the fifteenth aspect or any implementation thereof (having the same advantages as above). According to an implementation of the seventeenth aspect, decoding the data comprises providing an entropy model using a neural network and entropy decoding the data based on the provided entropy model, and entropy decoding the data comprises the steps of the method of operating a neural network according to the fifteenth aspect or any implementation thereof. The entropy decoding may comprise entropy decoding using an arithmetic decoder. Also, on the decoder side, according to an implementation of the seventeenth aspect, the entropy model is a) Variational autoencoder hyperprior, b) Autoregressive prior of the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder It is provided using one of the following:

[0119] The data processed by the method of the sixteenth or seventeenth aspect may be image data, for example representing a still image or a frame of a video sequence.

[0120] According to an eighteenth aspect, there is provided a method of encoding at least a portion of an image, the method comprising: Converting the tensors representing the components of the image into latent tensors; Providing an entropy model; and processing the latent tensors using a neural network based on the provided entropy model to generate a bitstream; and providing the entropy model comprises performing steps of a method according to the fifteenth aspect or an implementation thereof (having the same advantages as above). The latent tensor may be processed using an arithmetic encoder.

[0121] The entropy model is a) Variational autoencoder hyperprior, b) Autoregressive prior of the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder may be provided using one of

[0122] According to a nineteenth aspect, there is provided a method of reconstructing at least a portion of an image, the method comprising: Providing an entropy model; processing the bitstream using a neural network based on the provided entropy model to obtain latent tensors representing components of the image; and processing the latent tensors to obtain tensors representing components of the image; and the step of providing the entropy model comprises performing the steps of the method of operating a neural network according to the fifteenth aspect (having the same advantages as above) or any implementation thereof. The processing of the bitstream may be performed using an arithmetic decoder. The entropy model may be a) Variational autoencoder hyperprior, b) Autoregressive prior of the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder Furthermore, processing the latent tensors to obtain tensors representing components of the image may include performing steps of a method according to the twenty-first aspect or an implementation thereof.

[0123] The components of an image represented by a tensor according to the methods of the eighteenth and nineteenth aspects may be Y, U or V components, or alternatively R, G or B components.

[0124] According to a twentieth aspect, an approximate activation function (one that approximates a Softmax function) is provided as follows:

[0125]

number

[0126] According to a twenty-first aspect, an approximate activation function (one that approximates a Softmax function) is provided as follows:

[0127]

number

[0128] The positive constant ε can be easily selected, for example, according to the expected values ​​of the components of the input vector x. The positive constant ε is, for example, 10 -15 or 10 -11 The range may be:

[0129] These new activation functions can replace the traditional Softmax function and may prove to be superior in terms of device interoperability (as encoders and decoders).

[0130] According to a twenty-second aspect, there is provided an apparatus for encoding data, comprising a neural network according to the fourteenth aspect or any implementation thereof.

[0131] According to a twenty-third aspect, there is provided an apparatus for decoding data, comprising a neural network according to the fourteenth aspect or any implementation thereof.

[0132] According to a twenty-fourth aspect, there is provided an apparatus for encoding at least a part of an image, comprising a neural network according to the fourteenth aspect or any implementation thereof.

[0133] According to a twenty-fifth aspect, there is provided an apparatus for decoding at least a part of an image, comprising a neural network according to the fourteenth aspect or any implementation thereof.

[0134] According to a twenty-sixth aspect, there is provided an apparatus according to any of the twenty-second to twenty-fifth aspects, comprising a hyperprior of a variational autoencoder, the hyperprior comprising a neural network according to the fourteenth aspect or any implementation thereof.

[0135] According to a twenty-seventh aspect, there is provided an apparatus for encoding at least a portion of an image, the apparatus comprising: Convert the tensors representing the image components into latent tensors; Providing an entropy model using a neural network according to the fourteenth aspect or any implementation thereof; and Process the latent tensors based on the provided entropy model to generate a bitstream; The processing circuitry is configured to:

[0136] According to a twenty-eighth aspect, there is provided an apparatus for decoding at least a portion of an encoded image, the apparatus comprising: Providing an entropy model using a neural network according to the fourteenth aspect or any implementation thereof; Process the bitstream based on the provided entropy model to obtain latent tensors that represent the components of the image; and Process the latent tensors to obtain tensors that represent the components of the image; The processing circuitry is configured to:

[0137] The components of an image may be Y, U, or V components, or may be R, G, or B components.

[0138] According to a twenty-ninth aspect, there is provided a computer program product comprising program code stored on a non-transitory medium, the program being configured to, when executed on one or more processors, perform a method according to any of the above aspects relating to the method and any implementation thereof.

[0139] According to a 30th aspect, there is provided a computer program product comprising program code stored on a non-transitory medium, the program being configured to, when executed on one or more processors, perform a method according to any of the above aspects relating to the method and any implementation thereof.

[0140] According to a thirty-first aspect, there is provided a computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to encode video data, the instructions causing the one or more processors to perform a method according to any of the above aspects relating to the method and any implementation thereof.

[0141] Any of the above aspects relating to methods and any implementation thereof may be implemented in an apparatus, whereby an apparatus configured to perform the steps of such a method is provided (with the same advantages as above).

[0142] According to a thirty-second aspect, there is provided an apparatus for encoding data, the apparatus including processing circuitry configured to perform any method steps, not limited to decoding, according to any of the above aspects relating to the methods and any implementations thereof.

[0143] According to a thirty-third aspect, there is provided an apparatus for encoding at least a portion of an image, the apparatus including processing circuitry configured to convert tensors representing components of the image into latent tensors, provide an entropy model including performing any method steps according to any aspect above relating to the method and any implementation thereof, not limited to decoding, and process the latent tensors using a neural network based on the provided entropy model to generate a bitstream.

[0144] According to a thirty-fourth aspect, there is provided an apparatus for decoding data, the apparatus including a processing circuit configured to perform any method steps according to any of the above aspects relating to the methods and any implementations thereof, not limited to encoding.

[0145] According to a thirty-fifth aspect, there is provided an apparatus for decoding at least a portion of an encoded image, the apparatus including: providing an entropy model including performing any method steps according to any aspect above relating to the method and any implementation thereof, not limited to encoding; processing the bitstream using a neural network based on the provided entropy model to obtain latent tensors representing components of the image; and processing the latent tensor to obtain tensors representing components of the image.

[0146] It should be noted that the functions of the above-described devices may be realized by hardware, or may be realized by hardware executing corresponding software.

[0147] According to a further aspect, the present disclosure relates to a video stream encoding device including a processor and a memory storing instructions for causing the processor to perform any method steps, including but not limited to decoding, according to any of the above aspects related to the method and any implementation thereof.

[0148] According to a further aspect, the present disclosure relates to a video stream decoding device including a processor and a memory storing instructions for causing the processor to perform any method steps, not limited to encoding, according to any of the above aspects related to the method and any implementation thereof.

[0149] The above-described aspects and implementation methods and apparatus can be readily combined with one another where considered appropriate.

[0150] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. [Brief description of the drawings]

[0151] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings and figures. [Figure 1] Figure 1 is a schematic diagram showing the channels processed by the layers of a neural network. [Diagram 2] Figure 2 is a schematic diagram showing an autoencoder type neural network. [Figure 3A] FIG. 3A is a schematic diagram showing an example network architecture, where the encoder and decoder sides include a hyperprior model. [Figure 3B]Figure 3B is a schematic diagram showing a general network architecture, where the encoder side includes a hyperprior model. [Figure 3C] Figure 3C is a schematic diagram showing a general network architecture, where the decoder side includes a hyperprior model. [Figure 4] FIG. 4 is a schematic diagram showing an example network architecture, where the encoder and decoder sides include a hyperprior model. [Diagram 5] Figure 5 is a block diagram illustrating the structure of a cloud-based solution for machine-based tasks such as machine vision tasks. [Figure 6] Figure 6 is a block diagram illustrating an end-to-end video compression framework based on neural networks. [Figure 7] FIG. 7 shows the transformer. [Figure 8] Figure 8 shows the neural network layers. [Figure 9] FIG. 9 shows a neural network according to an embodiment. [Figure 10] FIG. 10 is a flow chart illustrating a method of operating a neural network including clipping input data. [Figure 11] FIG. 11 shows a neural network including units for clipping, scaling, and rounding, as well as de-scaling, according to an embodiment. [Figure 12] FIG. 12 is a flow chart illustrating a method of operating a neural network including adjusting weights of neural network layers. [Figure 13] FIG. 13 is a flow chart illustrating a method of operating a neural network including implementing an approximate activation function. [Figure 14] FIG. 14 illustrates a neural network including an approximate activation function according to an embodiment. [Figure 15]FIG. 15 illustrates an apparatus configured to perform the method steps illustrated in FIGS. [Figure 16] FIG. 16 is a block diagram illustrating an example of a video coding system configured to implement embodiments of the present disclosure. [Figure 17] FIG. 17 is a block diagram illustrating another example of a video coding system configured to implement embodiments of the present disclosure. [Figure 18] FIG. 18 is a block diagram showing an example of an encoding device or a decoding device. [Figure 19] 19 is a block diagram showing another example of an encoding device or a decoding device.Like reference numbers and names in different drawings may indicate like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0152] In the following description, reference is made to the accompanying drawings which form a part of this disclosure and which show by way of example certain aspects of the disclosed embodiments or in which the disclosed embodiments may be used. It is understood that the disclosed embodiments may be used in other ways and may include structural or logical changes not depicted in the drawings. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

[0153] For example, it is understood that disclosure related to a described method may also apply to a corresponding device or system configured to perform the method, and vice versa. For example, when one or more particular method steps are described, the corresponding device may include one or more units, e.g., functional units, for performing one or more method steps (e.g., one unit performing one or more steps, or multiple units performing one or more of the multiple steps, respectively), even if such one or more units are not explicitly described or shown in the drawings. On the other hand, for example, when a particular apparatus is described based on one or more units, e.g., functional units, the corresponding method may include one step of performing the function of one or more units (e.g., one step of performing the function of one or more units, or multiple steps of performing one or more of the units, respectively), even if such one or more steps are not explicitly described or shown in the drawings. Furthermore, it is understood that features of various exemplary embodiments and / or aspects described herein may be combined with each other, unless specifically indicated otherwise.

[0154] The following provides an overview of the framework within which embodiments of the present disclosure may be used and some of the terminology used.

[0155] Artificial Neural Networks Artificial neural networks (ANNs), or connectionist systems, are computing systems loosely inspired by the biological neural networks that make up animal brains. Such systems generally "learn" to perform a task by considering examples, without being programmed with task-specific rules. For example, in image recognition, a system might learn to identify images that contain cats by analyzing example images that have been manually labeled as "cat" or "not cat," and then use the results to identify cats in other images. They do this without any prior knowledge of cats, such as that they have fur, tails, whiskers, or a cat-like face. Rather, they automatically generate identifying features from the examples they process.

[0156] ANNs are based on a collection of connected units or nodes called artificial neurons, which loosely model the neurons in a biological brain. Each connection can transmit a signal to other neurons, similar to a synapse in a biological brain. The artificial neurons that receive the signal can then process it and send a signal to the neurons that are connected to it.

[0157] In an ANN implementation, the "signals" at the connections are real numbers, and the output of each neuron is calculated by some nonlinear function of the sum of its inputs. The connections are called edges. Neurons and edges typically have weights that are adjusted as learning progresses. The weights increase or decrease the strength of the signal at the connection. Neurons have thresholds so that a signal is sent only if the aggregated signal exceeds the threshold. Neurons are typically aggregated in layers. Different layers may perform different transformations on their inputs. A signal progresses from the first layer (input layer) to the last layer (output layer), possibly after traversing the layers multiple times.

[0158] The original goal of the ANN approach was to solve problems in the same way as the human brain. Over time, interest has shifted to performing specific tasks, leading to deviations from biology. ANNs are used in a variety of tasks including computer vision, speech recognition, machine translation, social network filtering, playing board and video games, medical diagnosis, and even activities traditionally thought of as exclusive to humans, such as painting.

[0159] The name "convolutional neural network" (CNN) refers to the network's use of a mathematical operation called convolution, which is a special kind of linear operation. A convolutional network is a neural network that uses convolution instead of the more common matrix multiplication in at least one of its layers.

[0160] Figure 1 shows a schematic of the general concept of processing by a neural network such as a CNN. A convolutional neural network consists of input and output layers, as well as multiple hidden layers. The input layer is the layer to which the input (such as a portion of the input image 11 as shown in Figure 1) is provided for processing. The hidden layers of a CNN typically consist of a series of convolutional layers that perform convolutions using multiplication or other dot products. The output of a layer is one or more feature maps (shown as solid, solid rectangles), sometimes called channels. There may be resampling (like subsampling) in the operations of some or all of the layers. As a result, the feature maps may be smaller, as shown in Figure 1. Note that convolutions using strides can also reduce (resample) the size of the input feature maps. The activation function in a CNN is usually a ReLU (rectified linear unit) layer, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers, which are referred to as hidden layers because their inputs and outputs are masked by the activation function and the final convolution. The layer is colloquially called convolutional, but this is only by convention. Mathematically, it is strictly a sliding dot product or cross-correlation. This is important to the index of the matrix in that it affects how the weights are determined at a particular index point.

[0161] When programming a CNN to process images, the input is a tensor with dimensions (number of images) x (image width) x (image height) x (image depth), as shown in Figure 1. It should be noted that the image depth can be constructed by the channels of the image. After passing through the convolutional layers, the image is abstracted into a feature map with dimensions (number of images) x (feature map width) x (feature map height) x (feature map channels). The convolutional layers in a neural network should have the following attributes: width and height (hyper parameters), convolution kernels defined by the number of input and output channels (hyper parameters). The depth of the convolutional filter (input channels) should be equal to the number of channels of the input feature map (depth).

[0162] Traditionally, traditional multi-layer perceptron (MLP) models have been used for image recognition. However, due to the full connectivity between nodes, they suffer from high dimensionality and do not scale well with higher resolution images. A 1000x1000 pixel image with RGB color channels has 3 million weights, which is excessively large and cannot be efficiently and feasibly processed at a scale with full connectivity. Also, such network architectures do not consider the spatial structure of the data and treat input pixels that are far away from each other the same as pixels that are close to each other. This ignores the locality of reference in the image data, both computationally and semantically. Thus, the full connectivity of neurons is wasteful for purposes such as image recognition, which is dominated by spatially local input patterns.

[0163] Convolutional neural networks are biologically inspired variants of multi-layer perceptrons, specifically designed to emulate the behavior of the visual cortex. These models mitigate the problems posed by MLP architectures by exploiting the strong spatially local correlations present in natural images. Convolutional layers are the core building blocks of CNNs. The parameters of a layer consist of a set of learnable filters (kernels, as mentioned above), which have small receptive fields but are spread across the depth of the input volume. During a forward pass, each filter is convolved across the width and height of the input volume, computing the dot product between the filter's entries and the input, generating a 2D activation map for that filter. As a result, the network learns a filter to activate if it detects some particular type of feature at some spatial location in the input.

[0164] Stacking the activation maps of all filters along the depth dimension forms the complete output volume of the convolutional layer. Therefore, every entry of the output volume can also be interpreted as the output of a neuron that looks at a small region of the input and shares parameters with neurons in the same activation map. A feature map or activation map is the output activation of a given filter. Feature map and activation have the same meaning. In some papers it is called an activation map because it is a mapping corresponding to the activation of different parts of the image, and also a feature map because it is also a mapping of where in the image certain features are found. High activation means that a particular feature is found.

[0165] Another important concept in CNNs is pooling, which is a form of nonlinear down-sampling. There are several nonlinear functions to implement pooling, among which max pooling is the most common: it divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each such subregion.

[0166] Intuitively, the exact location of a feature is less important than its rough location relative to other features. This is the idea behind using pooling in convolutional neural networks. Pooling layers serve to progressively reduce the representation space size, reducing the number of parameters in the network, the memory footprint, and the amount of computation, and thus curbing overfitting. In CNN architectures, it is common to periodically insert pooling layers between successive convolutional layers. The pooling operation provides another form of translation invariance.

[0167] A pooling layer operates independently on every depth slice of the input and resizes it spatially. The most common form is a pooling layer where a filter of size 2 × 2 is applied along both the width and height, two by two, on every depth slice of the input with a stride of 2, discarding 75% of the activations. In this case, all max operations are over four orders of magnitude. The depth dimension remains unchanged. In addition to max pooling, the pooling unit can use other functions such as average pooling or l2-norm pooling. Average pooling was often used historically, but has recently fallen out of popularity compared to max pooling, which often works better in practice. Due to the large reduction in the size of the representation, there is a recent trend to use smaller filters or even to discard the pooling layer altogether. "Region of interest" pooling (also known as ROI pooling) is a variant of max pooling where the output size is fixed and the input rectangle is a parameter. Pooling is a key component of convolutional neural networks for object detection based on the Faster R-CNN architecture.

[0168] ReLU, as mentioned above, stands for rectified linear unit, and applies a non-saturating activation function. It effectively removes negative values ​​from the activation map by setting them to zero. This increases the non-linearity of the decision function and the entire network without affecting the receptive fields of the convolutional layers. Other functions are also used to increase the non-linearity, for example the saturated hyperbolic tangent or sigmoid function. ReLU is often preferred over other functions because it trains neural networks many times faster without a significant penalty to generalization accuracy.

[0169] After several convolutional and max pooling layers, higher level inference in neural networks is done via fully connected layers. Neurons in fully connected layers have connections to all activations of the previous layer, as in regular (non-convolutional) artificial neural networks. Their activations can then be computed as affine transformations with matrix multiplications followed by bias offsets (vector addition of learned or fixed bias terms).

[0170] The "loss layer" (containing the computation of the loss function) specifies how training penalizes deviations between predicted (output) and true labels, and is usually the final layer of a neural network. It is possible to use different loss functions suited to different tasks: Softmax loss is used to predict one class out of K mutually exclusive classes; Sigmoid cross entropy loss is used to predict K independent probability values ​​in [0,1]; Euclidean loss is used for regression against real-valued labels.

[0171] In summary, Figure 1 shows the data flow in a typical convolutional neural network. First, an input image passes through a convolutional layer and is abstracted into a feature map containing several channels, which correspond to a number of filters in the set of learnable filters of this layer. The feature map is then subsampled, for example using a pooling layer that reduces the dimensionality of each channel in the feature map. The data then arrives at another convolutional layer, which may have a different number of output channels. As mentioned before, the number of input and output channels are hyper-parameters of the layer. To establish the network connections, these parameters need to be consistent between the two connected layers, so the number of input channels of the current layer should be equal to the number of output channels of the previous layer. For the first layer, which processes input data, e.g., an image, the number of input channels is usually equal to the number of channels of the data representation, e.g., 3 channels for an RGB or YUV representation of an image or video, or 1 channel for a grayscale image or video representation. The channels obtained by one or more convolutional layers (and possibly a resampling layer) can be passed to an output layer. Such an output layer may be a convolution or a resampling in some implementations. In an exemplary non-limiting embodiment, the output layer is a fully connected layer.

[0172] Autoencoders and Unsupervised Learning An autoencoder is a type of artificial neural network used to learn efficient data coding in an unsupervised way. Its schematic diagram is shown in Figure 2. An autoencoder has an encoder side 210 where an input x is fed into the input layer of an encoder sub-network 220, and a decoder side 250 where an output x' is fed from a decoder sub-network 260. The goal of an autoencoder is to learn a representation (encoding) 230 for a set of data x, typically for dimensionality reduction, by training the networks 220, 260 to ignore signal "noise". Together with the reduction (encoder) side sub-network 220, a reconstruction (decoder) side sub-network 260 is trained, where the autoencoder, as the name suggests, tries to generate a representation x' from the reduced encoding 230 that is as close as possible to its original input x. In the simplest case, given one hidden layer, the encoder stage of the autoencoder takes the input and maps it to h.

[0173]

number

[0174] This image h is usually called the code, latent variable, or latent representation. Here, σ is an element-wise activation function, such as the sigmoid function or the rectified linear unit. W is a weight matrix and b is a bias vector. The weights and biases are usually initialized randomly and then updated iteratively during training through backpropagation. The decoder stage of the autoencoder then maps h to a reconstruction x' that has the same shape as x.

[0175]

number

[0176] Variational autoencoder models impose strong assumptions on the distribution of the latent variables. They use a variational approach to latent representation learning, which results in an additional loss component and a specific estimator for the training algorithm called the Stochastic Gradient Variational Bayesian (SGVB) estimator. The data is represented by a directed graph model p θ (x|h), and the encoder is responsible for the posterior distribution p θ Approximation q for (h|x) φ We assume that we learn (h|x), where φ and θ denote the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vectors in a VAE typically matches the probability distribution of the training data much more closely than a standard autoencoder. The objective of a VAE has the following form:

[0177]

number

[0178] Here, D KL stands for Kullback-Leibler divergence. The prior distribution for the latent variables is usually a centrally isotropic multivariate Gaussian function p θ We set (h) = N(0,I). In general, the shapes of the variation and likelihood distributions are chosen such that they are factored Gaussian functions.

[0179]

number

[0180] With recent advances in the area of ​​artificial neural networks, especially convolutional neural networks, researchers are turning their attention to applying neural network-based techniques to the task of image and video compression. For example, end-to-end optimized image compression has been proposed, which uses networks based on variational autoencoders.

[0181] Data compression is therefore assumed as a fundamental and well-studied problem in engineering, and is generally formulated with the goal of designing a code for a given discrete data ensemble with minimum entropy. The solution relies heavily on knowledge of the probabilistic structure of the data, and therefore the problem is closely related to probabilistic source modeling. However, since all real codes must have finite entropy, continuous-valued data (such as vectors of image pixel intensities) must be quantized to a finite set of discrete values, which introduces errors.

[0182] In this context, known as a lossy compression problem, we must trade off two competing costs: the entropy of the discretized representation (rate), and the error resulting from quantization (distortion). Different compression applications, such as data storage or transmission over limited-capacity channels, call for different rate-distortion tradeoffs.

[0183] Joint optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in a high-dimensional space cannot be addressed. For this reason, most existing image compression methods work by linearly transforming the data vector into an appropriate continuous-valued representation, quantizing its elements independently, and then encoding the resulting discrete representation with a lossless entropy code. This scheme is called transform coding, due to the central role of the transform.

[0184] For example, JPEG uses the discrete cosine transform on blocks of pixels, and JPEG 2000 uses a multi-scale orthogonal wavelet decomposition. Typically, the three components of a transform coding method -- transform, quantization, and entropy code -- are optimized separately (often by manual parameter adjustment). Recent video compression standards such as HEVC, VVC, and EVC also use the transformed representation to code the residual signal after prediction. For this purpose, several transforms are used, such as the discrete cosine and sine transforms (DCT, DST) and the low-frequency non-separable manually optimized transform (LFNST).

[0185] Variational Image Compression The variational autoencoder (VAE) framework can be considered as a nonlinear transformation coding model. The transformation process can be divided into four main parts. This is illustrated in Figure 3A, which shows the VAE framework.

[0186] The transformation process can be divided into four main parts: Figure 3A illustrates the VAE framework. In Figure 3A, the encoder 101 maps the input image x to a latent representation (denoted as y) by a function y=f(x). This latent representation may be referred to as a portion or point in the "latent space" in the following. The function f() is a transformation function that transforms the input signal x into a more compressible representation y. The quantizer 102 converts the latent representation y into a quantizer y. ^ = Q(y) = quantized latent representation y ^ where Q represents the quantization function. The entropy model, or hyper-encoder / decoder (also known as hyper-prior) 103 converts the quantized latent representation y ^ We estimate the distribution of ,to obtain the minimum achievable rate with lossless entropy source coding.

[0187] A latent space can be understood as a compressed data representation where similar data points are close to each other in the latent space. Latent spaces are useful for learning features of the data and finding simpler representations of the data for analysis. The quantized latent representation, T,y, ^ and the side information z of Hyperprior 3 ^ is included in bitstream 2 (which is binarized) using arithmetic coding (AE). Furthermore, the quantized latent representation is ^ ,x ^ =g(y ^ ) is converted into a signal x ^ is an estimate of the input image x. x should be as close to x as possible. ^ It is desirable to have a reconstruction quality as close as possible to x ^ The higher the similarity between x and x, the higher the amount of side information required for transmission. The side information includes bitstream 1 and bitstream 2 shown in FIG. 3A, which are generated by the encoder and transmitted to the decoder. Typically, the higher the amount of side information, the higher the reconstruction quality. However, a high amount of side information means a low compression ratio. Therefore, one objective of the system described in FIG. 3A is to balance the reconstruction quality and the amount of side information carried in the bitstream.

[0188] In FIG. 3A, the component AE 105 stores samples of the quantized latent representation ŷ and side information z ^ y is an arithmetic coding module that converts y into a binary representation bit stream 1. ^ and z ^ The samples may, for example, comprise integers or floating-point numbers. One purpose of the arithmetic coding module is to convert the sample values ​​(via a binarization process) into a string of binary numbers (which are then included in a bitstream that may contain other parts corresponding to the coded image or other side information).

[0189] Arithmetic Decoding (AD) 106 is the process of reversing the binary processing, where binary numbers are converted to sample values. Arithmetic decoding is provided by an arithmetic decoding module 106.

[0190] It should be noted that the present disclosure is not limited to this particular framework. Furthermore, the present disclosure is not limited to image or video compression, but can be applied to object detection, image generation, and recognition systems as well.

[0191] In FIG. 3A, there are two sub-networks connected to each other. A sub-network in this context is a logical division between parts of the overall network. For example, in FIG. 3A, modules 101, 102, 104, 105, and 106 are called the "encoder / decoder" sub-network. The "encoder / decoder" sub-network is responsible for encoding (generating) and decoding (analyzing) a first bitstream, "Bitstream 1". The second network in FIG. 3A includes modules 103, 108, 109, 110, and 107 and is called the "hyper-encoder / decoder" sub-network. The second sub-network is responsible for generating a second bitstream, "Bitstream 2". The purpose of the two sub-networks is different.

[0192] The first sub-network has the following responsibilities: Transformation 101 of an input image x into a latent representation y (this is easier than compressing x), The latent representation y is denoted by the quantized latent representation y ^ quantize to, Quantized latent representation y ^ Compressing the bit stream "Bitstream 1" using AE by the arithmetic coding module 105; Parsing the bitstream 1 by AD using an arithmetic decoding module 106; and Using the analyzed data, 104 images (x ^ ) to be reconstructed.

[0193] The purpose of the second sub-network is to determine statistical properties of the samples of "Bitstream 1" (e.g., mean, variance and correlation between samples of Bitstream 1) so that the first sub-network compresses Bitstream 1 more efficiently. The second sub-network generates a second bitstream "Bitstream 2" that includes said information (e.g., mean, variance and correlation between samples of Bitstream 1).

[0194] The second network is the quantized latent representation y ^ to side information z, and converting the side information z into quantized side information z ^ and the quantized side information z ^ The second network includes a coding part that encodes (e.g., binarizes) 109 the input bitstream 2 into the decoded quantized side information z ^ '. ^ ' is z ^ Since the decoding operation of the arithmetic coding side is a lossless compression method, the decoded quantized side information z ^ ' is then decoded side information y ^ ' is converted to 107. y ^ ' is y ^ represents a statistical property of y (e.g., the sample mean of y^, or the variance of the sample values). ^ ' is then provided to the arithmetic encoder 105 and arithmetic decoder 106 as described above, ^ Control the probability model of

[0195] FIG. 3A shows an example of a variational auto encoder (VAE), the details of which may vary in different implementations. For example, in a particular implementation, there may be additional components to more efficiently obtain statistical properties of the samples of bitstream 1. In one such implementation, there may be a context modeler whose goal is to extract cross-correlation information of bitstream 1. The statistical information provided by the second sub-network may be used by the arithmetic encoder (AE) 105 and arithmetic decoder (AD) 106 components.

[0196] Figure 3A depicts the encoder and decoder in a single diagram, as will be apparent to those skilled in the art, the encoder and decoder may, and very often are, integrated into different devices.

[0197] FIG. 3B shows the encoder and FIG. 3C shows the decoder component of the VAE framework in isolation. According to some embodiments, as input, the encoder receives a picture. The input picture may include one or more channels, such as color channels or other types of channels, e.g., depth channels or motion information channels. The outputs of the encoder (as shown in FIG. 3B) are Bitstream 1 and Bitstream 2. Bitstream 1 is the output of the first sub-network of the encoder and Bitstream 2 is the output of the second sub-network of the encoder.

[0198] Similarly, in Fig. 3C, two bitstreams, Bitstream 1 and Bitstream 2, are received as inputs, and a reconstructed (decoded) image, ̂z, is produced at the output. As mentioned above, the VAE can be divided into different logical units performing different operations. This is illustrated in Fig. 3B and 3C, such that Fig. 3B depicts components involved in encoding a signal, such as a video, and provides encoded information. This encoded information is then received, for example, by a decoder component depicted in Fig. 3C for decoding. It is noted that the encoder and decoder components denoted by numerals 12x and 14x may correspond in their function to the components described above in Fig. 3A and denoted by numeral 10x.

[0199] In particular, as can be seen in Figure 3B, the encoder comprises an encoder 121 which converts an input x into a signal y which is then fed to a quantizer 322. The quantizer 122 provides information to an arithmetic coding module 125 and to a hyper-encoder 123. The hyper-encoder 123 provides the bitstream 2 already mentioned above to a hyper-decoder 147 which in turn provides information (125) to the arithmetic coding module 105.

[0200] The output of the arithmetic coding module is bitstream 1. Bitstream 1 and bitstream 2 are the output of the coding of the signal, which are then provided (transmitted) to the decoding process. Although unit 101 (121) is called an encoder, the complete sub-network shown in FIG. 3B can also be called an "encoder". An encoder generally means a unit (module) that converts an input into a coded (e.g. compressed) output. As can be seen from FIG. 3B, unit 121 can actually be considered as the core of the entire sub-network, since it performs the conversion of the input x into a compressed version of x, y. The compression in encoder 121 can be achieved, for example, by applying a neural network, or in general any processing network with one or more layers. In such a network, the compression can be performed by cascading processes, including down-sampling, which reduces the size of the input and / or the number of channels. Thus, the encoder may be called, for example, a neural network (NN)-based encoder, etc.

[0201] The remaining parts in the figure (quantization unit, hyper-encoder, hyper-decoder, arithmetic encoder / decoder) are all parts that either improve the efficiency of the encoding process or convert the compressed output y into a series of bits (bitstream). Quantization may be performed to further compress the output of the NN encoder 121 by lossy compression. The AE 125 in combination with the hyper-encoder 123 and the hyper-decoder 127 used to construct the AE 125 may perform binarization that can further compress the quantized signal by lossless compression. Therefore, the entire sub-network of FIG. 3B can be called an "encoder".

[0202] Most deep learning (DL) based image / video compression systems reduce the dimensionality of a signal before converting it to binary numbers (bits). For example, in the VAE framework, an encoder, which is a non-linear transform, maps an input image x to y, where y has a smaller width and height than x. Since y has a smaller width and height and thus a smaller size, the dimensionality of the signal is reduced, and thus it becomes easier to compress signal y. Note that in general, an encoder does not necessarily need to reduce the size in both (or generally all) dimensions. Rather, some example implementations may provide an encoder that reduces the size in only one dimension (or generally a subset thereof).

[0203] J. Balle, L. Valero Laparra, and EP Simoncelli (2015). “Density Modeling of Images Using a Generalized Normalization Transformation”, In: arXiv e-prints, Presented at the 4th Int. Conf. for Learning Representations, 2016 (referred to in the following as “Balle”) In the paper, we propose a framework for end-to-end optimization of image compression models based on nonlinear transforms. We optimize with respect to the mean squared error (MSE), but use a more flexible transform built from a cascade of linear convolutions and nonlinearities. Specifically, we use a generalized divisive normalization (GDN) joint nonlinearity inspired by models of neurons in the biological visual system, which proves effective for Gaussianizing image densities. This cascade transform is followed by a uniform scalar quantization (i.e., each element is rounded to the nearest integer), which effectively implements a parametric form of vector quantization in the original image space. The compressed image is reconstructed from these quantized values ​​using an approximate parametric nonlinear inverse transform.

[0204] Such an example of a VAE framework is shown in Figure 4, which utilizes six down-sampling layers marked 401 to 406. The network architecture includes a hyperprior model. a ,g s ) shows the image autoencoder architecture, and the right side (h a ,h s ) corresponds to an autoencoder that implements a hyperprior. The decomposed prior model is the model for the analysis and synthesis transformation g a and g s The same architecture is used for both. Q stands for quantization, and AE and AD stand for arithmetic encoder and decoder, respectively. The encoder converts the input image x into g a to generate a response y (latent representation) with spatially varying standard deviation. a contains multiple convolutional layers with subsampling and generalized decomposition normalization (GDN) as the activation function.

[0205] The response is h ato summarize the distribution of the standard deviation of z. z is then quantized, compressed, and transmitted as side information. The encoder then ^ Using σ ^ , and estimate the spatial distribution of standard deviations, which is used to obtain probability (or frequency) values ​​for arithmetic coding (AE), and use it to estimate the quantized image representation y ^ (or latent representation) is compressed and transmitted. The decoder first extracts z ^ Then, h s Using ^ Get h s was successful. ^ Then, y ^ g s to obtain a reconstructed image.

[0206] Layers that include down-sampling are indicated with a downward arrow in the layer description. The layer description “Conv N,k1,2↓” means that the layer is a convolutional layer, has N channels, and the convolution kernel is of size k1×k1. For example, k1 may be equal to 5 and k2 may be equal to 3. As mentioned before, 2↓ means that down-sampling by a factor of 2 is performed in this layer. Down-sampling by a factor of 2 results in one of the dimensions of the input signal being reduced by half at the output. In FIG. 4, 2↓ indicates that both the width and height of the input image are reduced by a factor of 2. Since there are 6 down-sampling layers, if the width and height of the input image 414 (denoted by x) are given by w and h, then the output signal z ^413 has a width and height equal to w / 64 and h / 64, respectively. The modules denoted AE and AD are the arithmetic encoder and arithmetic decoder, as described in connection with Figs. 3A to 3C. The arithmetic encoder and decoder are specific implementations of entropy coding. AE and AD can be replaced by other means of entropy coding. In information theory, entropy coding is a lossless data compression method used to convert the values ​​of symbols into a binary representation, which is a reversible process. Also, "Q" in the figure corresponds to the quantization operation mentioned above in connection with Fig. 4, and further described above in the "Quantization" section. Also, the quantization operation and the corresponding quantization unit as part of the component 413 or 415 need not necessarily be present and / or can be replaced by other units.

[0207] Also shown in FIG. 4 is a decoder including upsampling layers 407-412. Another layer 420 is provided between upsampling layers 411 and 410 in the input processing order, which is implemented as a convolutional layer but does not provide upsampling on the received input. A corresponding convolutional layer 430 is also shown for the decoder. Such a layer can be provided in the NN to perform operations on the input that change certain characteristics without changing the size of the input. However, it is not essential that such a layer is provided.

[0208] When viewed in the processing order of the bitstream 2 through the decoder, the upsampling layers are performed in reverse order, i.e. from upsampling layer 412 to upsampling layer 407. Each upsampling layer is shown here as providing an upsampling with an upsampling ratio of 2, denoted by ↑. Of course, it is not necessary that all upsampling layers have the same upsampling ratio, other upsampling ratios such as 3, 4, 8, etc. may be used. Layers 407 to 412 are realized as convolutional layers (conv). In particular, the upsampling layers can be intended to provide an operation on the input that is the inverse of that of the encoder, so that they apply a deconvolution operation to the received input, so that their size is increased by a factor corresponding to the upsampling ratio. However, the present disclosure is not generally limited to deconvolution, and upsampling may be performed in some other way, such as by bilinear interpolation between two adjacent samples, or by nearest neighbor sample copying, etc.

[0209] In the first sub-network, several convolutional layers (401 to 403) are followed by Generalized Divided Normalization (GDN) on the encoder side and Inverse GDN (IGDN) on the decoder side. In the second sub-network, the activation function applied is ReLU. It should be noted that the present disclosure is not limited to such implementation and in general, other activation functions may be used instead of GDN or ReLU.

[0210] Machine Task Cloud Solution Machine-oriented video coding (VCM) is another computer science direction that is popular today. The main idea behind this approach is to transmit a coded representation of image or video information intended for further processing by computer vision (CV) algorithms like object segmentation, detection and recognition. In contrast to traditional image and video coding aimed at human perception, the quality characteristic is not the reconstructed quality but the performance of a computer vision task, for example object detection accuracy. This is illustrated in Figure 5.

[0211] Machine directed video coding, also called collaborative intelligence, is a relatively new paradigm for efficient deployment of deep neural networks on mobile cloud infrastructures. By splitting the network between the mobile side 510 and the cloud side 590 (e.g., cloud servers), it is possible to distribute the computational load such that the overall energy and / or latency of the system is minimized. In general, collaborative intelligence is a paradigm in which the processing of a neural network is distributed between two or more different computational nodes; e.g., devices, but generally, some functionally defined nodes. Here, the term "node" does not refer to the neural network nodes mentioned above. Rather, a (computational) node here refers to a separate device / module (physically, or at least logically) that realizes a part of a neural network. Such devices may be different servers, different end user devices, a mix of servers and / or user devices and / or clouds and / or processors, etc. In other words, computational nodes can be considered as nodes that belong to the same neural network and communicate with each other carrying data coded in / for the neural network. For example, one or more layers may run on a first device (such as a device on the mobile side 510) and one or more layers may run on another device (such as a cloud server on the cloud side 590) to be able to perform complex computations. However, the distribution may be finer and a single layer may run on multiple devices. In this disclosure, the term "multiple" refers to two or more. In existing solutions, part of the neural network functionality is run on a device (such as a user device or an edge device) or on multiple such devices, and then the output (feature map) is passed to the cloud. The cloud is a collection of processing or computational systems located outside the device that runs parts of the neural network. The concept of collaborative intelligence is also extended to model training.In this case, data flows bidirectionally: from cloud to mobile during backpropagation during training, and from mobile to cloud during forward propagation during training as well as inference (as shown in Figure 5).

[0212] Several works have presented semantic image compression by encoding deep features and then reconstructing the input image from them. Compression based on uniform quantization has been presented, followed by Context-Based Adaptive Arithmetic Coding (CABAC) with H.264. In some scenarios, it may be more efficient to transmit the output of the hidden layer (deep feature map) 550 from the mobile part 510 to the cloud 590 than to transmit compressed natural image data to the cloud and perform object detection using the reconstructed image. Therefore, it may be advantageous to compress the data (features) generated by the mobile side 510, which may include a quantization layer 520 for this purpose. Correspondingly, the cloud side 590 may include an inverse quantization layer 560. Efficient compression of feature maps is beneficial for image and video compression and reconstruction, both for human perception and machine vision. Entropy coding methods, e.g. arithmetic coding, are a common approach to compression of deep features (i.e., feature maps).

[0213] Currently, video content contributes to over 80% of Internet traffic, and the percentage is expected to further increase. Therefore, it is important to build an efficient video compression system to generate higher quality frames for a given bandwidth budget. Furthermore, most video-related computer vision tasks, such as video object detection or video object tracking, are sensitive to the quality of the compressed video, and efficient video compression may benefit other computer vision tasks. Meanwhile, video compression techniques also help in action recognition and model compression. However, in the past decades, video compression algorithms have relied on handcrafted modules, such as block-based motion estimation and discrete cosine transform (DCT), to reduce redundancy in video sequences, as mentioned above. Although each module is well designed, the entire compression system is not optimized end-to-end. It is desirable to further improve the video compression performance by optimizing the entire compression system together.

[0214] End-to-end image or video compression DNN-based image compression methods can take advantage of large-scale end-to-end training and highly nonlinear transformations that are not used in traditional approaches. However, it is not trivial to directly apply these techniques to build end-to-end learning systems for video compression. First, learning how to generate and compress motion information tailored for video compression remains an open problem. Video compression methods rely heavily on motion information to reduce temporal redundancy in video sequences.

[0215] A straightforward solution is to represent the motion information using learning-based optical flow. However, current learning-based optical flow approaches aim to generate flow fields that are as accurate as possible. Accurate optical flow is often suboptimal for a given video task. Furthermore, the amount of optical flow data increases significantly when compared to the motion information in traditional compression systems, and directly applying existing compression approaches to compress the optical flow values ​​would significantly increase the number of bits required to store the motion information. Second, it is unclear how to build a DNN-based video compression system by minimizing a rate-distortion-based objective over both residual and motion information. Rate-distortion optimization (RDO) aims to achieve higher quality (i.e., less distortion) of the reconstructed frames given the number of bits (or bit rate) for compression. RDO is critical for video compression performance. To exploit the power of end-to-end training for learning-based compression systems, an RDO strategy that optimizes the entire system is required.

[0216] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, Zhiyong Gao; “DVC: ​​An End-to-end Deep Video Compression Framework”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11006-11015 In this paper, the authors propose an end-to-end deep video compression (DVC) model that jointly learns motion estimation, motion compression, and residual coding.

[0217] Such an encoder is shown in Figure 6. In particular, Figure 6 shows the overall structure of an end-to-end trainable video compression framework. To compress the motion information, the CNN uses the optical flow v t to the corresponding representation mt, which is suitable for better compression.

[0218] Transformer In recent years, Transformers have attracted increasing attention in both the fields of language processing (e.g., text translation) and image processing. Video coding can be facilitated by the use of Transformer-based neural networks. Transformers can be used for image enhancement and classification purposes. Transformers do not include recurrent or convolutional neural networks and rely on self-attention. Transformers may be implemented, for example, in the configuration shown in Figure 6. In particular, Transformers can be combined with recurrent or convolutional neural networks.

[0219] FIG. 7 shows an example of a transformer 700. The transformer 700 includes a neural layer 810 (transformer layer). The transformer 700 can include an encoder-decoder architecture including an encoder-neural layer and a decoder-neural layer. Alternatively, the transformer 700 can include only an encoder stack of neural layers. Input data is input to the transformer and output data is output. For example, the transformer 700 may be configured for image processing and output an enhanced image. The input data may be, for example, patches of an image or words of a sentence. For example, a tokenizer generates tokens in the form of words from the sentence to be processed or patches from the image. These tokens can be converted into embeddings (continuous values) by some embedding algorithm. According to the example shown in FIG. 7, a linear projection layer 720 converts the input patches into tensor representations (embeddings) of the parts of the object to be processed (in the latent space). These tensor representations of the signal input are processed by the transformer 700. A position coding layer 730 is provided to provide information about the relative positions of parts of the processed object (e.g. an image or a text), e.g. the relative positions of patches of an image or words of a sentence with respect to each other. A sinusoidal function may be used for the position coding.

[0220] The last one of the neural layers 710 outputs an output data tensor in latent space that is transformed back to object space (e.g., image or text space) by a linear backprojection layer 740.

[0221] The processing by the neural layers (transformer layers) 710 is based on the concept of self-attention. Each neural layer 710 of the transformer 700 includes a multi-head self-attention layer and a (fully connected) feed-forward neural network. The self-attention layer helps the encoder stack to look at other parts of the object (e.g., image patches or words) when it encodes a particular part of the object (e.g., an image patch or a word of a sentence). The output of the self-attention layer is fed into a feed-forward neural network. The decoder stack also has both these components and, between them, an additional "encoder-decoder" attention layer that helps the decoder stack to focus on the relevant part of the input data. Each part of the object (e.g., an image patch or a word of a sentence) that is processed at each position flows through its own path in the encoder. There are dependencies between these paths in the self-attention layer. However, the feed-forward layer does not have these dependencies and therefore the various paths can run in parallel while flowing through the feed-forward layer.

[0222] In the multi-head self-attention layer of the encoder stack, we store the query Q, key K, value V tensors, and the self-attention

[0223]

number

[0224] Each sub-layer (self-attention layer and feed-forward neural network) in each encoder and decoder layer has residual connections in its neighborhood, followed by a normalization layer.

[0225] The output of the top encoder layer is transformed into a set of attention vectors K and V. These are used by each decoder layer in its “encoder-decoder-attention” layer, which works similarly to the multi-head self-attention layer of the encoder stack, except that it creates a query matrix from each layer below and obtains the key and value matrices from the output of the encoder stack.

[0226] The decoder stack outputs a vector of floating point numbers which are converted to parts of objects (e.g. patches in an image or words in a sentence) by a final linear layer (a fully connected neural network) which outputs logits, followed by a Softmax layer which yields the highest probability output.

[0227] As mentioned above, the use of neural networks (e.g. CNNs) is becoming increasingly important in the field of image coding systems, i.e. in data compression systems in general, e.g. in the context of neural network-based compression of images, video, audio, 3D-PCC, etc.

[0228] In reality, neural networks in real applications are implemented on various platforms / devices that differ from each other in terms of numerical computing architecture. For example, one device may be or contain a CPU, while another device in data communication with the one that is or contains a CPU may be or contain a GPU. Different platforms (e.g., one that contains a CPU in one device and a GPU in another device) usually handle delicate integer and real situations differently, especially for floating point arithmetic but also for fixed point arithmetic. One class of delicate real situations is overflow of the registers / memories used. There is no standardized handling of overflow situations and it is hardly imaginable what procedure would be commonly followed in this regard.

[0229] Integer overflow occurs when an arithmetic operation attempts to create a number that is outside the range that can be represented by a given number of digits - either larger than the maximum representable value or smaller than the minimum representable value. This situation may be handled differently by different compilers, devices (CPU, GPU), etc. In order to achieve bit-exact results of integer operations on different platforms, integer overflow should be avoided. In this context, it is noted that obtaining bit-exact results on different platforms is not important for all of the applications being processed. However, for some applications it is. For example, as described above with reference to Figures 2 to 4, in the context of autoencoding, e.g. variational autoencoders, an entropy model for lossless data compression is used on the encoder side and on the decoder / receiver side to model the symbol probabilities of the arithmetic encoder and the decoder, respectively. The encoder side and the decoder / receiver side may use different kinds of platforms. For reliable reconstruction of the encoded image, it is important that the entropy model is applied in the same way on both sides. In fact, even a small deviation between the entropy model used at the decoder / receiver side and the one used at the encoder side can lead to a complete breakdown of the image decoding / reconstruction process due to incorrect symbol interpretation. For example, in the configuration shown in Figure 4, the entropy model σ ^ must be provided by the hyperprior in the arithmetic encoder AE on the encoder side and in the arithmetic decoder AD on the decoder side. Bit-exact application of the same entropy model is required for reliable image reconstruction. The problem of incorrect symbol interpretation due to the entropy model used on the decoder / receiver side deviating from the one used on the encoder side arises not only for image coding systems, but also for all compression systems that use arithmetic coding and neural networks in the entropy part.

[0230] With regard to real arithmetic, a distinction must be made between fixed-point and floating-point arithmetic. Fixed-point arithmetic is arithmetic that represents fractional numbers (non-integers) by storing a fixed number of digits in the fractional part. In general, it is integer arithmetic; to convert such fixed-point values ​​to real ones, division by a scaling factor corresponding to the number of bits used to store the fractional part is used. Floating-point arithmetic is arithmetic of the form: Mantissa*Radix 指数 Floating-point arithmetic is an arithmetic that uses a representation of real numbers in the form: Mantissa*Radix 指数 The only difference is that in fixed-point arithmetic, both the base and the exponent are fixed, whereas in floating-point arithmetic, only the base is fixed and the exponent is part of the representation of the number. Thus, in fixed-point arithmetic, the precision of the result (the exponent) does not depend on the precision of the arguments, which means, for example, that the result of an addition does not depend on the order of addition (if no overflow occurs, the operation satisfies the associative law just like normal integer addition). On the other hand, in floating-point arithmetic, the exponent is part of the representation of the number and is calculated during the arithmetic operation based on the exponents of the arguments. This causes non-associative addition; for example, if many very small numbers are added one by one to a large number, the result will be equal to the large number, because the effect of the relatively small numbers will be lost after rounding with the exponent of the large number. Conversely, if many small numbers are added together first, the effect of adding each number is not lost by rounding, and the resulting sum can be relatively large and is not completely lost after rounding with the exponent of the large number. Thus, fixed-point addition is associative unless there is overflow, while in principle floating-point arithmetic is non-associative, i.e. in general: (A+B)+C≠A+(B+C) Since the order of additions may be different on the encoder and decoder sides, or even not completely determined in advance, this creates problems with regard to identical results to be achieved on both sides. Therefore, fixed-point arithmetic may be preferred over floating-point arithmetic. In the context of entropy coding based data compression to ensure proper decoding among different platforms, especially in systems with massive parallelism, it may be preferable to avoid the use of floating-point arithmetic at least in the entropy part of the image coding net, and use fixed-point arithmetic instead. The potential problem of different floating-point implementations on different devices is solved in this case, since fixed-point (integer) arithmetic is much more portable. However, merely restricting to fixed-point arithmetic does not avoid some overflows. To guarantee bit-exact behavior on different platforms, it is not enough to use fixed-point arithmetic, it is important to ensure that there are no integer overflows.

[0231] More specifically, consider a neural network layer as shown in FIG. 8. The neural network layer may be included in, for example, a convolutional neural network or a fully connected neural network. i are inputted on various input channels, and weights w ij where index i is the in , and the index j indicates an input channel among C out jth channel output is obtained according to:

[0232]

number

[0233] The (usually trainable) bias D is ignored in what follows for simplicity. For example, for a convolutional neural network layer, the output of the jth channel is given by the convolution operator * as:

[0234]

number

[0235] The accumulator registers of a neural network used for summing buffering have a predefined size, i.e. some accumulator register bit depth (size) n (e.g. n = 16 bits or 32 bits). To avoid integer overflow, for example the following conditions need to be met:

[0236]

number

[0237] A) Clipping and Scaling According to an embodiment, with respect to the fixed-point arithmetic used, the input data x to be processed by a neural network layer of a neural network (e.g., a convolutional neural network) is i and the real-valued weights w of the neural network layers ij is an integer x i ~ and integer weight w ij ~, respectively. The integerized version of Equation 2 can be read as:

[0238]

number

[0239] To avoid the performance loss caused by this conversion to fixed-point arithmetic,

[0240]

number

[0241]

number

[0242]

number

[0243]

number

[0244] Thus, according to an embodiment, the following units are added to the neural network (inference pipeline): units configured for scaling, rounding and clipping the input data values ​​before input to the neural network layers, and a unit for dividing the output of the neural network layers by a scaling factor. The scaling and / or rounding units may be omitted (or may not be operational) if it is guaranteed that only integer-valued data (data containing only integers) should be processed. Units may be added to neural networks used for data compression / coding, in particular for (variational) autoencoder or transformer architectures as mentioned above.

[0245] An exemplary embodiment is shown in FIG. 9. FIG. 9 shows, for exemplary purposes, the neural network layers of a convolutional neural network. The neural network includes a stack of convolutional layers followed by an activation function, in this case the LeackyReLU function. The black arrows in FIG. 9 indicate the inputs of new units that are added to a conventional neural network. Before the convolutional layers, the input values ​​of the real-valued input data are multiplied by a scaling factor and the result of the multiplication process is rounded to the nearest integer. If the input data is already integer-valued, the scaling and / or rounding process can be omitted. If the resulting integer value is smaller than a predefined lower integer threshold-A, the integer value is clipped to a predefined lower integer threshold-A, and if the integer value is larger than a predefined upper integer threshold B, the integer value is clipped to an upper integer threshold B. The resulting input values ​​are then processed by the convolutional neural network layers and the outputs of these layers are input. According to an embodiment, the output of the activation function is divided by the scaling factor applied to the input data. According to another embodiment, the output of the convolutional neural network layer is divided by a scaling factor and then fed to the activation function. According to a further embodiment, a decomposition of each of the scaling factors into two parts is performed, where the division by the first part can be performed before the activation function and the division by the second part can be performed after the activation function.

[0246] Neural networks such as the one shown in Fig. 9, including units for scaling, rounding, clipping and for dividing the outputs of the neural network layers by scaling factors (before and / or after the respective activation functions), can be provided on both the encoder and decoder sides of a coding system, e.g., in the encoder for encoding images and in the decoder for decoding the encoded images, and can be used, for example, by the encoder / decoder included in the configurations shown in Figs.

[0247] B) Weight The condition in Equation 5 can be transformed into another condition on the weights of the neural network layers. Here, i and weight w ij Assume that x, y, and z have already been converted to integers (if necessary) by scaling and rounding as described above. Assume further that clipping to -A and -B, respectively, per Equation 6 has been performed if necessary. The terms of Equation 5 are transformed as follows (for each output channel, if different output channels are provided, the output channel index j is omitted below):

[0248]

number

[0249] If the condition of Equation 7 is satisfied, the inference including clipping of the input data to lower and upper thresholds respectively will not cause any overflow of the accumulator registers of the neural network. If the model weights in a data compression system are allowed to be defined by the user, the condition of Equation 7 must be satisfied if bit-exact behavior of the neural network at the encoder and decoder side needs to be guaranteed.

[0250] According to the embodiment, the integer lower threshold -A is -2 k-1 and the upper integer threshold B is given by 2 k-1 −1, where k denotes the predefined bit depth of the layer's input data.

[0251] In this case an easy to check condition on the weights can be given as follows:

[0252]

number

[0253] This form of condition on weights can be easily computed especially for user-defined weights to check whether it is guaranteed that overflow never occurs during inference. The conditions according to Equations 8 and 9 must be satisfied for each output j of a neural network layer.

[0254] The bias D could in principle be taken to be zero, in which case equations 8 and 9 would be more simply checked.

[0255] In particular, for one-dimensional convolutional neural network layers, addition

[0256]

number

[0257]

number

[0258] For a 2D convolutional neural network layer, addition

[0259]

number

[0260]

number

[0261] Here, C in denotes the number of input channels of the neural network layer, and K 1 and K 2 where j denotes the convolution kernel size and j denotes the index of the output channel of the neural network layer. Correspondingly, for an N-dimensional convolutional neural network layer, the similar summation can be calculated as:

[0262]

number

[0263] C) Scaling factors for the weights A larger scaling factor for the weights, s j (More precisely, 2 sj (see above) will result in a lower performance loss due to the conversion to fixed-point arithmetic than a smaller one (see above). The largest possible value that ensures that the condition of Equation 8 or 9 is satisfied is the scaling factor s j If we assume that

[0264]

number

[0265]

number

[0266] According to another implementation, the second scaling factor s for the j-th output channel of at least one neural network layer is j is given by:

[0267]

number

[0268] Both conditions can advantageously guarantee that integer overflow will not occur for the accumulator register.

[0269] According to another implementation, the condition is:

[0270]

number

[0271]

number

[0272]

number

[0273] In these conditions, n is the bit size of the accumulator register, and w ij denotes the real-valued weights, k denotes the predefined bit depth of the input data, and b j denotes the bias value (which may be zero).

[0274] Both conditions can advantageously guarantee that integer overflow will not occur for the accumulator register.

[0275] To convert the entire neural network pipeline to fixed-point arithmetic, the parameters p and k should be known. These parameters can be calculated, for example, by all possible combinations of (p,k), 0≦p≦k≦n, and the corresponding s obtained using equation (10) for a chosen k and a given n in some calibration data set. jand the best pair (p,k) can be selected by checking. The minimum of a predefined loss function corresponds to the best pair (p,k). The loss function can represent, for example, the number of bits required to code some image or its likelihood estimation. It is also possible to use distortion as part of the loss function used, in case not only the entropy part but also the analysis and synthesis parts (see, for example, the left side of the configuration shown in FIG. 4) should be converted to fixed-point arithmetic.

[0276] D) Activation function To ensure device interoperability across neural networks, bit-exact reproducibility of activation functions on different platforms / devices is desirable. For linear activation functions and relatively simple non-linear activation functions, e.g., the ReLU function, which basically defines a clipping procedure, this requirement can be met relatively easily. For more complex non-linear ones, especially those involving exponential functions (which may use the basis e, but other bases may also be included), e.g., Softmax:

[0277]

number

[0278] According to an embodiment, the mathematically defined non-linear activation function is replaced by an approximation function selected from the group consisting of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function. The mathematically defined non-linear activation function to be replaced may be a Softmax function, a Sigmoid function, a Hyperbolic Tangent function, a Swish function, a Gaussian Error Linear Unit function, or a Scaled Exponential Linear Unit function.

[0279] The approximation functions may be used, for example, by encoders / decoders in the configurations shown in Figures 2 to 4, 6 and 7, and generally in data compression / coding contexts.

[0280] In general, for any nonlinear activation function, it is possible to use the sum of the first few elements of its Taylor series. The Taylor series of a function f(x) mathematically defined around a predefined value a is defined as follows:

[0281]

number

[0282]

number

[0283]

number

[0284] It should be noted that, depending on the actual application, the approximation of mathematically defined non-linear activation functions by rational or polynomial functions may be considered particularly appropriate.

[0285] In particular, according to an embodiment, a Softmax function that operates on the i component of a vector x is used.

[0286]

number

[0287]

number

[0288]

number

[0289]

number

[0290] In particular, an embodiment is provided that relates to a method 1000 of operating a neural network capable of avoiding integer overflow. The neural network includes a neural network layer including or connected to an accumulator register for buffering summation results and having a predefined accumulator register size. The method 1000 includes a step S1010 of defining a lower integer threshold and an upper integer threshold for values ​​of integers included in data entities (e.g., numbers, vectors, or tensors) of input data of the neural network layer. The method also includes a step of clipping S1020 the integer values ​​included in the data entities of the input data to the predefined lower integer threshold if the values ​​of the integers included in the data entities of the input data are smaller than the predefined lower integer threshold, and clipping S1020 the integer values ​​included in the data entities of the input data to the predefined upper integer threshold if the values ​​of the integers included in the data entities of the input data are greater than the predefined upper integer threshold to avoid integer overflow of the accumulator register. When performing the same procedure with the same input data on two devices / platforms, the method 1000 as shown in FIG. 10 allows obtaining the same results for both devices / platforms because the clipping procedure can be used to avoid integer overflow.

[0291] The method shown in FIG. 10 can be implemented in any kind of neural network, for example, a neural network including one or more fully connected neural network layers or one or more convolutional neural network layers. In particular, a neural network 1100 is provided as shown in FIG. 11. The neural network 1100 can utilize the method 1000. The neural network 1100 includes a unit 1110 for scaling, rounding, and clipping. The unit 1110 may be divided into sub-units, each configured for different operations, such as scaling, rounding, and clipping of the input data, and on and off operation of the first unit and each sub-unit, respectively, can be switched. The unit 1110 may be configured to receive real-valued input data (data including real numbers) and convert the input data values ​​to integer-valued input data by multiplying the input data values ​​by a scaling factor and rounding the multiplication result to the next integer as described above. The resulting integer values ​​are clipped to integer lower and upper thresholds as necessary.

[0292] The integerized and clipped data is input to the neural network layer 1120. Downstream of the neural network layer 1120, a unit for de-scaling 1130 is provided. The unit for de-scaling 1130 can divide the output of the neural network layer 1120 by the scaling factor. The output of the unit for de-scaling 1130 can be input to the activation function 1140. Alternatively, the output of the neural network layer 1120 can be directly input to the activation function 1140, and the unit for de-scaling 1130 is provided downstream of the activation function 1140 and divides its output. Alternatively, the de-scaling by the unit for de-scaling 1130 is performed partially on the output of the neural network layer 1120 (by a first decomposition part of the scaling factor) and subsequently partially on the output of the activation function 1140 (by a second decomposition part of the scaling factor).

[0293] In particular, an embodiment is provided that relates to a method 1200 of operating a neural network based on adjusted weights as shown in FIG. 12. The neural network includes a neural network layer including or connected to an accumulator register for buffering summation results and having a predefined accumulator register size. The method 1200 includes S1210 defining a lower integer threshold A and an upper integer threshold B for integer values ​​included in data entities (e.g., numbers, vectors, or tensors) of input data of the neural network layer. The method also includes a step of clipping S1220 the integer values ​​included in the data entities of the input data to the predefined lower integer threshold if the integer values ​​included in the data entities of the input data are smaller than the predefined lower integer threshold, and clipping S1220 the integer values ​​included in the data entities of the input data to the predefined upper integer threshold if the integer values ​​included in the data entities of the input data are greater than the predefined upper integer threshold. Further, the method includes a step S1230 of determining integer-valued weights (i.e., weights that include integers, e.g., only integers) of the neural network layers based on a defined integer lower threshold, a defined integer upper threshold, and a predefined accumulator register size, such that integer overflow of the accumulator registers can be avoided.

[0294] The method 1200 as shown in Fig. 12 allows obtaining the same results for both devices / platforms when performing the same procedure with the same input data in the two devices / platforms. Particularly in the context of entropy model based coding and / or compression and decompression of data such as image data, it is an important issue to provide substantially bit-exact processing results on the encoding and decoding sides, respectively, to provide the same or complementary technical effects obtainable by the method 1200 as shown in Fig. 12.

[0295] In particular, according to an embodiment, a method 1300 of operating a neural network is provided (see FIG. 13). The method 1300 comprises a step 1310 of implementing an approximation function of a mathematically defined real-valued non-linear activation function as an activation function of at least one neural network layer, the approximation function allowing processing only integers of a fixed-point representation of the input values ​​of the approximation function. The approximation function may comprise at least one of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function. The mathematically defined non-linear activation function may be selected from the group consisting of a Softmax function, a sigmoid function, a hyperbolic tangent function, a swish function, a Gaussian error linear unit function, and a scaling exponential linear unit function. In particular, the approximation activation function may be any of the following:

[0296]

number

[0297] The provision of said approximation activation functions allows for bit-exact reproduction of important numerical operations at the encoder and decoder side, respectively, since in particular the calculation of exponential functions can be avoided, as a result of which the technical effects obtained by these numerical operations are identical or complementary to each other.

[0298] According to an embodiment, a neural network 1400 is provided as shown in Fig. 14. The neural network 1400 includes at least one neural network layer 1410 and an activation function 1420 connected to the output of the at least one neural network layer 1410, the activation function 1420 being implemented as an approximation function of a mathematically defined real-valued non-linear activation function (taking real numbers as arguments and outputting real numbers), the approximation function allowing processing only integers of a fixed-point representation of the input values ​​of the approximation function.

[0299] At least some of the steps of the methods 1000, 1200 and 1300 described with reference to figures 10, 12 and 13 may be included in a method for encoding or decoding at least part of data of an image, for example. According to an embodiment, the encoding or decoding is based on an entropy model providing statistical (probabilistic) properties of the symbols to be encoded or decoded, for example mean, variance, (cross)correlation, etc. The entropy model may be provided by a hyper-prior of a variational autoencoder, an auto-regressive prior of a variational autoencoder, or a combination of a hyper-prior and an auto-regressive prior of a variational autoencoder. The use of the methods 1000, 1200 and 1300 described with reference to figures 10, 12 and 13 for data coding, for example image coding, may prove advantageous with regard to a high-quality reconstruction of the encoded / compressed data without suffering serious corruption.

[0300] The methods 1000, 1200 and 1300 described with reference to Figures 10, 12 and 13 may be implemented in an apparatus 1500 as shown in Figure 15, which may be configured to perform steps of these methods. According to an embodiment, the apparatus 1500 includes a processing circuit configured to perform steps of the methods 1000, 1200 and 1300 described with reference to Figures 10, 12 and 13. Furthermore, the apparatus 1500 may include at least one of the neural networks shown in Figures 9, 11 and 14. The apparatus 1500 may be constituted by an encoder (e.g. the encoder 20 shown in Figures 16 and 17) or a decoder (e.g. the decoder 20 shown in Figures 16 and 17), or may be constituted by the video coding device 8000 shown in Figure 18 or the apparatus 9000 shown in Figure 19.

[0301] The apparatus 1500 may be, for example, an apparatus for encoding or decoding data, such as at least a portion of an image. Furthermore, the apparatus 1500 may include or be comprised within a (variational) autoencoder or a transformer, as described above.

[0302] Although operations are depicted in a particular order in the figures, this should not be understood as requiring that such operations be performed in the particular order or sequential order depicted, or that all of the depicted operations be performed, to achieve desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0303] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order depicted, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0304] Some Hardware and Software Implementations A corresponding system in which the above-mentioned encoder / decoder processing chain can be deployed is shown in FIG. 16. FIG. 16 is a schematic block diagram illustrating an example coding system, such as a video, image, audio, and / or other coding system (or coding system for short), that can utilize the techniques of the present application. The video encoder 20 (or encoder for short) and the video decoder 30 (or decoder for short) of the video coding system 10 represent examples of devices that can be configured to perform the techniques according to various examples described in the present application. For example, the video encoding and decoding can use a neural network that can be distributed and can apply the above-mentioned bitstream analysis and / or bitstream generation to communicate feature maps between distributed computing nodes (two or more).

[0305] As shown in FIG. 16, coding system 10 includes a source device 12 configured to provide encoded picture data 21 to a destination device 14, which may, for example, decode encoded picture data 13.

[0306] The source device 12 includes an encoder 20 and may additionally, i.e. optionally, include a picture source 16, a pre-processing processor (or pre-processing unit) 18, such as a picture pre-processing unit 18, and a communication interface or unit 22.

[0307] Picture source 16 may include or be any kind of picture capture device, such as a camera capturing real-world pictures, and / or any kind of picture generation device, such as a computer graphics processor generating computer-animated pictures, or any kind of other device acquiring and / or providing real-world pictures, computer-generated pictures (e.g. screen content or virtual reality (VR) pictures), and / or any combination thereof (e.g. augmented reality (AR) pictures). Picture source could be any kind of memory or storage storing any of the above mentioned pictures.

[0308] In distinction to the pre-processing processor 18 and the processing performed by the pre-processing unit 18 , the pictures or picture data 17 may also be referred to as unprocessed pictures or unprocessed picture data 17 .

[0309] The pre-processing processor 18 is configured to receive the (raw) picture data 17 and perform pre-processing on the picture data 17 to obtain a pre-processed picture 19 or pre-processed picture data 19. The pre-processing performed by the pre-processing processor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It can be understood that the pre-processing unit 18 may be an optional component. It is noted that the pre-processing may also use a neural network (such as the one in any of Figures 1 to 7) using presence indicator signaling.

[0310] A video encoder 20 is configured to receive the pre-processed picture data 19 and to provide encoded picture data 21 .

[0311] The communications interface 22 of the source device 12 can be configured to receive the encoded picture data 21 and to transmit the encoded picture data 21 (or any further processed version thereof) via the communications channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0312] The destination device 14 includes a decoder 30 (e.g., a video decoder 30) and may additionally, i.e. optionally, include a communications interface or unit 28, a post-processing processor 32 (or post-processing unit 32), and a display device 34.

[0313] The communications interface 28 of the destination device 14 is configured to receive the encoded picture data 21 (or any further processed version thereof), e.g. directly from the source device 12 or from any other source, e.g. from a source device, e.g. from a coded picture data storage device, and to provide the encoded picture data 21 to a decoder 30.

[0314] The communication interface 22 and the communication interface 28 may be configured to transmit or receive the encoded picture data 21 or the encoded data 13 via a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or via any kind of network, such as a wired or wireless network or any combination thereof, or any kind of private and public network, or any combination thereof.

[0315] The communications interface 22 may, for example, be configured to package the encoded picture data 21 into a suitable format, such as packets, and / or process the encoded picture data using any type of transmission coding or processing for transmission over a communications link or network.

[0316] The communications interface 28, forming a counterpart of the communications interface 22, can for example be configured to receive transmitted data and to process the transmitted data using any kind of corresponding transmission decoding or processing and / or unpackaging to obtain the encoded picture data 21.

[0317] Both the communication interface 22 and the communication interface 28 may be configured as a unidirectional communication interface, as indicated by the arrow in Fig. 16 for the communication channel 13 from the source device 12 to the destination device 14, or as a bidirectional communication interface, for example to send and receive messages, for example to set up connections, to verify and exchange any other information related to the communication link and / or data transmission, for example encoded picture data transmission. The decoder 30 is configured to receive the encoded picture data 21 and to output decoded picture data 31 or decoded pictures 31.

[0318] The post-processing processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), e.g., a decoded picture 31, to obtain post-processed picture data 33, e.g., a post-processed picture 33. The post-processing performed by the post-processing unit 32 may include, e.g., color format conversion (e.g., from YCbCr to RGB), color correction, cropping, resampling, or any other processing, e.g., to prepare the decoded picture data 31 for display by a display device 34.

[0319] A display device 34 of destination device 14 is configured to receive the post-processed picture data 33 and display the picture, for example to a user or viewer. The display device 34 may be or include any type of display that presents the reconstructed picture, for example an integrated or external display or monitor. The display may include, for example, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0320] 16 depicts source device 12 and destination device 14 as separate devices, an embodiment of the devices could also include both or both functionality, source device 12 or corresponding functionality, and destination device 14 or corresponding functionality. In such an embodiment, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0321] As will be clear to a person skilled in the art based on the specification, the presence and (exact) division of functions or functions of various units in the source device 12 and / or destination device 14 shown in FIG. 16 may vary depending on the actual device and application.

[0322] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30) or both the encoder 20 and the decoder 30 may be implemented by processing circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video coding, or any combination thereof. The encoder 20 may be implemented by processing circuitry 46 to implement various modules, including a neural network or portions thereof. The decoder 30 may be implemented by processing circuitry 46 to implement any coding system or subsystem described herein. The processing circuitry may be configured to perform various operations, as described below. If the techniques are implemented in part in software, the device may store the software instructions on a suitable non-transitory computer readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques of the present disclosure. Either video encoder 20 or video decoder 30 may be integrated as part of a combined encoder / decoder (CODEC) in a single device, for example as shown in FIG.

[0323] Source device 12 and destination device 14 may include any of a wide range of devices, including any type of handheld or stationary device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or a content delivery server), a broadcast receiving device, a broadcast transmitting device, and the like, and may or may not use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0324] In some cases, the video coding system 10 shown in FIG. 16 is merely an example, and the techniques of the present application may be applied to video coding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between the encoding device and the decoding device. In other examples, data may be retrieved from local memory, streamed over a network, etc. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other, but simply encode data to memory and / or retrieve data from memory and decode it.

[0325] 18 is a schematic diagram of a video coding device 8000 according to an embodiment of the present disclosure. The video coding device 8000 is suitable for implementing the disclosed embodiments as described herein. In an embodiment, the video coding device 8000 may be a decoder, such as the video decoder 30 of FIG. 16, or an encoder, such as the video encoder 20 of FIG. 16.

[0326] The video coding device 8000 includes an ingress port 8010 and a receiver unit (Rx) 8020 for receiving data, a processor, logic unit, or central processing unit 8030 (CPU) for processing data, a transmitter unit (Tx) 8040 and an egress port 8050 (or output port 8050) for transmitting data, and a memory 8060 for storing data. The video coding device 8000 may also include optical-electrical (OE) and electrical-optical (EO) components coupled to the ingress port 8010, the receiver unit 8020, the transmitter unit 8040, and the egress port 8050 for the entry and exit of optical or electrical signals.

[0327] The processor 8030 is implemented by hardware and software. The processor 8030 may be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 8030 communicates with the ingress port 8010, the receiver unit 8020, the transmitter unit 8040, the egress port 8050, and the memory 8060. The processor 8030 includes a neural network-based codec 8070. The neural network-based codec 8070 implements the disclosed embodiments described above. For example, the neural network-based codec 8070 implements, processes, prepares, or provides various coding operations. Thus, the inclusion of the neural network-based codec 8070 provides a significant improvement to the functionality of the video coding device 8000 and affects the transformation of the video coding device 8000 into different states. Alternatively, the neural network based codec 8070 may be implemented as instructions stored in the memory 8060 and executed by the processor 8030 .

[0328] The memory 8060 may include one or more disks, tape drives, and solid state drives, and may be used as overflow data storage devices, to store programs when such programs are selected for execution, and to store instructions and data loaded during the execution of the programs. The memory 8060 may be, for example, volatile and / or non-volatile, and may be read only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), and / or static random access memory (SRAM).

[0329] FIG. 19 is a simplified block diagram of an apparatus that can be used as either or both of the source device 120 and destination device 14 of FIG. 16, according to an example embodiment.

[0330] The processor 9002 in the device 9000 may be a central processing unit. Alternatively, the processor 9002 may be any other type of device or devices capable of manipulating or processing information, now existing or later developed. Although the disclosed implementations may be implemented using a single processor, such as the processor 9002, as shown, advantages in speed and efficiency may be obtained using more than one processor.

[0331] The memory 9004 in the device 9000 may be a read-only memory (ROM) device or a random access memory (RAM) device in implementation. Any other suitable type of storage device may be used as the memory 9004. The memory 9004 may include code and data 9006 that is accessed by the processor 9002 using the bus 9012. The memory 9004 may further include an operating system 9008 and application programs 9010, which include at least one program that enables the processor 9002 to perform the methods described herein. For example, the application programs 9010 may include applications 1 through N, which further include a video coding application that performs the methods described herein.

[0332] The apparatus 9000 may also include one or more output devices, such as a display 9018. The display 9018, in one example, may be a touch-sensitive display that combines a display with touch-sensing elements operable to sense touch input. The display 9018 may be coupled to the processor 9002 via the bus 9012.

[0333] Although depicted herein as a single bus, the bus 9012 of the device 9000 may be comprised of multiple buses. Additionally, the secondary storage may be directly coupled to other components of the device 9000 or may be accessed over a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, the device 9000 may be implemented in a wide variety of configurations.

Claims

1. 1. A method of operating a neural network including a neural network layer including or connected to an accumulator register for buffering summation results and having a predefined accumulator register size, comprising: defining a lower integer threshold A and an upper integer threshold B for integer values ​​included in data entries of input data for said neural network layer; clipping the integer value contained in the data entry of the input data to the predefined lower integer threshold if the integer value contained in the data entry of the input data is less than a predefined lower integer threshold, and clipping the integer value contained in the data entry of the input data to the predefined upper integer threshold if the integer value contained in the data entry of the input data is greater than a predefined upper integer threshold; and determining integer weights of the neural network layers based on the predefined integer lower threshold, the predefined integer upper threshold, and the predefined accumulator register size so as to avoid integer overflow of the accumulator register; The method includes:

2. The method of claim 1 , wherein the neural network layer is a fully connected layer.

3. 2. The method of claim 1, wherein the neural network layer is a convolutional neural network layer.

4. 4. The method of claim 1, wherein the accumulator register size is n bits, where n is a positive integer value.

5. 5. The method of claim 4, wherein the accumulator register size is equal to one of 32 bits and 16 bits.

6. 6. The method of claim 1, wherein the lower integer threshold is less than or equal to 0 and the upper integer threshold is greater than or equal to 0.

7. 7. The method of claim 6, wherein the integer lower threshold A is −2 k-1 and the upper integer threshold B is given by: k-1 −1, where k indicates a predefined bit depth of the layer input data.

8. 8. The method according to claim 1, wherein the neural network layer comprises: [0063] wherein D denotes an integer value, W denotes a subset of trainable layer weights, and X denotes one of the set and subset of input data for the neural network layer.

9. The method of claim 8, wherein the integer weights {w i } is the condition [0,64] The method is determined to satisfy the following:

10. The method of claim 8 when dependent on claim 7, wherein the integer weights {w i } is the condition [0.65] The method is determined to satisfy the following:

11. The method of claim 8 when dependent on claim 7, wherein the integer weights {w i } is the condition [0.66] The method is determined to satisfy the following:

12. 12. The method according to claim 8, wherein D is equal to 0.

13. The method according to claim 9 when dependent on claim 3, or claim 10 when dependent on claim 3, or claim 11 when dependent on claim 3, wherein the neural network layer is a two-dimensional convolutional neural network layer, [0.67] is expressed as follows: [0068] is calculated by C in denotes the number of input channels of the neural network layer, and K 1 and K 2 denote a convolution kernel size, and j denotes an index of an output channel of the neural network layer.

14. The method according to claim 9 when dependent on claim 3, or claim 10 when dependent on claim 3, or claim 11 when dependent on claim 3, wherein the neural network layer is an N-dimensional convolutional neural network layer, [0,69] is expressed as follows: [Number 70] is calculated by C in denotes the number of input channels of the neural network layer, and K 1 ,K 2 ,..K N denote a convolution kernel size, and j denotes an index of an output channel of the neural network layer.

15. 15. The method of any one of claims 1 to 14, wherein the neural network layer includes an attention mechanism.

16. 16. The method of claim 1, further comprising providing weights, scaling the weights by a first scaling factor to obtain scaled weights, and rounding the scaled weights to their nearest integer values ​​to obtain the integer-valued weights.

17. 17. The method of claim 16, wherein the rounding is performed by a floor function or a ceiling function.

18. 18. The method of claim 17, wherein the weights are real-valued weights and the first scaling factor is 2 sj is given by s j indicates the number of bits to represent the fractional part of the real-valued weight.

19. 19. The method of any one of claims 1 to 18, further comprising the step of scaling data entries of the input data by a second scaling factor to obtain scaled values ​​of the data entries.

20. 20. The method of claim 19, further comprising the step of rounding the scaled value of the data entry to the nearest integer value to obtain the integer value of the data entry.

21. 21. A method of encoding data comprising the steps of a method of operating a neural network according to any one of claims 1 to 20.

22. 22. The method of claim 21, wherein encoding the data comprises providing an entropy model using a neural network and entropy encoding the data based on the provided entropy model, and wherein providing the entropy model comprises performing the steps of the method for operating a neural network of any one of claims 1 to 20.

23. 21. The method of claim 20, wherein the entropy model comprises: a) Variational autoencoder hyperprior, b) Autoregressive priors for the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder The method is provided using one of the following:

24. 21. A method of decoding encoded data comprising the steps of a method of operating a neural network according to any one of claims 1 to 20.

25. 25. The method of claim 24, wherein decoding the data comprises providing an entropy model using a neural network and entropy decoding the data based on the provided entropy model, and wherein entropy decoding the data comprises steps of a method of operating a neural network as claimed in any one of claims 1 to 20.

26. 26. The method of claim 25, wherein the entropy model comprises: a) Variational autoencoder hyperprior, b) Autoregressive priors for the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder The method is provided using one of the following:

27. 27. The method of any one of claims 21 to 26, wherein the data is image data.

28. 1. A method for encoding at least a portion of an image, comprising the steps of: converting tensors representing components of the image into latent tensors; Providing an entropy model; and processing the latent tensors using a neural network based on a provided entropy model to generate a bitstream; and wherein providing the entropy model comprises performing the steps of the method according to any one of claims 1 to 20.

29. 29. The method of claim 28, wherein the entropy model comprises: a) Variational autoencoder hyperprior, b) Autoregressive priors for the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder The method is provided using one of the following:

30. 1. A method for reconstructing at least a portion of an image, comprising the steps of: Providing an entropy model; processing the bitstream using a neural network based on the provided entropy model to obtain latent tensors representing components of the image; and processing the latent tensors to obtain tensors representing components of the image; and wherein providing the entropy model comprises performing the steps of the method according to any one of claims 1 to 20.

31. 31. The method of claim 30, wherein the entropy model comprises: a) Variational autoencoder hyperprior, b) Autoregressive priors for the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder The method is provided using one of the following:

32. 32. A method according to any one of claims 28 to 31, wherein the component is a Y, U or V component, or an R, G or B component.

33. A computer program product comprising a program code stored on a non-transitory medium, the program code being adapted, when executed on one or more processors, to perform the method according to any one of claims 1 to 32.

34. 30. Apparatus for encoding data, comprising processing circuitry configured to perform the steps of the method according to any one of claims 1 to 20, 21 to 23 and 28 to 29.

35. 21. An apparatus for encoding at least a portion of an image, the apparatus comprising: a processing circuit configured to convert tensors representing components of the image into latent tensors, provide an entropy model comprising performing the steps of the method of any one of claims 1 to 20, and process the latent tensors using a neural network based on the provided entropy model to generate a bitstream.

36. 32. Apparatus for decoding data comprising a processing circuit configured to perform the steps of the method according to any one of claims 1 to 20, 24 to 26 and 30 to 31.

37. 21. An apparatus for decoding at least a portion of an encoded image, the apparatus comprising: a processing circuit configured to provide an entropy model comprising performing the steps of the method of any one of claims 1 to 20; to process a bitstream using a neural network based on the provided entropy model to obtain latent tensors representing components of the image; and to process the latent tensors to obtain tensors representing components of the image.

Images