Neural network operation with adjusted weights

Abstract

To provide a method for operating a neural network based on an adjusted weight.SOLUTION: The method includes defining, for an integer value included in a data entry of input data for a neural network layer, an integer lower threshold and an integer upper threshold, clipping the integer value included in the data entry of input data to the lower threshold if the integer value included in the data entry of input data is less than the integer lower threshold, and clipping the integer value included in the data entry of input data to the upper threshold if the integer value included in the data entry of input data is greater than the integer upper threshold, and determining an integer value weight based on the lower threshold, the upper threshold, and a predefined accumulator register size to avoid integer overflow of the accumulator register.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the present disclosure relate generally to the field of encoding and decoding data based on neural network architectures. In particular, some embodiments relate to methods and apparatus for so encoding and decoding images and / or video from a bitstream using multiple processing layers. [Background technology]

[0002] Hybrid image and video codecs have been used for decades to compress image and video data. In such codecs, signals are typically encoded block-by-block by predicting the block and then coding only the difference between the original block and its prediction. In particular, such coding involves transforming, quantizing, and generating a bitstream, which usually may include some form of entropy coding. Typically, the three components of a hybrid coding method—transform, quantizing, and entropy coding—are optimized separately. Modern video compression standards, such as High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and Essential Video Coding (EVC), also use transformed representations to code the predicted residual signal.

[0003] In recent years, neural network architectures have been applied to image and / or video coding. Generally, these neural network (NN)-based approaches can be applied to image and video coding in a variety of different ways. For example, several end-to-end optimized image or video coding frameworks have been discussed. Furthermore, deep learning has been used to determine or optimize parts of end-to-end coding frameworks, such as prediction parameter selection or compression. Furthermore, several neural network-based approaches have been discussed for use in hybrid image and video coding frameworks, for example, as pre-trained deep learning models for intra- or inter-prediction in image or video coding.

[0004] The end-to-end optimized image or video coding applications mentioned above have in common that they generate some kind of feature map data that is communicated between the encoder and decoder.

[0005] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. A corresponding feature map can be provided as the output of each hidden layer. Such corresponding feature map of each hidden layer can be used as the input to the next layer in the network, i.e., a subsequent hidden layer or output layer. Each layer of the network generates an output from the received input according to the current values ​​of its respective set of parameters. In a neural network that is partitioned between different devices, such as between an encoder and a decoder or between a device and the cloud, the feature map at the output of the partitioning site (e.g., a first device) is compressed and transmitted to the remaining layers of the neural network (e.g., a second device).

[0006] Further improvements in encoding and decoding using trained network architectures would be desirable. Summary of the Invention

[0007] This disclosure provides methods and apparatus for improving device interoperability between devices / platforms of different architectures, including neural networks. By device interoperability, it is meant that performing the same procedure on the same input data in two devices / platforms will produce the same results on both devices / platforms. Particularly in the context of entropy model-based coding and / or compression and decompression of data, such as image data, providing substantially bit-exact processing results on the encoding and decoding sides, respectively, to provide the same or complementary technical effect is a key issue.

[0008] These and other objects are achieved by the subject matter of the independent claims. Further implementations are evident from the dependent claims, the description and the drawings.

[0009] Particular implementations are outlined in the accompanying independent claims, with other implementations defined in the dependent claims.

[0010] According to a first aspect, the present disclosure relates to a method for operating a neural network including a neural network layer including or connected to an accumulator register for buffering addition results and having a predefined accumulator register size. The method includes defining a lower integer threshold A and an upper integer threshold B for integer values ​​included in data entries (e.g., numbers, vectors, or tensors) of input data for the neural network layer. The method further includes clipping the integer values ​​included in the data entries of the input data to the predefined lower integer threshold if the integer values ​​included in the data entries of the input data are less than the predefined lower integer threshold, and clipping the integer values ​​included in the data entries of the input data to the predefined upper integer threshold if the integer values ​​included in the data entries of the input data are greater than the predefined upper integer threshold. Further, the method includes determining integer-valued weights (i.e., determined weights that include integers, e.g., only integers) of the neural network layers based on a predefined integer lower threshold, a predefined integer upper threshold, and a predefined accumulator register size so as to avoid integer overflow of the accumulator register.

[0011] According to the method of the first aspect, integer weights of neural network layers of a neural network are adjusted to avoid integer overflow of the accumulator register, assuming that input data is clipped to a predefined threshold. This significantly improves interoperability between different platforms / devices compared to the prior art. The weight adjustment enables bit-exact reproduction of important mathematical operations on the encoder and decoder sides, so that the technical effects achieved by these mathematical operations are identical or complementary to each other. For example, if integer overflow of the accumulator register can be avoided on both sides, a region of an image (a still image or a frame of a video sequence) can be (entropy) coded on the encoder side and reconstructed without corruption on the decoder side. Furthermore, by providing adjusted weights determined according to the method of the first aspect, the internal operation (state) of the associated device is defined in substantially the same way. The integer weights can be determined to avoid integer overflow of the accumulator register in various ways, as described below, but this disclosure is not limited to any one of these specific methods.

[0012] The method of the first aspect works for any kind of neural network layer, including fully connected neural network layers and convolutional neural network layers. The method of the first aspect can also be advantageously implemented in a transformer architecture, where the neural network layer includes some kind of attention mechanism (see also the detailed description below). The accumulator register size is n bits, where n is a positive integer value, for example, n=32 bits or n=16 bits. The stored value is -2 n-1 or 2 n-1 Within the range of -1 or 0 to 2 n-1 The accumulator register size can be a fixed size or can be dynamically allocated.

[0013] Depending on the implementation, the lower integer threshold is less than or equal to 0 and the upper integer threshold is greater than or equal to 0. Thus, input data can be restricted to non-negative integer values, or possible or negative integer input values ​​may be allowed if deemed appropriate for the practical application.

[0014] According to the implementation, the integer lower threshold A is -2 k-1 and the upper integer threshold B is given by 2 k-1 −1, where k denotes a predefined bit depth of the layer input data. The bit depth of the layer input data is typically known for the particular application and configuration used, and can therefore be easily used to define the clipping threshold to be applied to the input data.

[0015] The adjusted weights are used in the summation process typically performed by neural network layers. Depending on the implementation, neural network layers may perform the summation (which is what is stored in the so-called accumulator registers mentioned above).

[0016]

number

[0017] A specific formulation for determining the integer weights is considered in this paper to avoid integer overflow of the accumulator register. According to the implementation, the integer weights of the neural network layers are determined by the following formula: i} is a condition

[0018]

number

[0019] Assuming that the input data values ​​are constrained to lie between specified lower and upper thresholds A and B, integer overflow of the accumulator register can be reliably avoided if these conditions are met. n-1 -1 and 2 n-1 Note that is only given as an example for the accumulator register size limit and may be replaced by any other appropriate limit that defines the accumulator register size.

[0020] Another implementation uses integer weights {w i} is a condition

[0021]

number

[0022] Another implementation uses integer weights {w i} is a condition

[0023]

number

[0024] Addition in these implementations

[0025]

number

[0026]

number

[0027]

number

[0028]

number

[0029] In particular, user-dependent application weights of neural network layers of a neural network may be provided (e.g., as real-valued weights, i.e., the weights include real numbers, e.g., only real numbers). In the case of real-valued weights, the method of the first aspect and its implementation can also be advantageously used. In this case, according to the implementation, the provided real-valued weights can be scaled by a first scaling factor to obtain scaled weights, and the scaled weights can be rounded to their nearest integer value to obtain integer-valued weights. The scaling factor can be selected with great flexibility. 2 sj It seems appropriate to have a first scaling factor given by j indicates the number of bits to represent the fractional part of the real-valued weight.

[0030] In particular, according to an implementation of the method of the first aspect, a first scaling factor s for the j-th output channel of at least one neural network layer is j satisfies the following conditions:

[0031]

number

[0032] According to another implementation, the second scaling factor s for the jth output channel of at least one neural network layer is j is given by:

[0033]

number

[0034]

number

[0035] All these conditions can ensure that integer overflow will not occur.

[0036] Similarly, according to the implementation, the method includes scaling data entries of the input data (having real numbers or integers) by a second scaling factor to obtain scaled values ​​of the data entries. The scaled values ​​of the data entities may be rounded to their nearest integer values ​​to obtain integer values ​​of the data entities. The rounding may be performed by a floor function or a ceiling function.

[0037] As already mentioned, the method of the first aspect may be advantageously applied to data coding (encoding and decoding) with the same or similar advantages as described above. Thus, according to a second aspect, there is provided a method for encoding data, including steps of the method for operating a neural network according to the first aspect or any implementation thereof. In an implementation of the method of the second aspect, encoding the data includes providing an entropy model using a neural network and entropy coding the data based on the provided entropy model, and providing the entropy model includes performing steps of the method for operating a neural network according to the first aspect or any implementation thereof. In the context of entropy coding, interoperability between different platforms is essential for reliable reconstruction of (compressed) data, with regard to substantially bit-exact reproduction of the entropy model used on both the encoder and decoder sides.

[0038] The entropy model provides statistical (probabilistic) properties of the symbols being coded or decoded, such as mean, variance, (cross)correlation, etc., and it a) Variational autoencoder hyperprior, b) the autoregressive prior of the variational autoencoder, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder The method of the second aspect can therefore advantageously be implemented in a (variational) autoencoding system (see also the detailed description below).

[0039] According to a third aspect (and complementary to the second aspect), there is provided a method of decoding encoded data, comprising the steps of the method of operating a neural network according to the first aspect or any implementation thereof. According to an implementation of the third aspect, decoding the data comprises providing an entropy model using the neural network and entropy decoding the data based on the provided entropy model, wherein entropy decoding the data comprises the steps of the method of operating a neural network according to the first aspect or any implementation thereof. Also, on the decoder side, according to an implementation of the third aspect, the entropy model is a) Hyperprior for variational autoencoder, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder It is provided using one of the following:

[0040] The data processed by the method of the second or third aspect may be image data, for example representing a still image or a frame of a video sequence.

[0041] According to a fourth aspect, there is provided a method of encoding at least a portion of an image, the method comprising: A step of converting the tensors representing the components of the image into latent tensors; Providing an entropy model; and Processing the latent tensors using a neural network based on the provided entropy model to generate a bitstream; and the step of providing the entropy model comprises performing the steps of the method according to the first aspect or an implementation thereof (having the same or similar advantages as above). According to an implementation of the fourth aspect, the entropy model comprises: a) Hyperprior for variational autoencoder, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder It is provided using one of the following:

[0042] According to a fifth aspect, there is provided a method of reconstructing at least a portion of an image, the method comprising: Providing an entropy model; processing the bitstream using a neural network based on the provided entropy model to obtain latent tensors representing components of the image; and processing the latent tensors to obtain tensors representing components of the image; and the step of providing the entropy model comprises performing the steps of the method according to the first aspect or an implementation thereof (which has the same or similar advantages as above). Again, the entropy model is a) Hyperprior for variational autoencoder, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder may be provided using one of:

[0043] The components of the image mentioned above may be Y, U or V components, or may be R, G or B components.

[0044] According to a sixth aspect, the present disclosure relates to a method for operating a neural network including at least one neural network layer including or connected to an accumulator register for buffering addition results. The method includes defining a lower integer threshold and an upper integer threshold for integer values ​​included in data entries of input data for the at least one neural network layer. The method further includes clipping the integer values ​​included in the data entries of the input data to the defined lower integer threshold if the integer values ​​included in the data entries of the input data are less than the defined lower integer threshold, and clipping the integer values ​​included in the data entries of the input data to the defined upper integer threshold if the integer values ​​included in the data entries of the input data are greater than the defined upper integer threshold, thereby avoiding integer overflow of the accumulator register.

[0045] The method of the sixth aspect significantly improves interoperability between different platforms / devices by clipping integer-valued data (including integer values) input to neural network layers. Clipping enables bit-exact reproduction of important mathematical operations on both the encoder and decoder sides, so that the technical effects achieved by these operations are identical or complementary to each other. For example, if integer overflow, which can lead to unpredictable behavior on different platforms, can be reliably avoided, an image region (a still image or a frame of a video sequence) can be (entropy) coded on the encoder side and reconstructed without corruption on the decoder side. Generally, there is no standardized handling of overflow situations, and it is difficult to imagine a commonly followed procedure in this regard. Integer overflow occurs when an arithmetic operation attempts to create a number outside the range that can be represented by a given number of digits—either larger than the largest representable value or smaller than the smallest representable value. This situation may be handled differently by different compilers, devices (CPUs, GPUs), etc. To achieve bit-exact results for integer operations on different platforms, integer overflow should be avoided. According to the method of the sixth aspect, integer overflow of the accumulator register can be avoided on both the encoder side and the decoder side. Furthermore, by clipping input data values ​​according to the method of the sixth aspect, the internal operation (state) of the involved device is defined in a substantially similar manner.

[0046] According to an implementation, the method of the sixth aspect further includes scaling data entities of the input data having real numbers by a first scaling factor (i.e., multiplying the data entities of the input data by the first scaling factor) to obtain scaled values ​​of the data entities of the input data. The scaling may be performed to improve processing of the input data values ​​on different devices. In particular, the scaling may be supplemented by rounding the scaled values ​​of the data entities of the input data to their nearest integer values ​​to obtain integer values ​​of the data entities of the input data. This allows input data containing real numbers, which may be inherently provided, to be processed by the disclosed method of operating a neural network.

[0047] The output data may be obtained by processing the input data by at least one neural network layer, and if a first scaling factor is applied, according to the implementation, output data entities included in the output data are divided by the first scaling factor to obtain de-scaled results. De-scaling may be performed directly on the output of the at least one neural network layer or after processing by an activation function. Thus, according to the implementation, the input data is processed by the at least one neural network layer to obtain output data including output data entities, the output data entities are processed by the activation function to obtain an output of the activation function, and the output of the activation function is divided by the first scaling factor.

[0048] According to another implementation, the method of the sixth aspect further comprises: processing the input data through at least one neural network layer to obtain output data including output data entities; factorizing the first scaling factor into a first part and a second part; dividing the output data entities by a first part of the factored first scaling factor to obtain partially un-scaled output data entities; processing the partially unscaled output data entities through an activation function to output an output of the activation function; and dividing the output of the activation function by the second part of the factorized first scaling factor. Since all descaling options can be implemented, the descaling process can be performed with a high degree of flexibility.

[0049] The lower integer threshold used in the method of the sixth aspect may be less than or equal to 0, and the upper integer threshold may be greater than or equal to 0. Suitable possible choices for the thresholds used are -2 for the lower integer threshold and k-1 and for the upper integer threshold, 2 k-1 -1, where k denotes the predefined bit depth (bit size) of the input data.

[0050] The method of the sixth aspect may be applied to any type of neural network and neural network layer. Depending on the implementation, at least one neural network layer is or includes one of a fully connected neural network layer and a convolutional neural network layer. Furthermore, at least one neural network layer may include an attention mechanism (see detailed description below).

[0051] Depending on the implementation, at least one neural network layer may contain integer-valued weights, which may further improve platform / device interoperability and further reduce the risk of integer overflow of accumulator register sizes (see also the discussion below).

[0052] According to an implementation, real-valued weights (weights including real numbers) are originally provided and scaled by a second scaling factor to obtain scaled weights, and the scaled weights are rounded to their nearest integer value to obtain integer-valued weights for at least one neural network, thereby allowing the method of the sixth aspect to be used with real-valued weights that are originally provided (e.g., dependent on a user).

[0053] The second scaling factor is the number of bits representing the fractional part of the real-valued weights, 2 sj In particular, depending on the implementation, the second scaling factor s for the j-th output channel of at least one neural network layer is j satisfies the following conditions:

[0054]

number

[0055] According to another implementation, the second scaling factor s for the jth output channel of at least one neural network layer is j is given by:

[0056]

number

[0057]

number

[0058] All these conditions can ensure that integer overflow will not occur.

[0059] As already mentioned, the method of the sixth aspect may be advantageously applied to data coding (encoding and decoding) with the same or similar advantages as described above. In particular, data coding and decoding based on entropy coding represents a delicate process in which interoperability of at least some of the procedures performed on different devices should yield the same results in terms of numerical precision. Thus, according to the seventh aspect, a method of encoding data (having the same advantages as described above) is provided, including steps of the method of operating a neural network according to the sixth aspect or any implementation thereof. In an implementation of the method of the seventh aspect, encoding the data includes providing an entropy model using a neural network and entropy encoding the data based on the provided entropy model, and providing the entropy model includes performing steps of the method of operating a neural network according to the sixth aspect or any implementation thereof. The entropy encoding may include entropy encoding using an arithmetic encoder.

[0060] In the context of entropy coding, interoperability between different platforms, with regard to virtually bit-exact reproduction of the entropy model used on both the encoder and decoder side, is essential for reliable reconstruction of the (compressed) data.

[0061] The entropy model provides statistical (probabilistic) properties of the symbols being coded or decoded, such as mean, variance, (cross)correlation, etc., and it a) Hyperprior for variational autoencoder, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder The method of the seventh aspect may therefore advantageously be implemented in a (variational) auto-encoding system (see also the detailed description below).

[0062] The method for encoding data as described above may further include signaling the defined lower threshold and the defined upper threshold to a decoder side, thereby easily providing the decoder side with information necessary for the same clipping process of the input data as performed at the encoder side. Similarly, at least one of the first scaling factor, the second scaling factor, and the third scaling factor may be signaled to the decoder side in the bitstream.

[0063] The method may further include signaling to the decoder side the difference relative to the predefined lower threshold and the difference relative to the predefined upper threshold. The method may further include signaling to the decoder side in the bitstream at least one of the difference relative to the predefined first scaling factor and the difference relative to the predefined second scaling factor. Exponential-Golomb coding may be used for the signaling.

[0064] According to an eighth aspect (and complementary to the seventh aspect), there is provided a method of decoding encoded data, the method comprising the steps of the method of operating a neural network according to the sixth aspect or any implementation thereof (having the same advantages as above). According to the eighth implementation, decoding the data comprises providing an entropy model using a neural network and entropy decoding the data based on the provided entropy model, and entropy decoding the data comprises the steps of the method of operating a neural network according to the sixth aspect or any implementation thereof. The entropy decoding may comprise entropy decoding using an arithmetic decoder. Also, on the decoder side, according to the implementation of the eighth aspect, the entropy model may be a) Hyperprior for variational autoencoder, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder It is provided using one of the following:

[0065] Information about the defined lower threshold and the defined upper threshold may be received at the decoder side in the bitstream from the encoder side. Similarly, information about at least one of the first scaling factor, the second scaling factor, and the third scaling factor may be received at the decoder side in the bitstream from the encoder side.

[0066] Further, the method may include receiving, at the decoder side, information regarding the difference relative to a predefined lower threshold and the difference relative to a predefined upper threshold. Further, the method may include receiving, at the decoder side, information regarding at least one of the difference relative to a predefined first scaling factor and the difference relative to a predefined second scaling factor in the bitstream.

[0067] The data processed by the method of the second or eighth aspect may be image data, for example a still image or a frame of a video sequence.

[0068] According to a ninth aspect, there is provided a method of encoding at least a portion of an image, the method comprising: A step of converting the tensors representing the components of the image into latent tensors; Providing an entropy model; and Processing the latent tensors using a neural network based on the provided entropy model to generate a bitstream; and wherein providing the entropy model comprises performing the steps of a method according to the sixth aspect or an implementation thereof (having the same or similar advantages as above). The latent tensor may be processed using an arithmetic encoder.

[0069] The entropy model is a) Hyperprior for variational autoencoder, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder may be provided using one of:

[0070] According to implementation, the method of the ninth aspect may further include signaling the defined lower threshold and the defined upper threshold to the decoder side in the bitstream.

[0071] Furthermore, at least one of the first scaling factor, the second scaling factor, and the third scaling factor may be signaled to the decoder side in the bitstream.

[0072] The method may further include signaling to the decoder side the difference relative to the predefined lower threshold and the difference relative to the predefined upper threshold. The method may further include signaling to the decoder side in the bitstream at least one of the difference relative to the predefined first scaling factor and the difference relative to the predefined second scaling factor. Exponential-Golomb coding may be used for the signaling.

[0073] According to a tenth aspect, there is provided a method of reconstructing at least a portion of an image, the method comprising: Providing an entropy model; processing the bitstream using a neural network based on the provided entropy model to obtain latent tensors representing components of the image; and processing the latent tensors to obtain tensors representing components of the image; and the step of providing the entropy model comprises performing the steps of the method for operating a neural network according to the sixth aspect or any implementation thereof (having the same advantages as above). The processing of the bitstream may be performed using an arithmetic decoder. The entropy model may be a) Hyperprior for variational autoencoder, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder may be provided using one of:

[0074] According to an implementation, the method of the tenth aspect includes reading information about the defined lower threshold and the defined upper threshold from the bitstream. Further, at least one of the first scaling factor, the second scaling factor, and the third scaling factor may be read from the bitstream. Further, the method may include receiving, at the decoder side, information regarding the difference relative to a predefined lower threshold and the difference relative to a predefined upper threshold. Further, the method may include receiving, at the decoder side, information regarding at least one of the difference relative to a predefined first scaling factor and the difference relative to a predefined second scaling factor in the bitstream.

[0075] The components of an image represented by a tensor according to the methods of the third and ninth aspects may be Y, U or V components, or may be R, G or B components.

[0076] According to an eleventh aspect, there is provided a neural network including: a neural network layer configured to process input data to obtain output data; an activation function configured to process the output data to obtain activation function output data; a first unit configured to scale, round, and clip the input data input to the neural network layer; and a second unit configured to de-scale at least one of the output data and the activation function output data. The first unit is divided into sub-units, each configured for various operations such as scaling, rounding, and clipping of the input data, and the first unit and each sub-unit are switchable between on and off.

[0077] The first unit may be configured to perform the steps of the methods according to the first to sixth aspects and their implementations.

[0078] According to implementations, the neural network of the eleventh aspect is configured to perform the steps of the methods of the first to sixth aspects and their implementations.

[0079] According to a twelfth aspect, there is provided an apparatus for encoding at least a portion of an image, the apparatus including a neural network according to the twelfth aspect or an implementation thereof. a) Hyperprior for variational autoencoder, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder and each prior comprises a neural network of the twelfth aspect or an implementation thereof.

[0080] According to a thirteenth aspect, there is provided an apparatus for decoding at least a portion of an encoded image, the apparatus comprising a neural network according to the twelfth aspect or an implementation thereof. This may be an apparatus for decoding at least a portion of an image, and a) Hyperprior for variational autoencoder, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder and each prior comprises a neural network of the twelfth aspect or an implementation thereof.

[0081] To ensure device interoperability across neural networks, bit-exact reproducibility of activation functions across different platforms / devices is desirable. For example, this requirement can be met relatively easily for linear activation functions and relatively simple non-linear activation functions, such as the ReLU function, which essentially defines a clipping procedure. For more complex non-linear ones, especially those involving exponential functions, such as Softmax:

[0082]

number

[0083] Thus, according to a fourteenth aspect, there is provided a neural network comprising at least one neural network layer and an activation function connected to an output of the at least one neural network layer, wherein the activation function is implemented as an approximation function of a mathematically defined real-valued non-linear activation function (one that takes real numbers as arguments and outputs real numbers), the approximation function allowing processing of only integers in a fixed-point representation of input values ​​of the approximation function.

[0084] In fixed-point representation / arithmetic, real numbers are mantissa*base 指数 where both the base and the exponent are fixed, and thus fixed-point representations of fractions are essentially integers (see also the detailed explanation below).

[0085] The approximation function approximates a mathematically defined real-valued nonlinear activation function used in the art (which often includes one or more exponential functions (with base e) and operates on floating-point representations of the input values) so that only integers in the fixed-point representation of the input values ​​can be processed (as in the case of, for example, the ReLU function).

[0086] Integer-only processing enables fixed-point neural networks, potentially achieving bit-exact behavior across different platforms. This significantly improves interoperability between different platforms / devices compared to the prior art. Providing such approximation activation functions allows bit-exact reproduction of key numerical operations on both the encoder and decoder sides, so that the technical effects achieved by these operations are identical or complementary to each other. For example, if bit-exact behavior is possible on both sides, a region of an image (a still image or a frame of a video sequence) can be (entropy) coded on the encoder side and reconstructed without corruption on the decoder side. Furthermore, by providing neural networks including approximated activation functions according to the method of the fourteenth aspect for different devices communicating with each other, the internal operation (state) of the associated devices is defined in a substantially identical manner. The approximate activation functions implemented in the neural network of the fourteenth aspect can be selected in different ways, as described below, and the present disclosure is not limited to one of these specific methods.

[0087] According to some implementations, the approximation function includes at least one of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function. According to further embodiments of the neural network of the fourteenth aspect, the mathematically defined nonlinear activation function (i.e., the one approximated by the approximation function) is selected from the group consisting of a Softmax function, a sigmoid function, a hyperbolic tangent function, a Swish function, a Gaussian error linear unit function, and a scaling exponential linear unit function. For all of these mathematically defined nonlinear activation functions that may be suitable for practical applications, it is possible to find approximation functions that allow processing only integers in a fixed-point representation of the input values.

[0088] According to the implementation, the approximate function (approximate activation function) is a) the expected input values ​​of the data input to the approximation function or at least one neural network layer, b) the accumulator register size for buffering the summation results formed by the neural network, and c) Approximation functions The method includes a finite number of Taylor series sums determined based on at least one of:

[0089] According to another implementation, the approximation function is a polynomial function, the maximum degree of which is a) the expected input values ​​of the data input to the approximation function or at least one neural network layer, b) the accumulator register size for buffering the summation results formed by the neural network, and c) Approximation functions The determination is based on at least one of the following:

[0090] According to another embodiment, the approximation function is a rational function, and the maximum degree of the polynomials in the numerator and denominator of the rational function is: a) the expected input values ​​of the data input to the approximation function or at least one neural network layer, b) the accumulator register size for buffering the summation results formed by the neural network, and c) Approximation functions The determination is based on at least one of the following:

[0091] For a mathematically defined nonlinear activation function f(x), the Taylor series

[0092]

number

[0093]

number

[0094] 2 n-1 -1 and -2 n-1 Note that is only an example of a limit on the accumulator register size and can be replaced by other limits that properly define the accumulator register size in the condition.

[0095] A particularly important activation function in many applications is Softmax

[0096]

number

[0097] According to the neural network implementation of the fourteenth aspect, the mathematically defined nonlinear activation function is a Softmax function, and the approximation function (approximate activation function) is defined as follows:

[0098]

number

[0099] According to another implementation, the (approximate activation function) is defined as:

[0100]

number

[0101] The positive constant ε can be easily selected depending on, for example, the expected value of the components of the input vector x. The positive constant ε is, for example, 10 -15 or 10 -11 The range may be within.

[0102] Numerical experiments prove that this kind of approximation function to the Softmax function based on the ReLU function can guarantee the bit-exact behavior of the application of the activation function on different platforms.

[0103] The implementation of the approximate activation function can advantageously be realized in any neural network, in particular a neural network that may be a convolutional or fully connected neural network or may include convolutional or fully connected neural network layers.

[0104] According to a fifteenth aspect, there is provided a method of operating a neural network (e.g. a convolutional or fully connected neural network) comprising at least one neural network layer (having the same advantages as those provided by the neural network of the fourteenth aspect), the method comprising implementing an approximation function of a mathematically defined real-valued non-linear activation function as an activation function of the at least one neural network layer, the approximation function allowing processing of only integers in fixed-point representation of input values ​​of the approximation function.

[0105] According to an implementation, the approximation function includes at least one of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function. According to a further implementation, the mathematically defined nonlinear activation function is selected from the group consisting of a Softmax function, a sigmoid function, a hyperbolic tangent function, a swish function, a Gaussian error linear unit function, and a scaling exponential linear unit function.

[0106] The approximation function may include a finite number of Taylor series sums determined based on expected input values ​​of the data input to at least one neural network layer or approximation function.

[0107] Taylor series of mathematically defined nonlinear activation function f(x)

[0108]

number

[0109]

number

[0110] The mathematically defined non-linear activation function may be a Softmax function, in which case the approximation function implemented according to the fifteenth aspect or method of implementation thereof is given as:

[0111]

number

[0112] Alternatively, the approximation function can be defined as follows:

[0113]

number

[0114]

number

[0115] The approximate activation function implemented in the neural network of the fourteenth aspect and the method of the fifteenth aspect may be advantageously applied to data coding (encoding and decoding) with the same or similar advantages as those described above. According to a sixteenth aspect, a method for encoding data is provided, including steps of the method for operating a neural network according to the fourteenth aspect or any implementation thereof (with the same advantages as described above). In an implementation of the method of the sixteenth aspect, encoding the data includes providing an entropy model using a neural network and entropy encoding the data based on the provided entropy model, and providing the entropy model includes performing steps of the method for operating a neural network according to the fifteenth aspect or any implementation thereof. The entropy encoding may include entropy encoding using an arithmetic encoder.

[0116] In the context of entropy coding, interoperability between different platforms, with regard to virtually bit-exact reproduction of the entropy model used on both the encoder and decoder side, is essential for reliable reconstruction of the (compressed) data.

[0117] The entropy model provides statistical (probabilistic) properties of the symbols being coded or decoded, such as mean, variance, (cross)correlation, etc., and it a) Hyperprior for variational autoencoder, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder The method of the sixteenth aspect may therefore advantageously be implemented in a (variational) auto-encoding system (see also the detailed description below).

[0118] According to a seventeenth aspect (and complementary to the sixteenth aspect), there is provided a method of decoding encoded data, the method comprising the steps of the method of operating a neural network according to the fifteenth aspect or any implementation thereof (having the same advantages as above). According to an implementation of the seventeenth aspect, decoding the data comprises providing an entropy model using a neural network and entropy decoding the data based on the provided entropy model, and entropy decoding the data comprises the steps of the method of operating a neural network according to the fifteenth aspect or any implementation thereof. The entropy decoding may comprise entropy decoding using an arithmetic decoder. Also, on the decoder side, according to an implementation of the seventeenth aspect, the entropy model is a) Hyperprior for variational autoencoder, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder It is provided using one of the following:

[0119] The data processed by the method of the sixteenth or seventeenth aspect may be image data, for example representing a still image or a frame of a video sequence.

[0120] According to an eighteenth aspect, there is provided a method of encoding at least a portion of an image, the method comprising: A step of converting the tensors representing the components of the image into latent tensors; Providing an entropy model; and Processing the latent tensors using a neural network based on the provided entropy model to generate a bitstream; and providing the entropy model comprises performing the steps of a method according to the fifteenth aspect or an implementation thereof (having the same advantages as above). The latent tensor may be processed using an arithmetic encoder.

[0121] The entropy model is a) Hyperprior for variational autoencoder, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder may be provided using one of:

[0122] According to a nineteenth aspect, there is provided a method of reconstructing at least a portion of an image, the method comprising: Providing an entropy model; processing the bitstream using a neural network based on the provided entropy model to obtain latent tensors representing components of the image; and processing the latent tensors to obtain tensors representing components of the image; and the step of providing the entropy model comprises performing the steps of the method for operating a neural network according to the fifteenth aspect or any implementation thereof (having the same advantages as above). The processing of the bitstream may be performed using an arithmetic decoder. The entropy model may be a) Hyperprior for variational autoencoder, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder Further, processing the latent tensors to obtain tensors representing components of the image may include performing steps of a method according to the 21st aspect or an implementation thereof.

[0123] The components of an image represented by a tensor according to the methods of the eighteenth and nineteenth aspects may be Y, U, or V components, or may be R, G, or B components.

[0124] According to a twentieth aspect, an approximate activation function (one that approximates a Softmax function) is provided as follows:

[0125]

number

[0126] According to a twenty-first aspect, an approximate activation function (one that approximates a Softmax function) is provided as follows:

[0127]

number

[0128] The positive constant ε can be easily selected depending on, for example, the expected value of the components of the input vector x. The positive constant ε is, for example, 10 -15 or 10 -11 The range may be within.

[0129] These new activation functions can replace the traditional Softmax function and may prove superior in terms of device interoperability (as encoders and decoders).

[0130] According to a twenty-second aspect, there is provided an apparatus for encoding data, comprising a neural network according to the fourteenth aspect or any implementation thereof.

[0131] According to a twenty-third aspect, there is provided an apparatus for decoding data, comprising a neural network according to the fourteenth aspect or any implementation thereof.

[0132] According to a twenty-fourth aspect, there is provided an apparatus for encoding at least a portion of an image, comprising a neural network according to the fourteenth aspect or any implementation thereof.

[0133] According to a twenty-fifth aspect, there is provided an apparatus for decoding at least a part of an image, comprising a neural network according to the fourteenth aspect or any implementation thereof.

[0134] According to a 26th aspect, there is provided an apparatus according to any of the 22nd to 25th aspects, comprising a hyperprior of a variational autoencoder, the hyperprior comprising a neural network according to the 14th aspect or any implementation thereof.

[0135] According to a 27th aspect, there is provided an apparatus for encoding at least a portion of an image, the apparatus comprising: Convert the tensors representing the image components into latent tensors; Providing an entropy model using a neural network according to the fourteenth aspect or any implementation thereof; and Process the latent tensors based on the provided entropy model to generate a bitstream; The processing circuitry is configured to:

[0136] According to a twenty-eighth aspect, there is provided an apparatus for decoding at least a portion of an encoded image, the apparatus comprising: Providing an entropy model using a neural network according to the fourteenth aspect or any implementation thereof; Process the bitstream based on the provided entropy model to obtain latent tensors that represent the components of the image; and Process the latent tensors to obtain tensors that represent the components of the image; The processing circuitry is configured to:

[0137] The components of an image may be Y, U, or V components, or may be R, G, or B components.

[0138] According to a twenty-ninth aspect, there is provided a computer program product comprising program code stored on a non-transitory medium, the program, when executed on one or more processors, performing a method according to any of the above aspects relating to the method and any implementation thereof.

[0139] According to a thirtieth aspect, there is provided a computer program product comprising program code stored on a non-transitory medium, the program, when executed on one or more processors, performing a method according to any of the above aspects relating to the method and any implementation thereof.

[0140] According to a thirty-first aspect, there is provided a computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to encode video data, the instructions causing the one or more processors to perform a method according to any of the above aspects relating to the method and any implementation thereof.

[0141] Any of the above aspects relating to methods and methods according to any implementation thereof may be implemented in an apparatus, whereby an apparatus configured to perform the steps of such a method is provided (with the same advantages as above).

[0142] According to a thirty-second aspect, there is provided an apparatus for encoding data, the apparatus including processing circuitry configured to perform any method steps, not limited to decoding, according to any of the above aspects relating to the methods and any implementations thereof.

[0143] According to a thirty-third aspect, there is provided an apparatus for encoding at least a portion of an image, the apparatus including processing circuitry configured to convert tensors representing components of the image into latent tensors, provide an entropy model including performing any method steps according to any aspect above relating to the method and any implementation thereof, including but not limited to decoding, and process the latent tensors using a neural network based on the provided entropy model to generate a bitstream.

[0144] According to a thirty-fourth aspect, there is provided an apparatus for decoding data, the apparatus including processing circuitry configured to perform any method steps, not limited to encoding, according to any of the above aspects relating to methods and any implementation thereof.

[0145] According to a thirty-fifth aspect, there is provided an apparatus for decoding at least a portion of an encoded image, the apparatus including: providing an entropy model that includes performing any method steps according to any of the above aspects relating to the method and any implementation thereof, not limited to encoding; processing the bitstream using a neural network based on the provided entropy model to obtain latent tensors representing components of the image; and processing the latent tensors to obtain tensors representing the components of the image.

[0146] The functions of the above-described devices may be realized by hardware, or by hardware that executes corresponding software.

[0147] According to a further aspect, the present disclosure relates to a video stream encoding device including a processor and a memory storing instructions for causing the processor to perform any method steps, including but not limited to decoding, according to any of the above method-related aspects and any implementation thereof.

[0148] According to a further aspect, the present disclosure relates to a video stream decoding device including a processor and a memory storing instructions for causing the processor to perform any method steps, not limited to encoding, according to any of the above method-related aspects and any implementation thereof.

[0149] The above-described aspects and implementation methods and apparatus may be readily combined with one another where considered appropriate.

[0150] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0151] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings and figures. [Figure 1] Figure 1 is a schematic diagram showing the channels processed by the layers of a neural network. [Figure 2] Figure 2 is a schematic diagram showing an autoencoder type neural network. [Figure 3A] Figure 3A is a schematic diagram showing an exemplary network architecture, where the encoder and decoder sides include hyperprior models. [Figure 3B] Figure 3B is a schematic diagram showing the general network architecture, where the encoder side includes a hyperprior model. [Figure 3C] Figure 3C is a schematic diagram showing the general network architecture, where the decoder side includes a hyperprior model. [Figure 4] Figure 4 is a schematic diagram showing an example network architecture, where the encoder and decoder sides include a hyperprior model. [Figure 5] Figure 5 is a block diagram illustrating the structure of a cloud-based solution for machine-based tasks such as machine vision tasks. [Figure 6] Figure 6 is a block diagram illustrating a neural network-based end-to-end video compression framework. [Figure 7] FIG. 7 shows the transformer. [Figure 8] Figure 8 shows the neural network layers. [Figure 9] Figure 9 shows a neural network according to an embodiment. [Figure 10] FIG. 10 is a flowchart illustrating a method of operating a neural network including clipping input data. [Figure 11] FIG. 11 illustrates a neural network including units for clipping, scaling, and rounding, as well as de-scaling, according to an embodiment. [Figure 12] FIG. 12 is a flowchart illustrating a method of operating a neural network including adjusting the weights of neural network layers. [Figure 13] FIG. 13 is a flowchart illustrating a method of operating a neural network, including implementing an approximate activation function. [Figure 14] FIG. 14 illustrates a neural network including an approximate activation function according to an embodiment. [Figure 15] FIG. 15 illustrates an apparatus configured to perform the steps of the methods illustrated in FIGS. [Figure 16] FIG. 16 is a block diagram illustrating an example of a video coding system configured to implement embodiments of the present disclosure. [Figure 17] FIG. 17 is a block diagram illustrating another example of a video coding system configured to implement embodiments of the present disclosure. [Figure 18] FIG. 18 is a block diagram showing an example of an encoding device or a decoding device. [Figure 19] 19 is a block diagram showing another example of an encoding or decoding device.Like reference numbers and names in different drawings may indicate like elements. DETAILED DESCRIPTION OF THE INVENTION

[0152] In the following description, reference is made to the accompanying drawings that form a part of this disclosure and that show, by way of illustration, specific aspects of embodiments of the present disclosure or in which embodiments of the present disclosure may be used. It is understood that embodiments of the present disclosure may be used in other ways and may include structural or logical changes not depicted in the drawings. Therefore, the following detailed description is not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

[0153] For example, it is understood that disclosure related to a described method may also apply to a corresponding device or system configured to perform that method, and vice versa. For example, when one or more particular method steps are described, a corresponding device may include one or more units, e.g., functional units, for performing the one or more method steps (e.g., one unit performing one or more steps, or multiple units performing one or more of the steps, respectively), even if such one or more units are not explicitly described or shown in a drawing. Conversely, for example, when a particular apparatus is described based on one or more units, e.g., functional units, a corresponding method may include a step that performs the function of one or more units (e.g., one step that performs the function of one or more units, or multiple steps that perform one or more functions of multiple units, respectively), even if such one or more steps are not explicitly described or shown in a drawing. Furthermore, it is understood that features of various exemplary embodiments and / or aspects described herein may be combined with each other, unless specifically indicated otherwise.

[0154] The following provides an overview of the framework in which embodiments of the present disclosure may be used and some of the terminology used.

[0155] Artificial Neural Networks Artificial neural networks (ANNs), or connectionist systems, are computing systems loosely inspired by the biological neural networks that make up animal brains. Such systems generally "learn" to perform tasks by considering examples, rather than being programmed with task-specific rules. For example, in image recognition, a system might learn to identify images containing cats by analyzing example images that have been manually labeled as "cat" or "not cat," and then use the results to identify cats in other images. They do this without any prior knowledge of cats, such as whether they have fur, tails, whiskers, or cat-like faces. Rather, they automatically develop discriminatory features from the examples they process.

[0156] ANNs are based on a collection of connected units or nodes called artificial neurons, which loosely model the neurons in a biological brain. Each connection can transmit a signal to other neurons, similar to a synapse in a biological brain. The artificial neuron can then process the signal and send a signal to the neurons connected to it.

[0157] In an ANN implementation, the "signals" in the connections are real numbers, and the output of each neuron is calculated by some nonlinear function of the sum of its inputs. The connections are called edges. Neurons and edges typically have weights that are adjusted as learning progresses. The weights increase or decrease the strength of the signal in the connection. Neurons have thresholds so that a signal is sent only if the aggregated signal exceeds the threshold. Neurons are typically aggregated in layers. Different layers may perform different transformations on their inputs. A signal progresses from the first layer (input layer) to the last layer (output layer), possibly after traversing the layers multiple times.

[0158] The original goal of the ANN approach was to solve problems in the same way as the human brain. Over time, interest has shifted to performing specific tasks, leading to a deviation from biology. ANNs are used in a variety of tasks, including computer vision, speech recognition, machine translation, social network filtering, playing board and video games, medical diagnosis, and even activities traditionally considered exclusive to humans, such as painting.

[0159] The name "convolutional neural network" (CNN) indicates that the network uses a mathematical operation called convolution, which is a special kind of linear operation. A convolutional network is a neural network that uses convolution instead of the more common matrix multiplication in at least one of its layers.

[0160] Figure 1 shows a schematic of the general concept of processing by a neural network such as a CNN. A convolutional neural network consists of input and output layers and multiple hidden layers. The input layer is the layer to which input (such as a portion of the input image 11 shown in Figure 1) is provided for processing. The hidden layers of a CNN typically consist of a series of convolutional layers that perform convolutions using multiplication or other dot products. The output of a layer is one or more feature maps (shown as solid rectangles), sometimes called channels. There may be resampling (a kind of subsampling) in some or all of the layer operations. As a result, the feature maps may become smaller, as shown in Figure 1. Note that convolutions using strides can also reduce (resample) the size of the input feature maps. The activation function in a CNN is typically a ReLU (rectified linear unit) layer, followed by additional convolutions such as pooling, fully connected, and normalization layers, which are referred to as hidden layers because their inputs and outputs are masked by the activation function and the final convolution. The layer is colloquially called convolutional, but this is merely by convention. Mathematically, it is strictly a sliding dot product or cross-correlation. The index of the matrix is ​​important in that it affects how the weights are determined at a particular index point.

[0161] When programming a CNN to process images, the input is a tensor with dimensions (number of images) x (image width) x (image height) x (image depth), as shown in Figure 1. It is important to note that the image depth can be constructed by the image channels. After passing through the convolutional layer, the image is abstracted into a feature map with dimensions (number of images) x (feature map width) x (feature map height) x (feature map channels). A convolutional layer in a neural network should have the following attributes: width and height (hyper-parameters), and a convolution kernel defined by the number of input and output channels (hyper-parameters). The depth of the convolutional filter (input channels) should be equal to the number of channels (depth) of the input feature map.

[0162] Traditional multilayer perceptron (MLP) models have been used for image recognition. However, due to full connectivity between nodes, they suffer from high dimensionality and do not scale well to higher-resolution images. A 1000 x 1000 pixel image with RGB color channels has 3 million weights, which is excessively large and cannot be efficiently processed at a scale with full connectivity. Furthermore, such network architectures do not consider the spatial structure of the data, treating input pixels that are far apart the same as pixels that are close to each other. This ignores the locality of reference in image data, both computationally and semantically. Therefore, full neuronal connectivity is wasteful for purposes such as image recognition, which are dominated by spatially local input patterns.

[0163] Convolutional neural networks are biologically inspired variants of multilayer perceptrons specifically designed to emulate the behavior of the visual cortex. These models mitigate the problems posed by MLP architectures by exploiting the strong spatial local correlations present in natural images. Convolutional layers are the core building blocks of CNNs. The layer's parameters consist of a set of learnable filters (kernels, as described above) that have small receptive fields but extend across the entire depth of the input volume. During a forward pass, each filter is convolved across the width and height of the input volume, computing dot products between the filter's entries and the input, generating a two-dimensional activation map for that filter. As a result, the network learns a filter that activates when it detects some particular type of feature at some spatial location in the input.

[0164] Stacking the activation maps of all filters along the depth dimension forms the complete output volume of a convolutional layer. Therefore, every entry in the output volume can be interpreted as the output of a neuron that looks at a small region of the input and shares parameters with neurons in the same activation map. A feature map or activation map is the output activation of a given filter. Feature map and activation have the same meaning. In some papers, it is called an activation map because it is a mapping corresponding to the activation of different parts of the image, and it is also called a feature map because it is a mapping of where certain features are found in the image. High activation means that a particular feature is found.

[0165] Another important concept in CNNs is pooling, which is a form of nonlinear down-sampling. There are several nonlinear functions to implement pooling, among which max pooling is the most common. It divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each such subregion.

[0166] Intuitively, the exact location of a feature is less important than its rough location relative to other features. This is the idea behind using pooling in convolutional neural networks. Pooling layers serve to progressively reduce the spatial size of the representation, reducing the number of parameters in the network, its memory footprint, and the amount of computation, and thus curbing overfitting. In CNN architectures, it is common to periodically insert pooling layers between successive convolutional layers. The pooling operation provides another form of translation invariance.

[0167] A pooling layer operates independently on every depth slice of the input, spatially resizing it. The most common form is a pooling layer in which a 2x2 filter is applied along both the width and height of every depth slice of the input, two at a time, with a stride of 2, discarding 75% of the activations. In this case, all max operations are over four orders of magnitude larger. The depth dimension remains unchanged. In addition to max pooling, the pooling unit can use other functions such as average pooling or l2-norm pooling. While average pooling was often used historically, it has recently fallen out of favor compared to max pooling, which often performs better in practice. Due to the significant reduction in representation size, there is a recent trend to use smaller filters or even discard the pooling layer entirely. "Region of interest" pooling (also known as ROI pooling) is a variant of max pooling in which the output size is fixed and the input rectangle is a parameter. Pooling is a key component of convolutional neural networks for object detection based on the Faster R-CNN architecture.

[0168] The ReLU, mentioned above, stands for rectified linear unit and applies a non-saturating activation function. It effectively removes negative values ​​from the activation map by setting them to zero. This increases the non-linearity of the decision function and the network as a whole without affecting the receptive fields of the convolutional layers. Other functions, such as saturated hyperbolic tangents and sigmoid functions, are also used to increase non-linearity. ReLU is often preferred over other functions because it trains neural networks many times faster without a significant penalty to generalization accuracy.

[0169] After several convolutional and max-pooling layers, higher-level inference in neural networks occurs via fully connected layers. Neurons in fully connected layers have connections to all activations of the previous layer, as in regular (non-convolutional) artificial neural networks. These activations can then be computed as an affine transformation with a matrix multiplication followed by a bias offset (vector addition of a learned or fixed bias term).

[0170] The "loss layer" (which contains the computation of the loss function) specifies how training penalizes deviations between predicted (output) and true labels and is usually the final layer of a neural network. It is possible to use different loss functions suited to different tasks: softmax loss is used to predict one class out of K mutually exclusive classes; sigmoid cross entropy loss is used to predict K independent probability values ​​in [0,1]; and Euclidean loss is used for regression on real-valued labels.

[0171] To summarize, Figure 1 shows the data flow in a typical convolutional neural network. First, an input image passes through a convolutional layer and is abstracted into a feature map containing several channels, corresponding to a number of filters in the layer's set of learnable filters. The feature map is then subsampled, for example, using a pooling layer to reduce the dimensionality of each channel in the feature map. The data then enters another convolutional layer, which may have a different number of output channels. As mentioned previously, the number of input and output channels is a layer hyper-parameter. To establish network connectivity, these parameters must match between the two connected layers; therefore, the number of input channels in the current layer should be equal to the number of output channels in the previous layer. For the first layer, which processes input data, e.g., an image, the number of input channels is typically equal to the number of channels in the data representation, e.g., three channels for an RGB or YUV representation of an image or video, or one channel for a grayscale image or video representation. The channels obtained by one or more convolutional layers (and possibly a resampling layer) can be passed to the output layer. Such an output layer may be a convolution or a resampling in some implementations. In an exemplary, non-limiting embodiment, the output layer is a fully connected layer.

[0172] Autoencoders and Unsupervised Learning An autoencoder is a type of artificial neural network used to learn efficient data coding in an unsupervised manner. Its schematic is shown in Figure 2. An autoencoder has an encoder side 210, where an input x is input to the input layer of an encoder subnetwork 220, and a decoder side 250, where an output x' is output from a decoder subnetwork 260. The goal of an autoencoder is to learn a representation (encoding) 230 for a set of data x, typically for dimensionality reduction, by training the networks 220, 260 to ignore signal "noise." Along with the reduction (encoder) side subnetwork 220, a reconstruction (decoder) side subnetwork 260 is trained, where the autoencoder, as the name suggests, attempts to generate a representation x' from the reduced encoding 230 that is as close as possible to its original input x. In the simplest case, given a single hidden layer, the encoder stage of the autoencoder takes an input and maps it to h.

[0173]

number

[0174] This image h is usually called the code230, latent variable, or latent representation. Here, σ is an element-wise activation function such as the sigmoid function or the normalized linear unit. W is a weight matrix, and b is a bias vector. The weights and biases are usually initialized randomly and then iteratively updated during training through backpropagation. The decoder stage of the autoencoder then maps h to a reconstruction x' that has the same shape as x.

[0175]

number

[0176] Variational autoencoder models impose strong assumptions on the distribution of latent variables. They use a variational approach to latent representation learning, which results in an additional loss component and a specific estimator for the training algorithm, called the Stochastic Gradient Variational Bayes (SGVB) estimator. The data is a directed graph model p θ (x|h), and the encoder is given by the posterior distribution p θ Approximation q to (h|x) φ We assume that we learn (h|x), where φ and θ denote the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vectors in a VAE typically matches the probability distribution of the training data much more closely than a standard autoencoder. The goal of a VAE has the following form:

[0177]

number

[0178] where D KL denotes the Kullback-Leibler divergence. The prior distribution for the latent variables is usually a centrally isotropic multivariate Gaussian function p θ We set (h) = N(0,I). In general, the shapes of the variation and likelihood distributions are chosen so that they are factored Gaussian functions.

[0179]

number

[0180] Recent advances in the area of ​​artificial neural networks, especially convolutional neural networks, have drawn researchers' attention to applying neural network-based techniques to image and video compression tasks. For example, end-to-end optimized image compression has been proposed, which uses networks based on variational autoencoders.

[0181] Therefore, data compression is considered a fundamental and well-studied problem in engineering, commonly formulated with the goal of designing a code for a given discrete data ensemble using minimum entropy. The solution relies heavily on knowledge of the data's probabilistic structure, and the problem is therefore closely related to probabilistic source modeling. However, because all practical codes must have finite entropy, continuous-valued data (such as vectors of image pixel intensities) must be quantized into a finite set of discrete values, which introduces error.

[0182] In this context, known as a lossy compression problem, we must trade off two competing costs: the entropy of the discretized representation (rate) and the error resulting from quantization (distortion). Different compression applications, such as data storage or transmission over limited-capacity channels, require different rate-distortion tradeoffs.

[0183] Joint optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in high-dimensional spaces cannot be addressed. For this reason, most existing image compression methods work by linearly transforming the data vector into an appropriate continuous-valued representation, independently quantizing its elements, and then encoding the resulting discrete representation using a lossless entropy code. This approach is called transform coding, due to the central role of the transform.

[0184] For example, JPEG uses the discrete cosine transform on blocks of pixels, while JPEG 2000 uses multi-scale orthogonal wavelet decomposition. Typically, the three components of a transform coding method—transform, quantization, and entropy code—are optimized separately (often by manual parameter adjustment). Recent video compression standards, such as HEVC, VVC, and EVC, also use transformed representations to code the residual signal after prediction. Several transforms are used for this purpose, including the discrete cosine and sine transforms (DCT, DST) and the low-frequency non-separable manually optimized transform (LFNST).

[0185] Variational Image Compression The variational autoencoder (VAE) framework can be considered as a nonlinear transform coding model. The transformation process can be divided into four main parts. This is illustrated in Figure 3A, which shows the VAE framework.

[0186] The transformation process can be divided into four main parts: Figure 3A illustrates the VAE framework. In Figure 3A, the encoder 101 maps the input image x to a latent representation (denoted as y) by a function y = f(x). This latent representation may be referred to as a portion or point in the "latent space" below. The function f() is a transformation function that transforms the input signal x into a more compressible representation y. The quantizer 102 converts the latent representation y into a quantized representation y. ^ = Q(y) = Q(y) = Q(y) = Q(y) ^ where Q represents the quantization function. The entropy model, or hyper-encoder / decoder (also known as hyper-prior) 103, converts the quantized latent representation y ^ Estimate the distribution of ,to obtain the minimum achievable rate with lossless entropy source coding.

[0187] A latent space can be understood as a compressed data representation where similar data points are close to each other in the latent space. Latent spaces are useful for learning features of the data and finding simpler representations of the data for analysis. The quantized latent representation, T,y, ^ and the side information z of Hyperprior 3 ^ is included in bitstream 2 (which is binarized) using arithmetic coding (AE). Furthermore, the quantized latent representation is used to encode the reconstructed image x ^ ,x ^ =g(y ^ ) is provided. ^ is an estimate of the input image x. x is as close as possible to x ^ It is desirable that the reconstruction quality is as close as possible to x. ^ The higher the similarity between x and x, the greater the amount of side information required for transmission. The side information includes bitstream 1 and bitstream 2 shown in FIG. 3A, which are generated by the encoder and transmitted to the decoder. Typically, the greater the amount of side information, the higher the reconstruction quality. However, a large amount of side information means a lower compression ratio. Therefore, one objective of the system described in FIG. 3A is to balance the reconstruction quality with the amount of side information carried in the bitstream.

[0188] In FIG. 3A, the component AE 105 includes samples of the quantized latent representation ŷ and side information z ^ y is an arithmetic coding module that converts y into a binary representation bitstream 1. ^ and z ^ The samples may comprise, for example, integers or floating-point numbers. One purpose of the arithmetic coding module is to convert the sample values ​​(via a binarization process) into a string of binary numbers (which are then included in a bitstream that may contain other parts corresponding to the coded image or other side information).

[0189] Arithmetic decoding (AD) 106 is the process of reversing the binary processing, where binary numbers are converted into sample values. Arithmetic decoding is provided by an arithmetic decoding module 106.

[0190] It should be noted that the present disclosure is not limited to this particular framework. Furthermore, the present disclosure is not limited to image or video compression, but can be applied to object detection, image generation, and recognition systems as well.

[0191] In FIG. 3A, there are two interconnected subnetworks. A subnetwork in this context is a logical division between parts of the overall network. For example, in FIG. 3A, modules 101, 102, 104, 105, and 106 are called the "encoder / decoder" subnetwork. The "encoder / decoder" subnetwork is responsible for encoding (generating) and decoding (analyzing) a first bitstream, "Bitstream 1." The second network in FIG. 3A includes modules 103, 108, 109, 110, and 107 and is called the "hyper-encoder / decoder" subnetwork. The second subnetwork is responsible for generating a second bitstream, "Bitstream 2." The two subnetworks have different purposes.

[0192] The first subnetwork is responsible for: Transformation 101 of an input image x into a latent representation y (this is easier than compressing x), The latent representation y is converted into the quantized latent representation y ^ quantizing to, Quantized latent representation y ^ Compressing the above-mentioned data using AE by the arithmetic coding module 105 to obtain a bitstream "Bitstream 1"; Parsing the bitstream 1 by AD using an arithmetic decoding module 106; and The analyzed data was used to reconstruct 104 images (x ^ ) to be reconstructed.

[0193] The purpose of the second sub-network is to determine statistical properties of the samples of "Bitstream 1" (e.g., mean, variance, and correlation between samples of Bitstream 1), so that the first sub-network can compress Bitstream 1 more efficiently. The second sub-network generates a second bitstream, "Bitstream 2," that includes said information (e.g., mean, variance, and correlation between samples of Bitstream 1).

[0194] The second network generates the quantized latent representation y ^ into side information z 103 and converting the side information z into quantized side information z ^ and the quantized side information z ^ The second network includes an encoding part that encodes (e.g., binarizes) 109 the input bitstream 2 into the decoded quantized side information z ^ '. ^ ',z ^ Since the decoding operation on the arithmetic coding side is a lossless compression method, the decoded quantized side information z ^ ' is then decoded side information y ^ ' is converted to 107.y ^ ' is y ^ represents the statistical properties of y (e.g., the sample mean of y^ or the variance of the sample values). ^ ' is then provided to the arithmetic encoder 105 and arithmetic decoder 106 described above, and y ^ Control the probability model of

[0195] FIG. 3A shows an example of a variational auto-encoder (VAE), the details of which may vary in different implementations. For example, in a particular implementation, additional components may be present to more efficiently obtain statistical characteristics of the samples in bitstream 1. In one such implementation, there may be a context modeler whose goal is to extract cross-correlation information for bitstream 1. The statistical information provided by the second sub-network may be used by the arithmetic encoder (AE) 105 and arithmetic decoder (AD) 106 components.

[0196] Figure 3A depicts the encoder and decoder in a single diagram, and as will be apparent to those skilled in the art, the encoder and decoder may, and very often are, integrated into different devices.

[0197] FIG. 3B shows the encoder, and FIG. 3C shows the decoder component of the VAE framework in isolation. According to some embodiments, the encoder receives a picture as input. The input picture may include one or more channels, such as color channels or other types of channels, e.g., depth channels or motion information channels. The outputs of the encoder (as shown in FIG. 3B) are Bitstream 1 and Bitstream 2. Bitstream 1 is the output of the first sub-network of the encoder, and Bitstream 2 is the output of the second sub-network of the encoder.

[0198] Similarly, in Figure 3C, two bitstreams, Bitstream 1 and Bitstream 2, are received as inputs, and a reconstructed (decoded) image, ẑ, is produced at the output. As noted above, a VAE can be divided into different logical units that perform different operations. This is illustrated in Figures 3B and 3C, such that Figure 3B depicts components involved in encoding a video-like signal and provides coded information. This coded information is then received, for example, by a decoder component depicted in Figure 3C for decoding. Note that the encoder and decoder components designated 12x and 14x may correspond in function to the components described above in Figure 3A and designated 10x.

[0199] 3B, the encoder includes an encoder 121 that converts an input x into a signal y, which is then fed to a quantizer 322. The quantizer 122 provides information to an arithmetic coding module 125 and a hyper-encoder 123. The hyper-encoder 123 provides the bitstream 2 already described above to a hyper-decoder 147, which in turn provides information 125 to the arithmetic coding module 105.

[0200] The output of the arithmetic coding module is Bitstream 1. Bitstream 1 and Bitstream 2 are the output of the signal encoding and are subsequently provided (transmitted) to the decoding process. Although unit 101 (121) is referred to as an encoder, the complete subnetwork shown in FIG. 3B can also be referred to as an "encoder." An encoder generally refers to a unit (module) that converts an input into a coded (e.g., compressed) output. As can be seen from FIG. 3B, unit 121 can actually be considered the core of the entire subnetwork, since it converts the input x into a compressed version of x, y. Compression in encoder 121 can be achieved, for example, by applying a neural network, or in general, any processing network with one or more layers. In such a network, compression can be performed by cascading processes, including downsampling, to reduce the size of the input and / or the number of channels. Therefore, the encoder may be referred to, for example, as a neural network (NN)-based encoder.

[0201] The remaining parts in the figure (quantization unit, hyper-encoder, hyper-decoder, arithmetic encoder / decoder) are all parts that either improve the efficiency of the encoding process or convert the compressed output y into a series of bits (bitstream). Quantization may be performed to further compress the output of the NN encoder 121 using lossy compression. The AE 125, combined with the hyper-decoder 127 and hyper-encoder 123 used to form the AE 125, may also perform binarization to further compress the quantized signal using lossless compression. Therefore, the entire subnetwork in Figure 3B can also be referred to as an "encoder."

[0202] Most deep learning (DL)-based image / video compression systems reduce the dimensionality of a signal before converting it to binary digits (bits). For example, in a VAE framework, an encoder, which is a nonlinear transform, maps an input image x to y, where y has a smaller width and height than x. Because y has a smaller width and height, and thus a smaller size, the dimensionality of the signal is reduced, making it easier to compress signal y. Note that, in general, an encoder does not necessarily need to reduce the size in both (or generally all) dimensions. Rather, some exemplary implementations may provide an encoder that reduces the size in only one dimension (or generally a subset thereof).

[0203] J. Balle, L. Valero Laparra, and EP Simoncelli (2015). “Density Modeling of Images Using a Generalized Normalization Transformation”, In: arXiv e-prints, Presented at the 4th Int. Conf. for Learning Representations, 2016 (referred to in the following as “Balle”) In the paper, we proposed a framework for end-to-end optimization of image compression models based on nonlinear transforms. We optimize with respect to mean squared error (MSE), but use a more flexible transform constructed from a cascade of linear convolutions and nonlinearities. Specifically, we use generalized divisive normalization (GDN) joint nonlinearities inspired by models of neurons in the biological visual system and prove effective for Gaussianizing image densities. This cascade transform is followed by uniform scalar quantization (i.e., each element is rounded to the nearest integer), which effectively implements a parametric form of vector quantization in the original image space. The compressed image is reconstructed from these quantized values ​​using an approximate parametric nonlinear inverse transform.

[0204] Such an example of a VAE framework is shown in Figure 4, which utilizes six down-sampling layers marked 401 through 406. The network architecture includes a hyperprior model. a ,g s ) shows the image autoencoder architecture, and the right side (h a ,h s ) corresponds to an autoencoder that implements hyperpriors. The decomposed prior model is a representation of the analysis and synthesis transformation g a and g s The same architecture is used for the arithmetic encoder and decoder. Q stands for quantization, and AE and AD stand for arithmetic encoder and decoder, respectively. The encoder converts the input image x into g a to generate a response y (latent representation) with a spatially varying standard deviation. a contains multiple convolutional layers with subsampling and generalized decomposition normalization (GDN) as the activation function.

[0205] The response is h a, which summarizes the distribution of the standard deviation of z. Then z is quantized, compressed, and transmitted as side information. The encoder then ^ Using σ ^ and estimate the spatial distribution of standard deviations, which is used to obtain probability (or frequency) values ​​for arithmetic coding (AE), and use it to generate the quantized image representation y ^ (or latent representation) is compressed and transmitted. The decoder first extracts z from the compressed signal. ^ Then, h s Using y ^ Get h s was successful ^ Then, y ^ g s to obtain the reconstructed image.

[0206] Layers that include downsampling are indicated with a downward arrow in the layer description. The layer description "Conv N,k1,2↓" means that the layer is a convolutional layer, has N channels, and the convolution kernel is of size k1 x k1. For example, k1 may be equal to 5 and k2 may be equal to 3. As mentioned above, 2↓ means that downsampling by a factor of 2 is performed at this layer. Downsampling by a factor of 2 results in one of the dimensions of the input signal being reduced by half at the output. In Figure 4, 2↓ indicates that both the width and height of the input image are reduced by a factor of 2. Since there are six downsampling layers, if the width and height of the input image 414 (denoted by x) are given by w and h, then the output signal z ^413 has a width and height equal to w / 64 and h / 64, respectively. The modules denoted AE and AD are the arithmetic encoder and arithmetic decoder, as described in connection with Figures 3A-3C. The arithmetic encoder and decoder are specific implementations of entropy coding. AE and AD can be replaced by other means of entropy coding. In information theory, entropy coding is a lossless data compression method used to convert symbol values ​​into binary representations, a reversible process. Also, "Q" in the figure corresponds to the quantization operation mentioned above in connection with Figure 4 and further described above in the "Quantization" section. Also, the quantization operation and corresponding quantization unit as part of component 413 or 415 need not necessarily be present and / or can be replaced by other units.

[0207] Also shown in FIG. 4 is a decoder including upsampling layers 407-412. Another layer 420 is provided between upsampling layers 411 and 410 in the input processing order, which is implemented as a convolutional layer but does not provide upsampling of the received input. A corresponding convolutional layer 430 is also shown for the decoder. Such a layer can be provided in the neural network to perform operations on the input that change certain characteristics without changing the size of the input. However, such a layer is not required.

[0208] In terms of the processing order of bitstream 2 through the decoder, the upsampling layers are performed in reverse order, from upsampling layer 412 to upsampling layer 407. Each upsampling layer is shown here providing upsampling with an upsampling ratio of 2, denoted by ↑. Of course, it is not necessary for all upsampling layers to have the same upsampling ratio; other upsampling ratios, such as 3, 4, or 8, may be used. Layers 407 through 412 are implemented as convolutional layers (conv). Specifically, an upsampling layer can be designed to provide an operation on the input that is the inverse of that of the encoder, so that it applies a deconvolution operation to the received input, thereby increasing its size by a factor corresponding to the upsampling ratio. However, the present disclosure is not generally limited to deconvolution; upsampling may be performed in some other way, such as by bilinear interpolation between two adjacent samples or by nearest-neighbor sample copying.

[0209] In the first sub-network, several convolutional layers (401-403) are followed by a Generalized Divided Normalization (GDN) on the encoder side and an Inverse GDN (IGDN) on the decoder side. In the second sub-network, the activation function applied is ReLU. It should be noted that the present disclosure is not limited to such implementations and that in general, other activation functions may be used instead of GDN or ReLU.

[0210] Cloud Solutions for Machine Tasks Machine-oriented video coding (VCM) is another popular direction in computer science today. The main idea behind this approach is to transmit coded representations of image or video information for further processing by computer vision (CV) algorithms, such as object segmentation, detection, and recognition. In contrast to traditional image and video coding aimed at human perception, the quality characteristic is not the quality of the reconstruction, but rather the performance of a computer vision task, such as object detection accuracy. This is illustrated in Figure 5.

[0211] Machine-directed video coding, also known as collaborative intelligence, is a relatively new paradigm for efficiently deploying deep neural networks on mobile cloud infrastructures. By dividing the network between the mobile side 510 and the cloud side 590 (e.g., cloud servers), it is possible to distribute the computational load so that the overall energy and / or latency of the system is minimized. In general, collaborative intelligence is a paradigm in which neural network processing is distributed among two or more distinct computing nodes—e.g., devices, but generally, any functionally defined nodes. Here, the term "node" does not refer to the neural network nodes described above. Rather, a (computing) node here refers to a separate (physically, or at least logically) device / module that implements part of a neural network. Such devices may be different servers, different end-user devices, a mixture of servers and / or user devices and / or clouds and / or processors, etc. In other words, computational nodes can be thought of as nodes that belong to the same neural network and communicate with each other, carrying data coded within or for the neural network. For example, to perform complex computations, one or more layers may execute on a first device (such as a device on the mobile side 510) and one or more layers may execute on another device (such as a cloud server on the cloud side 590). However, the distribution may be more granular, and a single layer may execute on multiple devices. In this disclosure, the term "multiple" refers to two or more. In existing solutions, part of the neural network function executes on a device (such as a user device or edge device) or multiple such devices, and then the output (feature map) is passed to the cloud. The cloud is a collection of processing or computational systems located external to the device that runs part of the neural network. The concept of collaborative intelligence also extends to model training.In this case, data flows bidirectionally: from cloud to mobile during backpropagation during training, and from mobile to cloud during forward propagation during training as well as inference (as shown in Figure 5).

[0212] Several studies have presented semantic image compression by encoding deep features and then reconstructing the input image from them. Compression based on uniform quantization has been demonstrated, followed by context-based adaptive arithmetic coding (CABAC) in H.264. In some scenarios, transmitting the output of the hidden layer (deep feature map) 550 from the mobile side 510 to the cloud 590 may be more efficient than transmitting compressed natural image data to the cloud and performing object detection using the reconstructed image. Therefore, it may be advantageous to compress the data (features) generated by the mobile side 510, which may include a quantization layer 520 for this purpose. Correspondingly, the cloud side 590 may include an inverse quantization layer 560. Efficient compression of feature maps is beneficial for image and video compression and reconstruction, both for human perception and machine vision. Entropy coding methods, such as arithmetic coding, are common approaches to compressing deep features (i.e., feature maps).

[0213] Currently, video content contributes to over 80% of Internet traffic, and this proportion is expected to continue to increase. Therefore, it is important to build efficient video compression systems that can produce higher-quality frames within a given bandwidth budget. Furthermore, most video-related computer vision tasks, such as video object detection and video object tracking, are sensitive to the quality of the compressed video. Therefore, efficient video compression can benefit other computer vision tasks. Meanwhile, video compression techniques can also be useful for action recognition and model compression. However, for the past few decades, video compression algorithms have relied on handcrafted modules, such as block-based motion estimation and discrete cosine transform (DCT), to reduce redundancy in video sequences, as mentioned above. While each module is well-designed, the entire compression system has not been optimized end-to-end. It is desirable to further improve video compression performance by jointly optimizing the entire compression system.

[0214] End-to-end image or video compression DNN-based image compression methods can utilize large-scale end-to-end training and highly nonlinear transformations not used in traditional approaches. However, directly applying these techniques to build end-to-end learning systems for video compression is not trivial. First, learning how to generate and compress motion information tailored for video compression remains an open problem. Video compression methods rely heavily on motion information to reduce temporal redundancy in video sequences.

[0215] A straightforward solution is to represent motion information using learning-based optical flow. However, current learning-based optical flow approaches aim to generate flow fields that are as accurate as possible. Accurate optical flow is often suboptimal for a particular video task. Furthermore, the amount of optical flow data increases significantly when compared to motion information in traditional compression systems, and directly applying existing compression approaches to compress optical flow values ​​would significantly increase the number of bits required to store the motion information. Second, it is unclear how to build a DNN-based video compression system by minimizing a rate-distortion-based objective for both residual and motion information. Rate-distortion optimization (RDO) aims to achieve higher quality (i.e., less distortion) of reconstructed frames for a given number of bits (or bit rate) for compression. RDO is critical for video compression performance. To take advantage of the power of end-to-end training for learning-based compression systems, an RDO strategy that optimizes the entire system is required.

[0216] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, Zhiyong Gao; “DVC: ​​An End-to-end Deep Video Compression Framework”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11006-11015 In this paper, the authors proposed an end-to-end deep video compression (DVC) model that jointly trains motion estimation, motion compression, and residual coding.

[0217] Such an encoder is shown in Figure 6. In particular, Figure 6 shows the overall structure of an end-to-end trainable video compression framework. To compress motion information, the CNN uses the optical flow v t to the corresponding representation mt, which is suitable for better compression.

[0218] Transformer In recent years, Transformers have attracted increasing attention in both the fields of language processing (e.g., text translation) and image processing. Video coding can be facilitated by the use of Transformer-based neural networks. Transformers can be used for image enhancement and classification purposes. Transformers do not involve recurrent or convolutional neural networks and rely on self-attention. Transformers may be implemented, for example, in the configuration shown in Figure 6. In particular, Transformers can be combined with recurrent or convolutional neural networks.

[0219] FIG. 7 illustrates an example of a Transformer 700. The Transformer 700 includes a neural layer 810 (Transformer layer). The Transformer 700 can include an encoder-decoder architecture including an encoder neural layer and a decoder neural layer. Alternatively, the Transformer 700 can include only an encoder stack of neural layers. Input data is input to the Transformer, and output data is output. For example, the Transformer 700 may be configured for image processing and output an enhanced image. The input data may be, for example, patches of an image or words of a sentence. For example, a tokenizer generates tokens in the form of words from the sentence being processed or patches from the image. These tokens can be converted into embeddings (continuous values) by some embedding algorithm. According to the example shown in FIG. 7, a linear projection layer 720 converts the input patches into tensor representations (embeddings) of the portions of the object being processed (in latent space). These tensor representations of the signal input are processed by the Transformer 700. A position coding layer 730 is provided to provide information about the relative positions of parts of the object (e.g., image or text) being processed, e.g., the relative positions of patches of an image or words of a sentence with respect to each other. A sinusoidal function may be used for the position coding.

[0220] The last one of the neural layers 710 outputs an output data tensor in latent space that is transformed back to object space (e.g., image or text space) by a linear backprojection layer 740.

[0221] Processing by the neural layer (transformer layer) 710 is based on the concept of self-attention. Each neural layer 710 in the transformer 700 includes a multi-head self-attention layer and a (fully connected) feed-forward neural network. The self-attention layer helps the encoder stack look at other parts of the object (e.g., image patches or words) when it encodes a particular part of the object (e.g., an image patch or a word in a sentence). The output of the self-attention layer is fed into a feed-forward neural network. The decoder stack also contains both of these components, and an additional "encoder-decoder" attention layer between them that helps the decoder stack focus on the relevant part of the input data. Each part of the object (e.g., an image patch or a word in a sentence) processed at each location flows through its own path within the encoder. Dependencies exist between these paths in the self-attention layer. However, the feed-forward layer does not have these dependencies, and therefore various paths can run in parallel while flowing through the feed-forward layer.

[0222] The multi-head self-attention layer of the encoder stack takes a query Q, a key K, a value V tensor, and a self-attention

[0223]

number

[0224] Each sub-layer (self-attention layer and feed-forward neural network) in each encoder and decoder layer has residual connections in its neighborhood, followed by a normalization layer.

[0225] The output of the top encoder layer is converted into a set of attention vectors K and V, which are used by each decoder layer in its encoder-decoder-attention layer. The encoder-decoder-attention layer operates similarly to the multi-head self-attention layer in the encoder stack, except that it creates a query matrix from each layer below it and obtains key and value matrices from the output of the encoder stack.

[0226] The decoder stack outputs a vector of floating-point numbers that are converted to object parts (e.g., patches in an image or words in a sentence) by a final linear layer (a fully connected neural network) that outputs logits, followed by a Softmax layer that yields the highest probability output.

[0227] As mentioned above, the use of neural networks (e.g., CNNs) is becoming increasingly important in the field of image coding systems, i.e., in data compression systems in general, e.g., in the context of neural network-based compression of images, video, audio, 3D-PCC, etc.

[0228] In reality, neural networks in real applications are implemented on various platforms / devices that differ from each other in terms of their numerical computing architecture. For example, one device may be a CPU or contain a CPU, while another device that communicates data with the CPU or contains a CPU may be a GPU or contain a GPU. Different platforms (e.g., one containing a CPU and another containing a GPU) usually handle delicate integer and real number situations differently, especially for floating-point arithmetic, but also for fixed-point arithmetic. One class of delicate real number situations is overflow of the registers / memory used. There is no standardized handling of overflow situations, and it is difficult to imagine a commonly followed procedure in this regard.

[0229] Integer overflow occurs when an arithmetic operation attempts to create a number outside the range that can be represented by a given number of digits—either larger than the largest representable value or smaller than the smallest representable value. This situation may be handled differently by different compilers, devices (CPUs, GPUs), etc. To achieve bit-exact results for integer operations on different platforms, integer overflow should be avoided. In this context, it should be noted that obtaining bit-exact results on different platforms is not important for all applications. However, it is for some applications. For example, as described above with reference to Figures 2 through 4, in an autoencoding context, such as a variational autoencoder, an entropy model for lossless data compression is used on the encoder side and the decoder / receiver side to model the symbol probabilities for the arithmetic encoder and decoder, respectively. The encoder side and the decoder / receiver side may use different types of platforms. For reliable reconstruction of the encoded image, it is important that the entropy model is applied in the same way on both sides. In fact, even a small deviation between the entropy model used at the decoder / receiver side and the one used at the encoder side can lead to a complete breakdown of the image decoding / reconstruction process due to incorrect symbol interpretation. For example, in the configuration shown in Figure 4, the entropy model σ ^ must be provided by hyperpriors in the arithmetic encoder AE on the encoder side and in the arithmetic decoder AD on the decoder side. Bit-exact application of the same entropy model is required for reliable image reconstruction. The problem of incorrect symbol interpretation due to the entropy model used on the decoder / receiver side deviating from that used on the encoder side arises not only for image coding systems, but also for all compression systems that use arithmetic coding and neural networks in the entropy part.

[0230] With regard to real arithmetic, a distinction must be made between fixed-point and floating-point arithmetic. Fixed-point arithmetic is arithmetic that represents fractional numbers (non-integers) by storing a fixed number of decimal places. In general, it is integer arithmetic; to convert such fixed-point values ​​to real ones, division by a scaling factor corresponding to the number of bits used to store the decimal place is used. Floating-point arithmetic is of the form: mantissa*base 指数 arithmetic that uses a representation of real numbers in , where the mantissa is an integer, the base is an integer greater than or equal to 2, and the exponent is also an integer. The base is fixed, and the mantissa and exponent pair represent the number. When compared to fixed-point arithmetic, using a non-fixed exponent allows a trade-off to be achieved between dynamic range and precision. Essentially, for both fixed-point and floating-point arithmetic, real numbers have the following form: mantissa*base 指数 The only difference is that in fixed-point arithmetic, both the base and the exponent are fixed, while in floating-point arithmetic, only the base is fixed and the exponent is part of the number's representation. Therefore, in fixed-point arithmetic, the precision of the result (the exponent) does not depend on the precision of the arguments, which means, for example, that the result of an addition does not depend on the order of addition (assuming no overflow occurs, the operation is associative, just like regular integer addition). In floating-point arithmetic, on the other hand, the exponent is part of the number's representation and is calculated during the arithmetic operation based on the exponents of the arguments. This leads to non-associative addition; for example, if many very small numbers are added one by one to a large number, the result will equal the large number because the contribution of the relatively small numbers will be lost after rounding with the exponent of the large number. Conversely, if many small numbers are added together first, the contribution of each addition is not lost by rounding, and the resulting sum can be relatively large and will not be completely lost after rounding with the exponent of the large number. Thus, fixed-point addition is associative unless there is overflow, while floating-point arithmetic is, in principle, non-associative, i.e., in general: (A+B)+C≠A+(B+C) Because the order of additions may differ between the encoder and decoder, or may not even be completely predetermined, this creates problems for achieving identical results on both sides. Therefore, fixed-point arithmetic may be preferable to floating-point arithmetic. In the context of entropy-coding-based data compression to ensure proper decoding across different platforms, especially in systems with massive parallelism, it may be preferable to avoid the use of floating-point arithmetic and use fixed-point arithmetic instead, at least for the entropy part of the image coding net. The potential problem of different floating-point implementations on different devices is resolved in this case, because fixed-point (integer) arithmetic is much more portable. However, simply restricting oneself to fixed-point arithmetic does not prevent some overflows. To guarantee bit-exact behavior across different platforms, it is not enough to use fixed-point arithmetic; it is important to ensure the absence of integer overflows.

[0231] More specifically, consider a neural network layer as shown in Figure 8. The neural network layer may be included in, for example, a convolutional neural network or a fully connected neural network. i are input in various input channels, and weights w ij (where index i is the in indicates an input channel among the input channels, and the index j is out (denotes the jth channel output). The jth channel output is obtained according to:

[0232]

number

[0233] The (usually trainable) bias D is ignored below for simplicity. For example, for a convolutional neural network layer, the output of the jth channel is given by the convolution operator * as follows:

[0234]

number

[0235] The neural network's accumulator registers used for buffering addition have a predefined size, i.e., some accumulator register bit depth (size) n (e.g., n = 16 bits or 32 bits). To avoid integer overflow, the following conditions must be met:

[0236]

number

[0237] A) Clipping and Scaling According to an embodiment, the input data x to be processed by a neural network layer of a neural network (e.g., a convolutional neural network) is represented by the fixed-point arithmetic used. i and the real-valued weights w of the neural network layers ij is an integer x i ~ and integer weight w ij ~, respectively. The integerized version of Equation 2 reads as follows:

[0238]

number

[0239] To avoid the performance loss caused by this conversion to fixed-point arithmetic,

[0240]

number

[0241]

number

[0242]

number

[0243]

number

[0244] Thus, according to an embodiment, the following units are added to a neural network (inference pipeline): a unit configured to scale, round, and clip the input data values ​​before inputting them to the neural network layers, and a unit that divides the output of the neural network layers by a scaling factor. The scaling and / or rounding units may be omitted (or may not be operational) if it is guaranteed that only integer-valued data (data containing only integers) should be processed. Units may also be added to neural networks used for data compression / coding, in particular (variational) autoencoder or transformer architectures as mentioned above.

[0245] An exemplary embodiment is shown in FIG. 9. For illustrative purposes, FIG. 9 illustrates neural network layers of a convolutional neural network. The neural network includes a stack of convolutional layers followed by an activation function, in this case, a LeackyReLU function. The black arrows in FIG. 9 indicate the inputs of new units added to a conventional neural network. Prior to the convolutional layers, the input values ​​of real-valued input data are multiplied by a scaling factor, and the result of the multiplication process is rounded to the nearest integer. If the input data is already integer-valued, the scaling and / or rounding process can be omitted. If the resulting integer value is less than a predefined lower integer threshold -A, the integer value is clipped to the predefined lower integer threshold -A. If the integer value is greater than a predefined upper integer threshold B, the integer value is clipped to the predefined upper integer threshold B. The resulting input values ​​are then processed by the convolutional neural network layers, and the outputs of these layers are input. According to an embodiment, the output of the activation function is divided by the scaling factor applied to the input data. According to another embodiment, the output of the convolutional neural network layer is divided by a scaling factor before being fed to the activation function. According to a further embodiment, each scaling factor is decomposed into two parts, and the division by the first part can be performed before the activation function and the division by the second part can be performed after the activation function.

[0246] Neural networks such as the one shown in Figure 9, which include units for scaling, rounding, clipping, and for dividing the outputs of neural network layers by scaling factors (before and / or after the respective activation functions), can be implemented on both the encoder and decoder sides of a coding system, e.g., in the encoder that encodes images and the decoder that decodes the encoded images, and can be used, for example, by the encoder / decoder included in the configurations shown in Figures 2 to 4, 6, and 7.

[0247] B) Weight The condition in Equation 5 can be transformed into another condition for the weights of the neural network layers. Here, i and weight w ij Assume that x, y, z ...

[0248]

number

[0249] If the condition of Equation 7 is satisfied, the inference including clipping of the input data to the lower and upper thresholds respectively will not cause any overflow of the accumulator register of the neural network.If the model weights in a data compression system are allowed to be defined by the user, the condition of Equation 7 must be satisfied if bit-exact behavior of the neural network at the encoder and decoder sides needs to be guaranteed.

[0250] According to the embodiment, the integer lower threshold -A is -2 k-1 and the upper integer threshold B is given by 2 k-1 −1, where k denotes the predefined bit depth of the layer's input data.

[0251] In this case, an easy-to-check condition on the weights can be given as follows:

[0252]

number

[0253] This form of the weight condition is particularly easy to compute for user-defined weights to ensure that overflow never occurs during inference. The conditions in Equations 8 and 9 must be satisfied for each output j of a neural network layer.

[0254] The bias D could in principle be taken to be zero, in which case equations 8 and 9 would be more simply checked.

[0255] In particular, for one-dimensional convolutional neural network layers, addition

[0256]

number

[0257]

number

[0258] For a 2D convolutional neural network layer, addition

[0259]

number

[0260]

number

[0261] where C in where denotes the number of input channels of the neural network layer, K1 and K2 denote the convolution kernel sizes, and j denotes the index of the output channel of the neural network layer. Correspondingly, for an N-dimensional convolutional neural network layer, the similar summation can be calculated as:

[0262]

number

[0263] C) Scaling factors for weights A larger scaling factor for the weights, s j (More precisely, 2 sj (see above) will result in a lower performance loss due to the conversion to fixed-point arithmetic than a smaller one (see above). The largest possible value that guarantees that the condition of Equation 8 or 9 is satisfied is the scaling factor s j If we assume that

[0264]

number

[0265]

number

[0266] According to another implementation, the second scaling factor s for the jth output channel of at least one neural network layer is j is given by:

[0267]

number

[0268] Both conditions can advantageously guarantee that integer overflows will not occur for the accumulator register.

[0269] Another implementation would be:

[0270]

number

[0271]

number

[0272]

number

[0273] In these conditions, n denotes the bit size of the accumulator register, and w ij denotes the real-valued weight, k denotes the predefined bit depth of the input data, and b j indicates the bias value (which may be zero).

[0274] Both conditions can advantageously guarantee that integer overflows will not occur for the accumulator register.

[0275] To convert the entire neural network pipeline to fixed-point arithmetic, the parameters p and k should be known. These parameters can be calculated, for example, by all possible combinations of (p, k), 0 ≤ p ≤ k ≤ n, and the corresponding s obtained using equation (10) for a chosen k and a given n in some calibration dataset. jand k. The minimum value of a predefined loss function corresponds to the best pair (p, k). The loss function can represent, for example, the number of bits required to encode some image or to estimate its likelihood. It is also possible to use distortion as part of the loss function used if not only the entropy part but also the analysis and synthesis parts (see, for example, the left side of the configuration shown in Figure 4) should be converted to fixed-point arithmetic.

[0276] D) activation function To ensure device interoperability across neural networks, bit-exact reproducibility of activation functions across different platforms / devices is desirable. For example, this requirement can be met relatively easily for linear activation functions and relatively simple nonlinear activation functions, such as the ReLU function, which essentially defines a clipping procedure. For more complex nonlinear ones, especially those involving exponential functions (which may use the basis e, but other bases may also be included), such as Softmax:

[0277]

number

[0278] According to an embodiment, the mathematically defined non-linear activation function is replaced by an approximation function selected from the group consisting of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function. The mathematically defined non-linear activation function to be replaced may be a Softmax function, a Sigmoid function, a Hyperbolic Tangent function, a Swish function, a Gaussian Error Linear Unit function, or a Scaled Exponential Linear Unit function.

[0279] The approximation functions can be used, for example, by encoders / decoders in the configurations shown in Figures 2 to 4, 6 and 7, and generally in data compression / coding contexts.

[0280] In general, for any nonlinear activation function, it is possible to use the sum of the first few elements of its Taylor series. The Taylor series of a mathematically defined function f(x) around a predefined value a is defined as follows:

[0281]

number

[0282]

number

[0283]

number

[0284] It should be noted that, depending on the actual application, approximation of mathematically defined non-linear activation functions by rational or polynomial functions may be considered particularly appropriate.

[0285] In particular, according to an embodiment, a Softmax that operates on the i component of the vector x is

[0286]

number

[0287]

number

[0288]

number

[0289]

number

[0290] In particular, an embodiment is provided that relates to a method 1000 for operating a neural network capable of avoiding integer overflow. The neural network includes a neural network layer including or connected to an accumulator register for buffering addition results and having a predefined accumulator register size. The method 1000 includes a step S1010 of defining lower and upper integer thresholds for integer values ​​included in data entities (e.g., numbers, vectors, or tensors) of input data of the neural network layer. The method also includes a step S1020 of clipping the integer values ​​included in the data entities of the input data to the predefined lower integer threshold if the integer values ​​included in the data entities of the input data are smaller than the predefined lower integer threshold, and clipping the integer values ​​included in the data entities of the input data to the predefined upper integer threshold if the integer values ​​included in the data entities of the input data are greater than the predefined upper integer threshold, thereby avoiding integer overflow of the accumulator register. When performing the same procedure with the same input data on two devices / platforms, the method 1000 as shown in FIG. 10 allows obtaining the same results for both devices / platforms because the clipping procedure can be used to avoid integer overflow.

[0291] The method shown in FIG. 10 can be implemented in any type of neural network, for example, a neural network including one or more fully connected neural network layers or one or more convolutional neural network layers. In particular, a neural network 1100 is provided as shown in FIG. 11. The neural network 1100 can utilize the method 1000. The neural network 1100 includes a unit 1110 for scaling, rounding, and clipping. The unit 1110 may be divided into sub-units configured for different operations, such as scaling, rounding, and clipping of input data, and the on and off operation of the first unit and each sub-unit may be respectively switchable. The unit 1110 may be configured to receive real-valued input data (data including real numbers) and convert the input data to integer-valued input data by multiplying the input data value by a scaling factor and rounding the multiplication result to the next integer as described above. The resulting integer value is clipped to lower and upper integer thresholds as needed.

[0292] The integerized and clipped data is input to the neural network layer 1120. A de-scaling unit 1130 is provided downstream of the neural network layer 1120. The de-scaling unit 1130 can divide the output of the neural network layer 1120 by a scaling factor. The output of the de-scaling unit 1130 can be input to the activation function 1140. Alternatively, the output of the neural network layer 1120 can be directly input to the activation function 1140, and the de-scaling unit 1130 is provided downstream of the activation function 1140 and divides its output. Alternatively, the de-scaling by the de-scaling unit 1130 is performed partially on the output of the neural network layer 1120 (by a first decomposition part of the scaling factor) and subsequently partially on the output of the activation function 1140 (by a second decomposition part of the scaling factor).

[0293] In particular, an embodiment is provided relating to a method 1200 for operating a neural network based on adjusted weights as shown in FIG. 12 . The neural network includes a neural network layer including or connected to an accumulator register for buffering summation results and having a predefined accumulator register size. The method 1200 includes defining S1210 a lower integer threshold A and an upper integer threshold B for integer values ​​included in data entities (e.g., numbers, vectors, or tensors) of input data for the neural network layer. The method also includes S1220 clipping the integer values ​​included in the data entities of the input data to the predefined lower integer threshold if the integer values ​​included in the data entities of the input data are less than the predefined lower integer threshold, or clipping S1220 the integer values ​​included in the data entities of the input data to the predefined upper integer threshold if the integer values ​​included in the data entities of the input data are greater than the predefined upper integer threshold. Further, the method includes a step S1230 of determining integer-valued weights (i.e., weights that include integers, e.g., only integers) of the neural network layers based on a defined integer lower threshold, a defined integer upper threshold, and a predefined accumulator register size, so as to avoid integer overflow of the accumulator register.

[0294] The method 1200 as shown in Figure 12 allows for obtaining the same results for both devices / platforms when performing the same procedure with the same input data on both devices / platforms. Particularly in the context of entropy model-based coding and / or compression and decompression of data such as image data, it is an important issue to provide substantially bit-exact processing results on the encoding and decoding sides, respectively, to provide the same or complementary technical effects obtainable by the method 1200 as shown in Figure 12.

[0295] In particular, according to an embodiment, a method 1300 for operating a neural network is provided (see FIG. 13). The method 1300 comprises a step 1310 of implementing an approximation function of a mathematically defined real-valued nonlinear activation function as an activation function of at least one neural network layer, the approximation function allowing for processing only integers in a fixed-point representation of the input values ​​of the approximation function. The approximation function may comprise at least one of a polynomial function, a rational function, a finite Taylor series, a ReLU function, a LeakyReLU function, and a parametric ReLU function. The mathematically defined nonlinear activation function may be selected from the group consisting of a Softmax function, a sigmoid function, a hyperbolic tangent function, a swish function, a Gaussian error linear unit function, and a scaling exponential linear unit function. In particular, the approximation activation function may be any of the following:

[0296]

number

[0297] The provision of the above approximation activation functions allows bit-exact reproduction of important numerical operations at the encoder and decoder sides, since in particular the calculation of exponential functions can be avoided, so that the technical effects achieved by these numerical operations are identical or complementary to each other.

[0298] According to an embodiment, a neural network 1400 is provided as shown in Figure 14. The neural network 1400 includes at least one neural network layer 1410 and an activation function 1420 connected to the output of the at least one neural network layer 1410, where the activation function 1420 is implemented as an approximation function of a mathematically defined real-valued nonlinear activation function (one that takes real numbers as arguments and outputs real numbers), where the approximation function allows processing only integers in fixed-point representation of the input values ​​of the approximation function.

[0299] At least some of the steps of methods 1000, 1200, and 1300 described with reference to Figures 10, 12, and 13 may be included in, for example, a method for encoding or decoding at least part of data of an image. According to an embodiment, the encoding or decoding is based on an entropy model that provides statistical (probabilistic) properties of the symbols to be encoded or decoded, such as mean, variance, (cross-)correlation, etc. The entropy model may be provided by a hyperprior of a variational autoencoder, an autoregressive prior of a variational autoencoder, or a combination of a hyperprior and an autoregressive prior of a variational autoencoder. The use of methods 1000, 1200, and 13 described with reference to Figures 10, 12, and 13 for data coding, e.g., image coding, may prove advantageous for high-quality reconstruction of encoded / compressed data without incurring severe corruption.

[0300] The methods 1000, 1200, and 1300 described with reference to Figures 10, 12, and 13 may be implemented in an apparatus 1500 as shown in Figure 15, which may be configured to perform the steps of these methods. According to an embodiment, the apparatus 1500 includes processing circuitry configured to perform the steps of the methods 1000, 1200, and 1300 described with reference to Figures 10, 12, and 13. Furthermore, the apparatus 1500 may include at least one of the neural networks shown in Figures 9, 11, and 14. The apparatus 1500 may be configured by an encoder (e.g., the encoder 20 shown in Figures 16 and 17) or a decoder (e.g., the decoder 20 shown in Figures 16 and 17), or may be configured by the video coding device 8000 shown in Figure 18 or the apparatus 9000 shown in Figure 19.

[0301] The device 1500 may be, for example, a device for encoding or decoding data, such as at least a portion of an image. Furthermore, the device 1500 may include or be included within a (variational) autoencoder or a transformer, as described above.

[0302] Although operations are depicted in a particular order in the figures, this should not be understood as requiring such operations to be performed in the particular order or sequential order depicted, or that all of the operations depicted be performed, to achieve desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0303] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0304] Some Hardware and Software Implementations A corresponding system in which the above-described encoder / decoder processing chain can be deployed is shown in FIG. 16. FIG. 16 is a schematic block diagram illustrating an exemplary coding system, such as a video, image, audio, and / or other coding system (or coding system for short), that can utilize the techniques of the present application. The video encoder 20 (or encoder for short) and the video decoder 30 (or decoder for short) of the video coding system 10 represent examples of devices that can be configured to perform the techniques according to various examples described herein. For example, video encoding and decoding can use a neural network that can be distributed, applying the above-described bitstream analysis and / or bitstream generation to communicate feature maps between distributed computing nodes (two or more).

[0305] As shown in FIG. 16, coding system 10 includes a source device 12 configured to provide encoded picture data 21 to a destination device 14, which may, for example, decode encoded picture data 13.

[0306] The source device 12 includes an encoder 20 and may additionally, i.e. optionally, include a picture source 16, a pre-processing processor (or pre-processing unit) 18, such as a picture pre-processing unit 18, and a communication interface or communication unit 22.

[0307] Picture source 16 may include or be any kind of picture capture device, such as a camera capturing real-world pictures, and / or any kind of picture generation device, such as a computer graphics processor generating computer-animated pictures, or any kind of other device that acquires and / or provides real-world pictures, computer-generated pictures (e.g., screen content or virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). Picture source can be any kind of memory or storage that stores any of the above pictures.

[0308] To distinguish between the preprocessing processor 18 and the processing performed by the preprocessing unit 18, the picture or picture data 17 may also be referred to as an unprocessed picture or unprocessed picture data 17.

[0309] The pre-processing processor 18 is configured to receive (raw) picture data 17 and perform pre-processing on the picture data 17 to obtain a pre-processed picture 19 or pre-processed picture data 19. The pre-processing performed by the pre-processing processor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It may be understood that the pre-processing unit 18 may be an optional component. It should be noted that the pre-processing may also use a neural network (such as in any of FIGS. 1 to 7) that uses presence indicator signaling.

[0310] The video encoder 20 is configured to receive the pre-processed picture data 19 and to provide encoded picture data 21 .

[0311] The communication interface 22 of the source device 12 can be configured to receive the encoded picture data 21 and transmit the encoded picture data 21 (or any further processed version thereof) via the communication channel 13 to another device, for example to the destination device 14 or any other device, for storage or direct reconstruction.

[0312] The destination device 14 includes a decoder 30 (e.g., a video decoder 30) and may additionally, i.e., optionally, include a communications interface or communications unit 28, a post-processing processor 32 (or post-processing unit 32), and a display device 34.

[0313] The communications interface 28 of the destination device 14 is configured to receive the encoded picture data 21 (or any further processed version thereof), e.g., directly from the source device 12 or from any other source, e.g., from a source device, e.g., from a coded picture data storage device, and to provide the coded picture data 21 to a decoder 30.

[0314] The communication interface 22 and the communication interface 28 may be configured to transmit or receive the encoded picture data 21 or the encoded data 13 via a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or via any kind of network, such as a wired or wireless network or any combination thereof, or any kind of private and public network, or any combination thereof.

[0315] The communications interface 22 may be configured, for example, to package the encoded picture data 21 into a suitable format, such as packets, and / or to process the encoded picture data using any type of transmission coding or processing for transmission over a communications link or network.

[0316] The communications interface 28, which forms a counterpart of the communications interface 22, can be configured to receive transmitted data and process the transmitted data using any kind of corresponding transmission decoding or processing and / or unpackaging to obtain the encoded picture data 21, for example.

[0317] 16 by the arrows for communication channel 13 from source device 12 to destination device 14, or can be configured as bidirectional communication interfaces, e.g., to send and receive messages, e.g., to set up connections, confirm and exchange any other information related to the communication link and / or data transmission, e.g., encoded picture data transmission. Decoder 30 is configured to receive encoded picture data 21 and to output decoded picture data 31 or decoded pictures 31.

[0318] Post-processing processor 32 of destination device 14 is configured to post-process decoded picture data 31 (also called reconstructed picture data), e.g., decoded picture 31, to obtain post-processed picture data 33, e.g., post-processed picture 33. The post-processing performed by post-processing unit 32 may include, e.g., color format conversion (e.g., from YCbCr to RGB), color correction, cropping, resampling, or any other processing, e.g., to prepare decoded picture data 31 for display by display device 34.

[0319] Display device 34 of destination device 14 is configured to receive post-processed picture data 33 and display the picture, e.g., to a user or viewer. Display device 34 may be or include any type of display, e.g., an integrated or external display or monitor, that presents the reconstructed picture. The display may include, for example, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a microLED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0320] 16 depicts source device 12 and destination device 14 as separate devices, device embodiments may also include both or both functionality, source device 12 or corresponding functionality, and destination device 14 or corresponding functionality. In such embodiments, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0321] As will be clear to those skilled in the art based on the specification, the presence and (exact) division of functions or functions of various units in the source device 12 and / or destination device 14 shown in Figure 16 may vary depending on the actual device and application.

[0322] Encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30), or both encoder 20 and decoder 30, may be implemented by processing circuitry, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video coding, or any combination thereof. Encoder 20 may be implemented by processing circuitry 46 to implement various modules, including a neural network or portions thereof. Decoder 30 may be implemented by processing circuitry 46 to implement any coding system or subsystem described herein. The processing circuitry may be configured to perform various operations, as described below. If the techniques are implemented partially in software, a device may store software instructions on a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Either video encoder 20 or video decoder 30 may be integrated as part of a combined encoder / decoder (CODEC) within a single device, for example as shown in FIG.

[0323] Source device 12 and destination device 14 may include any of a wide range of devices, including any type of handheld or stationary device, such as a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or content delivery server), a broadcast receiving device, a broadcast transmitting device, etc., and may or may not use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0324] In some cases, the video coding system 10 shown in FIG. 16 is merely an example, and the techniques of the present application may be applied to video coding settings (e.g., video encoding or video decoding) that do not necessarily involve any data communication between the encoding device and the decoding device. In other examples, data may be retrieved from local memory, streamed over a network, etc. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data to memory and / or retrieve data from memory and decode it.

[0325] 18 is a schematic diagram of a video coding device 8000 according to an embodiment of the present disclosure. The video coding device 8000 is suitable for implementing the disclosed embodiments as described herein. In an embodiment, the video coding device 8000 may be a decoder, such as the video decoder 30 of FIG. 16, or an encoder, such as the video encoder 20 of FIG. 16.

[0326] The video coding device 8000 includes an ingress port 8010 and a receiver unit (Rx) 8020 for receiving data, a processor, logic unit, or central processing unit 8030 (CPU) for processing data, a transmitter unit (Tx) 8040 and an egress port 8050 (or output port 8050) for transmitting data, and a memory 8060 for storing data. The video coding device 8000 may also include optical-electrical (OE) and electro-optical (EO) components coupled to the ingress port 8010, the receiver unit 8020, the transmitter unit 8040, and the egress port 8050 for inputting and outputting optical or electrical signals.

[0327] The processor 8030 is implemented in hardware and software. The processor 8030 may be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 8030 communicates with the ingress port 8010, the receiver unit 8020, the transmitter unit 8040, the egress port 8050, and the memory 8060. The processor 8030 includes a neural network-based codec 8070. The neural network-based codec 8070 implements the disclosed embodiments described above. For example, the neural network-based codec 8070 implements, processes, prepares, or provides various coding operations. Thus, the inclusion of the neural network-based codec 8070 provides a significant improvement to the functionality of the video coding device 8000 and affects the transformation of the video coding device 8000 into different states. Alternatively, the neural network based codec 8070 may be implemented as instructions stored in the memory 8060 and executed by the processor 8030 .

[0328] The memory 8060 may include one or more disks, tape drives, and solid state drives, and may be used as overflow data storage devices, store programs when such programs are selected for execution, and store instructions and data loaded during the execution of the programs. The memory 8060 may be, for example, volatile and / or nonvolatile, and may be read-only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), and / or static random access memory (SRAM).

[0329] FIG. 19 is a simplified block diagram of an apparatus that can be used as either or both of the source device 120 and the destination device 14 of FIG. 16, according to an example embodiment.

[0330] The processor 9002 in the apparatus 9000 may be a central processing unit. Alternatively, the processor 9002 may be any other type of device or devices, now existing or later developed, capable of manipulating or processing information. While the disclosed implementations may be implemented using a single processor, such as processor 9002, as shown, advantages in speed and efficiency may be obtained using more than one processor.

[0331] The memory 9004 in the device 9000 may be a read-only memory (ROM) device or a random access memory (RAM) device in some implementations. Any other suitable type of storage device may be used as the memory 9004. The memory 9004 may include code and data 9006 that is accessed by the processor 9002 using the bus 9012. The memory 9004 may further include an operating system 9008 and application programs 9010, which include at least one program that enables the processor 9002 to perform the methods described herein. For example, the application programs 9010 may include applications 1 through N, which may further include a video coding application that performs the methods described herein.

[0332] The apparatus 9000 may also include one or more output devices, such as a display 9018. The display 9018, in one example, may be a touch-sensitive display that combines a display with touch-sensitive elements operable to sense touch input. The display 9018 may be coupled to the processor 9002 via the bus 9012.

[0333] Although depicted herein as a single bus, the bus 9012 of the device 9000 may be comprised of multiple buses. Additionally, the secondary storage may be directly coupled to other components of the device 9000 or may be accessed over a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, the device 9000 may be implemented in a wide variety of configurations.

Claims

1. 1. A method of operating a neural network including a neural network layer including or connected to an accumulator register for buffering summation results and having a predefined accumulator register size, comprising: defining a lower integer threshold A and an upper integer threshold B for integer values ​​included in data entries of input data for said neural network layer; clipping the integer value contained in a data entry of the input data to a predefined lower integer threshold if the integer value contained in the data entry of the input data is less than a predefined lower integer threshold, and clipping the integer value contained in a data entry of the input data to a predefined upper integer threshold if the integer value contained in the data entry of the input data is greater than a predefined upper integer threshold; and determining integer weights of the neural network layers based on the predefined lower integer threshold, the predefined upper integer threshold, and the predefined accumulator register size so as to avoid integer overflow of the accumulator register; A method comprising:

2. 10. The method of claim 1, wherein the neural network layer is a fully connected layer.

3. 10. The method of claim 1, wherein the neural network layer is a convolutional neural network layer.

4. 4. The method of claim 1, wherein the accumulator register size is n bits, where n is a positive integer value.

5. 5. The method of claim 4, wherein the accumulator register size is equal to one of 32 bits and 16 bits.

6. 6. The method of claim 1, wherein the lower integer threshold is less than or equal to 0 and the upper integer threshold is greater than or equal to 0.

7. 7. The method of claim 6, wherein the integer lower threshold A is −2 k-1 and the upper integer threshold B is given by 2 k-1 −1, where k indicates the predefined bit depth of the layer input data.

8. 8. The method according to claim 1, wherein the neural network layer comprises: [Number 63] wherein D denotes an integer value, W denotes a subset of trainable layer weights, and X denotes one of the set and subset of input data for the neural network layer.

9. 9. The method of claim 8, wherein the integer weights {w i } is a condition [Number 64] The method is determined to satisfy the following:

10. The method of claim 8 when dependent on claim 7, wherein the integer weights {w i } is a condition [Number 65] The method is determined to satisfy the following:

11. The method of claim 8 when dependent on claim 7, wherein the integer weights {w i } is a condition [Number 66] The method is determined to satisfy the following:

12. 12. The method according to any one of claims 8 to 11, wherein D is equal to 0.

13. The method of claim 9 when dependent on claim 3, or claim 10 when dependent on claim 3, or claim 11 when dependent on claim 3, wherein the neural network layer is a two-dimensional convolutional neural network layer, and [Number 67] is expressed as follows: [Number 68] and C in denotes the number of input channels of the neural network layer, and K 1 and K. 2 denotes a convolution kernel size, and j denotes an index of an output channel of the neural network layer.

14. The method of claim 9 when dependent on claim 3, or claim 10 when dependent on claim 3, or claim 11 when dependent on claim 3, wherein the neural network layer is an N-dimensional convolutional neural network layer, and [Number 69] is expressed as follows: [Number 70] and C in denotes the number of input channels of the neural network layer, and K 1 ,K 2 ,..K N denotes a convolution kernel size, and j denotes an index of an output channel of the neural network layer.

15. 15. The method of any one of claims 1 to 14, wherein the neural network layer includes an attention mechanism.

16. 16. The method of any one of claims 1 to 15, further comprising providing weights, scaling the weights by a first scaling factor to obtain scaled weights, and rounding the scaled weights to their nearest integer values ​​to obtain the integer-valued weights.

17. 17. The method of claim 16, wherein the rounding is performed by a floor function or a ceiling function.

18. 18. The method of claim 17, wherein the weights are real-valued weights and the first scaling factor is 2 sj is given by s j indicates the number of bits representing the fractional part of the real-valued weight.

19. 19. The method of any one of claims 1 to 18, further comprising scaling data entries of the input data by a second scaling factor to obtain scaled values ​​of the data entries.

20. 20. The method of claim 19, further comprising the step of rounding the scaled value of the data entry to the nearest integer value to obtain the integer value of the data entry.

21. 21. A method of encoding data, comprising the steps of a method of operating a neural network according to any one of claims 1 to 20.

22. 22. The method of claim 21, wherein encoding the data comprises providing an entropy model using a neural network and entropy encoding the data based on the provided entropy model, and wherein providing the entropy model comprises performing the steps of the method for operating a neural network of any one of claims 1 to 20.

23. 21. The method of claim 20, wherein the entropy model comprises: a) Variational autoencoder hyperprior, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder The method is provided using one of the following:

24. 21. A method of decoding encoded data, comprising the steps of a method of operating a neural network according to any one of claims 1 to 20.

25. 25. The method of claim 24, wherein decoding the data comprises providing an entropy model using a neural network and entropy decoding the data based on the provided entropy model, and wherein entropy decoding the data comprises steps of a method of operating a neural network as claimed in any one of claims 1 to 20.

26. 26. The method of claim 25, wherein the entropy model comprises: a) Variational autoencoder hyperprior, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder The method is provided using one of the following:

27. 27. The method of any one of claims 21 to 26, wherein the data is image data.

28. 1. A method for encoding at least a portion of an image, comprising: converting tensors representing components of the image into latent tensors; Providing an entropy model; and processing the latent tensors using a neural network based on a provided entropy model to generate a bitstream; and wherein providing the entropy model comprises performing the steps of the method of any one of claims 1 to 20.

29. 29. The method of claim 28, wherein the entropy model comprises: a) Variational autoencoder hyperprior, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder The method is provided using one of the following:

30. 1. A method for reconstructing at least a portion of an image, comprising: providing an entropy model; processing the bitstream using a neural network based on the provided entropy model to obtain latent tensors representing components of the image; and processing the latent tensors to obtain tensors representing components of the image; and wherein providing the entropy model comprises performing the steps of the method of any one of claims 1 to 20.

31. 31. The method of claim 30, wherein the entropy model comprises: a) Variational autoencoder hyperprior, b) Autoregressive priors for variational autoencoders, and c) Combining hyperpriors and autoregressive priors in a variational autoencoder The method is provided using one of the following:

32. 32. The method of any one of claims 28 to 31, wherein the component is a Y, U, or V component, or an R, G, or B component.

33. 33. A computer program product causing one or more processors to carry out a method according to any one of claims 1 to 32.

34. 30. Apparatus for encoding data, comprising processing circuitry configured to perform the steps of the method according to any one of claims 1 to 20, 21 to 23 and 28 to 29.

35. 21. An apparatus for encoding at least a portion of an image, the apparatus comprising: a processing circuit configured to convert tensors representing components of the image into latent tensors; provide an entropy model comprising performing the steps of the method of any one of claims 1 to 20; and process the latent tensors using a neural network based on the provided entropy model to generate a bitstream.

36. 32. Apparatus for decoding data, comprising processing circuitry configured to perform the steps of the method of any one of claims 1 to 20, 24 to 26 and 30 to 31.

37. 21. An apparatus for decoding at least a portion of an encoded image, the apparatus comprising: a processing circuit configured to provide an entropy model comprising performing the steps of the method of any one of claims 1 to 20; to process a bitstream using a neural network based on the provided entropy model to obtain latent tensors representing components of the image; and to process the latent tensors to obtain tensors representing components of the image.

Images